Configuration Guide

Introduction

This guide is for when a cluster and services have been created and configured; for details on creating clusters please see the following guides:

• BSD
• Linux

Terminology

Services

In an RSF-1 cluster a service refers to a ZFS pool that is managed by the cluster. The cluster may consist of one or more services under it's control, i.e. multiple pools. Furthermore an individual service may consist of more than one pool - refered to as a pool group, where actions perfromed on that service will be performed on all pools in the group.

A service instance is the combination of a service and a cluster node that that service is eligible to run on. For example, in a 2-node cluster each service will be configured to have two available instances - one on each node in the cluster. Only one instance of a service will be active at any one time.

Modes (automatic/manual)

Each service instance has a mode setting of either automatic or manual. The mode of a service is specific to each node in the cluster, so a service can be manual on one node and automatic on another. The meaning of the modes are:

AUTOMATIC

Automatic mode means the service instance will be automatically started when all of the following requirements are satisfied:

• The service instance is in the stopped state
• The service instance is not blocked
• No other instance of this service is in an active state

MANUAL

Manual mode means the service instance will never be started automatically on that node.

State (running/stopped etc)

A service instance in the cluster will always be in a specific state. These states are divided into two main groups, active states and inactive states¹. Individual states within these groups are transitional, so for example, a starting state will transition to a running state once the startup steps for that service have completed successfully, and similarly a stopping state will transition to a stopped state once all the shutdown steps have completed successfully (note that this state change stopping==>stopped also moves the service instance from the active state group to the inactive state group).

Active States

When the service instance is in an active state, it will be utilising the resources of that service (e.g. an imported ZFS pool, a plumbed in VIP etc.). In this state the service is considered up and running and will not be started on any other node in the cluster until it transitions to a inactive state; for example if a service is STOPPING on a node it is still in an active state, and cannot yet be started on any other node in the cluster until it transitions to a inactive state - see below for the definition of inactive states.

The following table describes all the active states.

Inactive States

When a service instance is in an inactive state, no service resources are online. That means it is safe for another instance of the service to be started elsewhere in the cluster.

The following table describes all the inactive states.

Blocked (blocked/unblocked)

The service blocked state is similar to the service mode (AUTOMATIC/MANUAL) except that instead of being set by the user, it is controlled automatically by the cluster's monitoring features.

For example, if network monitoring is enabled then the cluster constantly checks the state of the network connectivity of any interfaces VIP's are plumbed in on. If one of those interfaces becomes unavailable (link down, cable unplugged, switch dies etc.) then the cluster will automatically transition that service instance to blocked.

If a service instance becomes blocked when it is already running, the cluster will stop that instance to allow it to be started on another node so long as there is another service instance in the cluster that is UNBLOCKED, AUTOMATIC and STOPPED, otherwise no action will be taken.

Also note, a service does not have to be running on a node for that service instance to become blocked - if a monitored resource such as a network interface becomes unavailable then the cluster will set the nodes service instance to a blocked state, thus blocking that node from starting the service. Should the resource become available again then the cluster will clear the blocked state.

The following table describes all the blocked states.

Dashboard

The Dashboard is the initial landing page when connecting to the webapp once a cluster has been created. It provides a quick overview of the current status of the cluster and allows you to perform operations such as stopping, starting and moving services between nodes:

The dashboard is made up of three main sections along with a navigation panel on the left hand side:

• The status panel located at the top of the page providing a instant view of the overall health of the cluster with node, service and heartbeat summary status.
• The nodes panel detailing each nodes availability in the cluster along with its IP address and heartbeat status.
• The services panel detailing the services configured in the cluster, which node thay are running on, if any, and any associated VIPs.

Clicking on the icon for an individual node or service brings up a context sensitive menu, described in the following sections.

Nodes panel

The nodes panel shows the status of each node in the cluster:

Clicking on a node opens a side menu that allows control of services known to that node. In the example above, clicking on the icon for node-a would bring up the following menu:

Available actions can then be viewed by clicking on the ⋮ button in the right hand column for an individual service:

Alternatively, the ⋮ button on the Clustered Services row brings up a menu that performs actions on all services on that node:

Services Panel

The services panel shows the status of each service in the cluster:

Clicking on a service opens up a side menu that allows control of that service in the cluster. In the example above clicking on the icon for pool1 would bring up the following menu:

Available actions can then be viewed by clicking on the ⋮ button in the right hand column for an individual service:

Clustering a Docker Container

These steps show the process of creating a Clustered docker container. The container will be created using a standard docker compose.yaml file.

1. Navigate to HA-Cluster -> Docker in the webapp:

   

2. Click Cluster a Docker application to get to the creation/addition page and fill in the fields

   Available options:

   • Select HA Service - Select the service/pool to associate the container to in the event of a failover
   • Container Description - Optional description of the container
   • Location of compose.yaml file within selected service - The path in the selected pool/service to save the compose.yaml
   • Contents of compose.yaml file - Enter the contents of the compose.yaml file for the container
   • An Example compose.yaml:

   services:
     apache:
      image: httpd:latest
      container_name: my-apache-app
      ports:
       - 8080:80
      volumes:
       - ./website:/usr/local/apache2/htdocs
      restart: no
   

   Warning

   When adding your content, make sure to add restart: no to your service configurations. RSF-1 will manage the restart of clustered containers in the event of a failover

   

   

3. When finished click Create.

   

4. By default the container will remain stopped until started. Click the Start button to spin up the container.

   

Heartbeats

In the cluster, heartbeats perform the following roles:

• To continually monitor the other nodes in the cluster, ensuring they are active and available.
• Communicate cluster and service status to the other nodes in the cluster. Status information includes mode and state for every service on that node (manual/automatic running/stopped etc), along with any services that are currently blocked.
• A checksum of the active cluster configuration on that node.

The cluster supports two types of heartbeats:

• Network heartbeats
• Disk heartbeats

Heartbeats are unidirectional therefore for each heartbeat configured there will be two channels (one to send and one to receive).

The same information and structures is transmitted over each type of heartbeat. The cluster supports multiple heartbeats of each type. When the cluster is first created a network hearbeat is automatically configured between cluster nodes using the node hostnames as the endpoints. Disk heartbeats are automatically configured when a service is created and under normal circumstances require no user intervention.

It is recommended practice to configure network heartbeats across any additional network interfaces. For example, if the hostnames are on a 10.x.x.x network, and an additional private network exists with 192.x.x.x addresses, then an additional heartbeat can be configured on that private network. Using the following example hosts file an additional network heartbeat can be configured using the node-a-priv and node-b-priv addresses as endpoints:

10.0.0.1 node-a
10.0.0.2 node-b
192.168.72.1 node-a-priv
192.168.72.2 node-b-priv

By specifying the endpoint using the address of an additional interface the cluster will automatically route heartbeat packets down the correct network for that interface.

To view the cluster heartbeats navigate to HA-Cluster -> Heartbeats on the left side-menu:

Adding a Network Heartbeat

To add an additional network heartbeat to the cluster, select Add Network Heartbeat Pair. In this example an additional physical network connection exists between the two nodes. The end points for this additional network are given the names SAM node-a-priv and node-b-priv respectively. These hostnames are then used when configuring the additional heartbeat:

Click Submit to add the heartbeat. The new heartbeat will now be displayed on the Heartbeats status page:

Removing a Network Heartbeat

To remove a network heartbeat select the heartbeat using the slider on the left hand side of the table and click the remove selected button:

Finally, confirm the action:

Disk heartbeats

Under normal circumstances it should not be necessary to add or remove disk heartbeats as this is handled automatically by the cluster.

NFS shares

Enabling clustered NFS

By default RSF-1 does not manage NFS shares - the contents of the /etc/exports file are left to be managed by the system administrator manually on each node in the cluster. To enable the management of the exports file from the webapp and synchronise it across all cluster nodes, navigate to Shares -> NFS and click ENABLE NFS SHARE HANDLING:

Once enabled the shares table will be shown:

Before creating new shares the option to import the existing /etc/exports file is available (this option is disabled once any new shares are added via the webapp):

Clustering an NFS share

1. Navigate to Shares -> NFS and click +Add on the NFS table to fill in the required info. The available options are:

   • Description - Description of the Share (optional)
   • Path - Path of the directory/dataset to share - for example /pool1/nfs
   • Export Options - For a detailed description of the available options click the SHOW NFS OPTIONS EXAMPLES button.

   

2. Click ✓ to add the share:

   

   The share will now be available and clustered.

/tank      10.10.23.4(fsid=1)
/sales     10.01.23.5(fsid=2,sync,wdelay,no_subtree_check,ro,root_squash)
/accounts  accounts.dept.foo.com(fsid=3,rw,no_root_squash)

/srv/nfs       192.168.7.0/24(rw,fsid=root)
/srv/nfs/data  192.168.7.0/24(fsid=1,sync,wdelay,no_subtree_check,ro,root_squash)

Modifying an NFS Share

To modify an NFS chare, click the pencil icon to the left of the dataset:

When done, click ✓ to update the share.

Deleting an NFS Share

To delete an NFS share click the trash can icon and then confirm the deletion

SMB shares

Enabling Samba/SMB in the cluster

By default RSF-1 does not manage SMB shares - the smb.conf file is left to be managed by the system administrator manually on each node in the cluster. To enable the management of the SMB from the webapp and synchronise it across all cluster nodes, navigate to Shares -> SMB and click ENABLE SMB SHARE HANDLING:

You will now be presented with the main SMB shares screen consisting of a number of tabs to handle different aspects of SMB configuration.

Initial SMB configuration

SMB/Samba provides numerous ways to configure authentication and sharing depending upon the environment and the complexity required. This guide documents two commonly used configurations:

• User Authentication - standalone clustered SMB sharing with local user authentication.
• ADS Authentication - member of an Active Directory domain with authentication managed by a domain controller.

Local User Authentication

With local user authentication, cluster users must be created with the SMB support enabled. A user created this way will have the same login name, UID and GID on all nodes in the cluster along with an equivalent Samba user entry to provide the required SMB authentication. See Unix users in this guide for further details.

Configuring Samba Globals for User Authentication

Navigate to Shares -> SMB, select the GLOBALS tab, then select User from the drop down security list and optionally set the desired workgroup name. Click save changes:

ADS Authentication

RSF-1 can also be configured to use Active Directory for user authentication (ADS) when being deployed for use in a Microsoft environment.

In a Microsoft environment users are identified using security identifiers (SIDs). A SID is not just a number, it has a structure and is composed of several values, whereas Unix user and group identifiers consist of just a single number. Therefore a mechanism needs to be chosen to map SIDs to Unix identifiers. Winbind (part of the Samba suite) is capable of performing that mapping using a number of such mechanisms known as Identity Mapping Backends; two of the most commonly used being tdb (Trivial Data Base) and rid (Relative IDentifier).

tdb - The default idmap backend is not advised for an RSF-1 cluster as tdb generates and stores UIDs/GIDs locally on each cluster node, and works on a "first come first served" basis. When allocating UIDs/GIDs it simply uses the next available number with no consideration given to a clustered environment, which can lead to UID/GID mismatches between cluster nodes.

rid - This mechanism is recommended as the idmap backend for a clustered environment. rid implements a read-only API to retrieve account and group information from an Active Directory (AD) Domain Controller (DC) or NT4 primary domain controller (PDC). Therefore using this approach ensures UID/GID continuity on all cluster nodes.

When using the rid backend, a windows SID (for example S-1-5-21-1636233473-1910532501-651140701-1105) is mapped to a UNIX UID/GID by taking the relative identifier part of the SID (the last set of digits - 1105 in the above example) and combining it with a preallocated range of numbers to provide a unique identifier that can be used for the UNIX UID/GID. This preallocated range is configured using the Samba IDMAP entry.

Configuring Samba Globals for ADS Authentication

1. Navigate to Shares -> SMB, select the GLOBALS tab, then select ADS from the drop down security list, set the workgroup and realm name. Click save changes:

   

2. Navigate and open the IDMAP setting section. By default a wildcard entry is preconfigured (this is used by Samba as a catchall and is a required entry). Click +Add and configure an entry for the rid mapping. Enter the same workgroup name used in the security settings and enter the desired range of numbers to use for mapping the IDs. Select a range that starts after the wildcard range and provide enough scope to cover the expected maximum number of windows users in the domain. Click the ✓ to update the mapping table:

   

3. Click SAVE CHANGES. The resulting configuration file (viewable from the CONFIG tab) should look similar to the following:

   [global]
     encrypt passwords = Yes
     idmap config * :    backend = tdb
     idmap config * :    range = 3000-100000
     idmap config HACLAB :    backend = rid
     idmap config HACLAB :    range = 100001-300000
     realm = HACLAB.COM
     security = ADS
     server role = Member Server
     workgroup = HACLAB
   

Samba is now configured to be able use ADS authentication.

Testing ADS authentication (optional)

ADS authentication can be tested by allowing users from the windows domain to login to the Unix cluster hosts. A successful login proves that Samba is able to authenticate using the Windows Domain Controller.

Some additional configuration is required as follows (remember to do this on all nodes in the cluster):

1. Configure winbind authentication for users and groups in the name service switch file /etc/nsswitch.conf. Add winbind as a resolver for users and groups:

   passwd:     files winbind systemd
   group:      files winbind systemd
   

   This tells the operating system to lookup users locally first (/etc/passwd), followed by winbind.

2. Change DNS in /etc/resolv.conf so it refers to the Active Directory server:

   domain haclab.com
   search haclab.com
   nameserver 10.254.254.111
   

3. Join the active directory domain.

   # net ads join -U Administrator
   Password for [HACLAB\Administrator]:
   Using short domain name -- HACLAB
   Joined 'WCALMA1' to dns domain 'haclab.com'
   No DNS domain configured for wcalma1. Unable to perform DNS Update.
   DNS update failed: NT_STATUS_INVALID_PARAMETER
   
   # net ads info
   LDAP server: 10.254.254.111
   LDAP server name: ws2022.haclab.com
   Realm: HACLAB.COM
   Bind Path: dc=HACLAB,dc=COM
   LDAP port: 389
   Server time: Fri, 13 Sep 2024 17:07:32 BST
   KDC server: 10.254.254.111
   Server time offset: 1
   Last machine account password change: Fri, 13 Sep 2024 16:57:04 BST
   

4. Restart winbind.

   # systemctl restart winbind
   

5. Query winbind to confirm it is able to query the Active Directory server:

   # wbinfo -U
   HACLAB\administrator
   HACLAB\guest
   HACLAB\krbtgt
   HACLAB\hacuser1
   HACLAB\hacuser2
   

Samba can be further configured to allow AD users to login if so desired. Two further steps are required:

1. Enable auto creation of home directories. For Debian based systems:

   # vi /etc/pam.d/common-session
   # add to the end if you need (auto create a home directory at initial login)
   session optional        pam_mkhomedir.so skel=/etc/skel umask=077
   

   For RedHat based systems:

   # authselect enable-feature with-mkhomedir
   # systemctl enable --now oddjobd
   

2. Configure a login shell for Samba. Navigate to Shares -> SMB, select the GLOBALS tab, expand the miscellaneous section and set an appropriate login shell for the system:

3. Finally, restart winbind.

   # systemctl restart winbind
   

It should now be possible to login to the Unix servers using AD users.

Clustering an SMB Share

These steps show the process of creating a clustered SMB share.

1. Shares are managed via the SHARES tab. Click +Add to create a new SMB Share

   Available options:

   • Share Name The name of the SMB share
   • Path Path of the folder to be shared, for example /pool1/SMB
   • Valid Users A space separated list of Valid users. When User authentication is in effect these will be Unix Cluster users; for ADS authentication this can be local Unix cluster and Windows domain users.

   

2. Click ✓ when done. The share will now be available and clustered (a Samba reload is automatically applied):

   

3. Advanced share settings can be applied once the share is created by clicking the cog on the left hand side of each individual share:

   

Additional SMB settings

LOCAL CONFIG

This tab is used to apply Samba configuration settings specific to each node rather than all cluster nodes. One example of this is the netbios name which needs to be unique on each node in the Windows Domain.

CONFIG

The tab shows the current Samba configuration for each cluster node; this view includes the globals, shares and local configuration.

STATUS

This tab shows the status of the Samba daemon services that are running. It also allows management of the services per node.

Unix Users

Creating Users

Creating Unix users in the WebApp will create the user across all cluster nodes using the same credentials (Username, UID and GUID).

1. In the WebApp, navigate to System -> UNIX Users, and click +Add:

   

2. Enter the Username and Password, and provide any of the additional information if required:

   • List of Groups - Add the user to any available groups (optional)
   • UID/GID - Specify the User ID and Group ID of the user (optional - if unspecified the next available UID/GID will be used).
   • Add user to sudo group - This user will be able to issue commands as a different user (requires sudo package to be installed).
   • Enable SMB support for user - Adds this user to the valid Samba users.
   • Home Directory - Specify location for the user home directory (optional)
   • Shell - Specify the default shell for the user (optional)

   

3. When done click SAVE. Once saved, the user will be created on all nodes in the cluster:

   

   Warning

   If the user name or UID specified already exists on any node in the cluster then the user add operation will fail with the message "Error creating user clusterwide..."

Modifying Users

To modify a user, click on the pencil icon on the left hand side of the user list table:

Deleting Users

To delete a user from all cluster nodes click trash can icon and then confirm the deletion: