General FAQs

I've received my license keys, how do I use them?

If you have setup your cluster using the Web App, your licenses will be automatically added to your machines. In some instances you may be required to manually add these to your system. You should have received an email when the licenses were generated, which will be of the form:

{
    "Info": {
        "Expiry": "2233-03-22"
    },
    "License": {
        "Error": "false",
        "license.b4e36c20-2ce9-d149-ffee-123aa2355d1f": "5ZHb4UZXRvlL2VnB/7huB1XX",
        "license.e259127f-6da2-6b42-bbdf-ba4002389c08": "ha45gP8lakgGG1o1MUAmb1XX"
    }
}

In the above example the licenses are 5ZHb4UZXRvlL2VnB/7huB1XX and ha45gP8lakgGG1o1MUAmb1XX respectively. The two license keys need to be placed in the file /opt/HAC/RSF-1/etc/licenses.txt on each node - the cluster will automatically pick up the relevant license from this file, thereby making administration simpler as a single file can be used to license all cluster nodes (i.e. multiple clusters can use a single license file). Comments can be introduced to the license file using a '#' symbol in column 0, blank lines are ignored.

# Licenses renewed 13 Sept 2020.

# license.b4e36c20-2ce9-d149-ffee-123aa2355d1f
5ZHb4UZXRvlL2VnB/7huB1XX

# license.e259127f-6da2-6b42-bbdf-ba4002389c08
ha45gP8lakgGG1o1MUAmb1XX

Finally, restart RSF-1 to reload the licenses (note - the license file is checked every hour, on the hour by the cluster as a matter of course; any changes detected will cause the file to be reloaded automatically).

Do all client systems that access the cluster have to be in the same network?

Clients access services in the cluster using a virtual IP (VIP) address for each service.

The VIP address is a normal, routable, IP address, and acts like any other such address. If a service is accessible when run as a non clustered service, then it will also be accessible when run as a clustered service.

How do I investigate what triggered a failover?

The main RSF-1 log file is /opt/HAC/RSF-1/log/rsfmon.log. Each time RSF-1 is restarted, or a set log file size limit is reached, this log file is rotated, so after a time the log directory will contain the current log file rsfmon.log, and a historical set prefixed with .0to .9, with .9 representing the oldest log file. Each entry in the log file is timestamped along with the PID of the process that updated the log.

A large amount of data is held in the log file(s), so to assist in extracting the relevant information, RSF-1 comes with the utility /opt/HAC/RSF-1/bin/rsflog-summary.sh which reads the log files and produces a summary of failovers, how long each step took and how long the overall failover took. By default all log files are scanned and summary information produced for all services defined in the cluster. This can be refined to just a spacific service and/or set of log files using options supplied to the command; use the -h option for a summary of all supported options:

root@node1:~# rsflog-summary.sh -h

Usage: rsflog-summary.sh [-l|--log <logfile|logdir>] [service]...

Options:
        -l --log <log>: Specify a log file or log directory to search.
                        If a file is given, only that file will be searched.
                        If a directory is given, that directory will be searched
                        for rsfmon.log and rsfmon.log.[0-9].
                        The default is to use the standard RSF-1 log directory
                        /opt/HAC/RSF-1/log

        -h --help:      Show this help text and exit

Here is an example of running the utility for a cluster with a single pool:

# rsflog-summary.sh
Services to search for:  cpw_pool
Searching log files in /opt/HAC/RSF-1/log:
rsfmon.log.9 rsfmon.log.8 rsfmon.log.7 rsfmon.log.6 rsfmon.log.5 rsfmon.log.4 rsfmon.log.3 rsfmon.log.2 rsfmon.log.1 rsfmon.log.0 rsfmon.log

###############################################
Service startups and shutdowns for cpw_pool:
###############################################
Reading rsfmon.log.9...
Reading rsfmon.log.8...
Reading rsfmon.log.7...
Reading rsfmon.log.6...
Reading rsfmon.log.5...
Reading rsfmon.log.4...
Reading rsfmon.log.3...
Reading rsfmon.log.2...
Reading rsfmon.log.1...
Reading rsfmon.log.0...
Reading rsfmon.log...

Delay before service start - 6 seconds
Service startup #1: Feb 04 10:59:25 (rsfmon.log)
Fping test complete for testvip: 2 seconds
Script S01announce          run time: 0 seconds (write to log file)
Script S02ApplianceStarting run time: 0 seconds (event notify)
Script S14res_drives        run time: 6 seconds (create initial res_drives)
Script S15zfs_mhdc          run time: 5 seconds (place reservations)
        Reservations taken in 4 seconds.
Script S20zfs               run time: 3 seconds (import pool + LUs)
        Zpool import completed status 0, in 3 seconds
        Comstar mapping restored in 0 seconds
Script S21res_drives        run time: 1 seconds (refresh res_drives)
Script S98ApplianceStarted  run time: 1 seconds (event notify)
Script S99announce          run time: 1 seconds (write to log file)
Service start scripts took 17 seconds
Plumb in VIP interface: 0 seconds
Total run time for service start: 17 seconds

Delay before service start - 6 seconds
Service startup #2: Feb 24 15:06:26 (rsfmon.log)
Fping test complete for testvip: 2 seconds
Script S01announce          run time: 0 seconds (write to log file)
        Reservations released in 0 seconds.
Script S02ApplianceStarting run time: 1 seconds (event notify)
Script S14res_drives        run time: 6 seconds (create initial res_drives)
Script S15zfs_mhdc          run time: 4 seconds (place reservations)
        Reservations taken in 4 seconds.
Script S20zfs               run time: 4 seconds (import pool + LUs)
        Zpool import completed status 0, in 3 seconds
        Comstar mapping restored in 0 seconds
Script S21res_drives        run time: 0 seconds (refresh res_drives)
Script S98ApplianceStarted  run time: 1 seconds (event notify)
Script S99announce          run time: 0 seconds (write to log file)
Service start scripts took 16 seconds
Plumb in VIP interface: 0 seconds
Total run time for service start: 16 seconds

For each failover summary a Service startup #<no>: line included near the beginning that documents the time of the failover. This timestamp is then used as a reference to search the logfiles on the other cluster node for an explanation of what event occured just prior to the failover being instigated.

What are the prop_zpool_fail_mode properties for?

The failmode property of a zpool controls how the pool handles I/O after it has gone into a 'faulted' state. There are 3 options:

1. wait - all I/O from clients will hang
2. continue - clients will get I/O errors for all I/O operations to the pool
3. panic - as soon as the pool goes faulted, ZFS triggers a kernel panic

For RSF clusters, panic should be used (which is not the default failmode property on newly created pools). The mode setting of panic means if a pool goes faulted due to a faulty controller card, broken fibre cable, etc. the active node will panic, and the service can automatically fail over to the other node.

RSF-1 is configured by default to change the failmode of all zpools to panic each time a service starts. If this behaviour is not wanted for any reason, it can be changed by altering an RSF database property.

The property prop_zpool_fail_mode controls the failmode on a cluster-wide basis. If it is necessary to have a pool use a different failmode, then a new property can be created with the format prop_zpool_fail_mode_<pool> (note that this is a pool name, not a service name; if a service contains more than one pool, then a separate property can be declared for each pool).

To modify the cluster wide failmode property to wait run:

rsfcdb update prop_zpool_fail_mode wait

To add a new property (in this case continue) specifically for the pool tank, run:

rsfcdb create prop_zpool_fail_mode_tank continue

Possible values for the global and individual pool setting are wait, continue, panic and none. A value of none means RSF will not set the failmode of pools on import, so they will retain the failmode setting they already had.

Possible values for the pool specific settings are wait, continue, panic, none and default. A value of default effectively disables the setting and causes that pool to use the global value prop_zpool_fail_mode. A value of none causes RSF not to set the failmode of this pool at all.

For example, if there are 5 pools in a cluster, pool1, pool2, pool3, pool4 and pool5, the properties:

prop_zpool_fail_mode       : panic
prop_zpool_fail_mode_pool1 : wait
prop_zpool_fail_mode_pool2 : default
prop_zpool_fail_mode_pool3 : none
prop_zpool_fail_mode_pool4 : continue

mean that the following failmodes are applied:

pool1 - wait
pool2 - panic
pool3 - no failmode setting used (keeps its original setting)
pool4 - continue
pool5 - panic (as there is no specific declaration for pool5, the default is used)

Does RSF-1 support one node cluster configurations?

Yes it does.

Here is an example of the RSF-1 configuration file for a one node cluster, where the pool name is tank1, the VIP floating address is 10.0.2.201, and the network interface is e1000g0:

#
# Created Wed 2014-10-08 15:31:48 BST by RSF-1 config_db
# This file is automatically generated by rsfcdb - DO NOT EDIT BY HAND
#
# [SINGLE NODE CONFIGURATION] - some properties will be disabled.
#
# [Global Section]
#  Global settings affecting the whole cluster.
#
CLUSTER_NAME My-Single_Cluster
#DISC_HB_BACKOFF 20,3,3600
#POLL_TIME 1
REALTIME 1
EVENT_NOTIFY "/opt/HAC/RSF-1/bin/event_notifier"
#IPDEVICE_MONITOR 3,2

#
# [Machine Section]
#  Specifies the machines in the cluster and the heartbeats between them.
#  Note! Heartbeats are disabled in single-node cluster mode.
#
MACHINE node-a

#
# [Services Section]
#  Specifies the services in the cluster, the nodes they can
#  run on and the IP addresses used to access those services.
#
SERVICE service1 10.0.2.201 "Single Node Service"
 OPTION "sdir=appliance"
 INITIMEOUT 20
 RUNTIMEOUT 8
 MOUNT_POINT "/volumes/tank1"
 SERVER node-a
  IPDEVICE "e1000g0"

Using rsfcdb to generate a single node configuration

In order to generate a single node cluster configuration file using rsfcdb, set the single node property prop_single_node_cluster to true using the following command:

rsfcdb update prop_single_node_cluster true

rsfcdb config_preview

Please note the following points when generating a single node cluster using rsfcdb:

• Any global section values that are not required for a single node cluster are commented out of the resulting configuration.
• Only CLUSTER_NAME, EVENT_NOTIFY and REALTIIME are valid for single node cluster.
• The heartbeat section is replaced by the machine name added via the ga_name sub-command of rsfcdb.
• Each service has also been limited to displaying one SERVER section where the SERVER parameter matches the machine name.

NFSv4 Failover

NFS Version 4 has removed support for UDP as an underlying transport protocol (as opposed to V3 which supported both UDP and TCP), therefore all NFSV4 connections are TCP based. This exclusive use of TCP in NFSv4 has implications for failover recovery time in certain scenarios due to TCP's TIME_WAIT (sometimes referred to as 2MSL) state that can be entered into during multiple failover operations.

The reason for a TCP socket to enter a TIME_WAIT state is to prevent delayed data packets from one connection being misinterpreted as part of a subsequent newly established connection on the same machine (applying stale data from a previous connection to the current one could have potentially disastrous affects on data integrity).

The implication of TIME_WAIT for failover is observed when HA services are moved from one node to another and then back again in a short period of time. Once the initial move is complete the originating server enters the TIME_WAIT state as part of the normal TCP protocol. If during this 'wait period' services are moved back to the originating server clients will be unable to re-connect until the TIME_WAIT period has expired (and in some cases the client connections will timeout themselves), therefore manual moves back and forth in quick succession (circa 2-4 minutes) between machines that provide data over NFSv4 should be avoided. This type of quick failover/failback scenario is only normally seen during system testing exercises and not representative of production environments.

For machines where the failover was instigated as a result of a systems crash, TIME_WAIT is irrelevant as the TCP connections will have no knowledge of the previous connection.

What does broken_safe and broken_unsafe mean and how do I fix it?

Broken_safe and broken_unsafe refer to a state of an RSF-1 service that has either failed to start up or shut down correctly.

As a service is started or stopped RSF-1 executes the scripts in the directory /opt/HAC/RSF-1/etc/rc.<service>.d/* where <service> is the service name itself; for example a service named web would have the service directory /opt/HAC/RSF-1/etc/rc.web.d/. The service directory contains three types of scripts:

• start - prefixed by an S<num>
• stop - prefixed by a K<num>
• panic - prefixed by a P<num>

The order in which the scripts are run is dictated by the <num> portion of the prefix, going from low to high. The scripts perform actions to either start or stop a service. Each script should run successfully and complete with a 0 exit code. However, if during the running of one of these scripts something goes wrong, then the script will exit with a non zero exit code (exit code definitions can be found in /opt/HAC/bin/rsf.sh).

If an error occurs when running the start or stop scripts, a script can indicate this in its exit code. If the failure occurred when starting a service, then the shutdown scrips are run to release any shared resources that the failed startup attempt may have reserved/started etc.

If the start scripts failed, and the following stop scripts succeeded, the service is marked broken_safe. Broken indicates that something is wrong - the service could not be started, and this should be investigated and remedied before trying to start the service on this server again. The safe part indicates that the stop scrips completed successfully, so no shared resources are allocated and it is safe to try to start the service on a different server.

However, if an error occurs when running the stop scripts, (e.g. failure to unmount a shared file system, even with a forcible unmount), then the service is marked broken_unsafe. As before broken indicates that some investigation is required, but this time the unsafe suffix means that shared resources may still be allocated, and therefore it is NOT safe to try to start the service elsewhere in the cluster (for example, if you were to try to mount and use the file system on another host data corruption could occur).

It is also possible for the startup scripts to indicate that the service should be marked broken_unsafe immediately, without running the stop scripts. This is to allow for situations in which a severe error has been detected by the start scrips, and that running the stop scrips, or allowing another server to try to start the service, may further exasperate the situation.

In either case, the underlying issue causing the broken state needs to be resolved. Check the log file /opt/HAC/RSF-1/log/rsfmon.log to discover where the error occured and what needs to be done. Once the problem has been resolved, RSF-1 needs to be told that the issue is now resolved; do this by first issuing the command (as root):

/opt/HAC/RSF-1/bin/rsfcli -i=0 repaired <servicename>

/opt/HAC/RSF-1/bin/rsfcli -i=0 auto <servicename>

Problems accessing the RSF-1 WebApp

If the WebApp becomes inaccessible and you cannot connect via a browser, a restart of the webapp services may be required.

!!!

Linux

systemctl restart rsf-gui.service

BSD

service rsfwebapp restart

TrueNAS

/opt/HAC/RSF-1/init.d/rc-webapp onerestart

OmniOS/Solaris

svcadm restart rsf-gui