SSH Binding

Using SSH to bind two nodes together

A shared nothing cluster operates by creating incremental snapshots of data sets and then synchronising them between cluster nodes using ZFS send/receive over an ssh tunnel.

The ssh tunnel created and used by the synchronisation process needs to be passwordless and therefore the two nodes need to be ssh-bound. To configure ssh binding perform the following steps on each node:

1. Create your ssh keys as the root user (press return to accept the defaults for all prompts):

   # ssh-keygen
   Generating public/private rsa key pair.
   Enter file in which to save the key (/root/.ssh/id_rsa):
   Enter passphrase (empty for no passphrase):
   Enter same passphrase again:
   Your identification has been saved in /root/.ssh/id_rsa
   Your public key has been saved in /root/.ssh/id_rsa.pub
   The key fingerprint is:
   SHA256:2dGrTFvaGz8QJbVeVGS5sFv/deJRngPvSOr6v1SaMXc root@NodeA
   The key's randomart image is:
   +---[RSA 3072]----+
   |             ...B|
   |           ....= |
   |          . .o+ o|
   |         o ..= +.|
   |        S o o+*=E|
   |         o *.oX+*|
   |          = =*oo=|
   |           ..=o..|
   |         .+oooo. |
   +----[SHA256]-----+
   

2. Once ssh-keygen has been run, a public key is saved to /root/.ssh/id_rsa.pub. This public key now needs to be added to the file /root/.ssh/authorized_keys on the other node (if the authorized_keys file does not exist simply create one).

3. Manually ssh NodeA > NodeB then NodeB > NodeA and accept the prompt to add each machine to the list of known hosts:

   root@NodeA:~# ssh root@NodeB
   The authenticity of host 'NodeB (10.10.10.2)' can't be established.
   ED25519 key fingerprint is SHA256:EDmzS45TqKabZ53/35vXb4YyKTQuzJxNnbFuIwFj9UU.
   Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
   Warning: Permanently added 'NodeB,10.10.10.2' (ED25519) to the list of known hosts.
   Last login: Tue Sep 12 09:54:49 2023 from 10.10.10.1
   Oracle Solaris 11.4.42.111.0                  Assembled December 2021
   root@NodeB:~#
   

Once this process has been completed you should be able to ssh between nodes without being prompted for a password.

SSH login between nodes taking a long time

If ssh is taking a long time, try running ssh -v to see any errors that may be causing the delay. A common issue is with GSS/Kerberos:

debug1: Next authentication method: gssapi-with-mic

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1000' not found


debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1000' not found

This can be disabled on the nodes by modifying /etc/ssh/ssh_config and disabling all Host options that begin with GSS. For example:

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
    GSSAPIAuthentication no
    GSSAPIDelegateCredentials no
    GSSAPIKeyExchange no
    GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
#   UserKnownHostsFile ~/.ssh/known_hosts.d/%k
    SendEnv LANG LC_*