Networking FAQs - Linux

Allowing secondary IP's on interfaces to be promoted when primary is removed

Synopsis

If service failover/halting is causing vips from another service to be removed, then the likely cause is the non-promotion of secondary IP addresses caused by the promote_secondaries IPv4 system configuration.

Note

This FAQ entry is only relevant to interfaces used in a cluster that have no permanent static IP address assigned, for example:

# ip a l ens19
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP group default qlen 1000
link/ether 5e:36:85:51:89:ef brd ff:ff:ff:ff:ff:ff
altname enp0s19
inet6 fe80::5c36:85ff:fe51:89ef/64 scope link
   valid_lft forever preferred_lft forever

When configuring IP's on Linux using the ip command, the first IP added to an interface (in a specific subnet) is assigned as the primary address; any additional addresses added in the same subnet will be flagged as secondary, for example:

# ip a l ens19
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP group default qlen 1000
    link/ether 5e:36:85:51:89:ef brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    inet 172.16.20.10/24 scope global ens19
       valid_lft forever preferred_lft forever
    inet 172.16.20.11/24 scope global secondary ens19
       valid_lft forever preferred_lft forever
    inet 172.16.20.12/24 scope global secondary ens19
       valid_lft forever preferred_lft forever
    inet6 fe80::5c36:85ff:fe51:89ef/64 scope link
       valid_lft forever preferred_lft forever

Should the primary address (172.16.20.10 in this case) be removed, any secondary IP's in the same subnet are impacted by the system setting net.ipv4.conf.<selector>.promote_secondaries¹. A value of 0 results in those addresses being removed, i.e.:

# ip -f inet address del 172.16.20.10/24 dev ens19
# ip a l ens19
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP group default qlen 1000
    link/ether 5e:36:85:51:89:ef brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    inet6 fe80::5c36:85ff:fe51:89ef/64 scope link
       valid_lft forever preferred_lft forever

Whereas a value of 1 results in one of the secondaries being promoted to primary:

# ip -f inet address del 172.16.20.10/24 dev ens19
# ip a l ens19
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP group default qlen 1000
    link/ether 5e:36:85:51:89:ef brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    inet 172.16.20.11/24 scope global ens19
       valid_lft forever preferred_lft forever
    inet 172.16.20.12/24 scope global secondary ens19
       valid_lft forever preferred_lft forever
    inet6 fe80::5c36:85ff:fe51:89ef/64 scope link
       valid_lft forever preferred_lft forever

In a cluster environment where multiple VIPs are configured on the same subnets, the removal of a primary VIP resulting in a secondary VIP(s) being removed can result in a loss of service to clients. Therefore the correct behaviour is always to promote secondary VIPs.

Example

Consider a cluster with two services, each with their own VIP. When those two services are running on a single node, one of the VIPs will be primary and the other secondary. Should the service with the primary VIP be moved to another server, then the removal of it's VIP as part of the failover will cause the secondary VIP to also be removed and thus impact the accessibility of that service to clients.

To enable on ALL interfaces:

# sysctl net.ipv4.conf.all.promote_secondaries=1

To make this the default action enable the default setting:

# sysctl net.ipv4.conf.default.promote_secondaries=1

OR for individual interfaces:

# sysctl net.ipv4.conf.enp3s0f3.promote_secondaries=1

To make a permanent change to the system, update /etc/sysctl.conf with:

# avoid deleting secondary IPs on deleting the primary IP
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1

and reload with:

# sysctl -p /etc/sysctl.conf

The <selector> can be all, default or a specific interface, i.e. enp0s19. ↩