Setting up an HA active/passive NAT gateway

THIS IS A DRAFT – the states are NOT synced although we simplified the PoC as much as possible

we rather prefer PFSYNC anyhow

tested on debian12 and slack150

Introduction

Connection tracker should help to keep up with the layer 4 states e.g. during VRRP fail-over.

Warnings / lessons learned

working setup:

nodes are reachable by SSH by other means (doing DNAT somewhere else) to actually be able to read the logs and open multiple sessions
nodes have THE-REAL-GATEWAY as default gateway anyway, to be ready for fail-over
nodes are all living on the same dom0 to avoid briding and network issues on the VMM farm (just in case)

about config replication:

nftables setup is identical on both nodes
keepalived setup idem
conntrackd needs to be tweaked according to its binding interface, peers and ignore list

Requirements

network setup

XEN guest setup: two guests have three interfaces each, xenbr0 going public, guestbr0 going dmz and br0 as cluster network

e.g. vrrp1

vif = [ 'bridge=xenbr0, vifname=vrrp1.0',
    'bridge=guestbr0, vifname=vrrp1.1',
    'bridge=br0, vifname=vrrp1.3' ]

those are reachable from the inside

e.g. vrrp1

auto eth0
iface eth0 inet static
    # fake ip for keepalive to avoid complaining
    # entering FAULT state (no IPv4 address for interface)
    address 10.4.4.31/24
    # eventhough it is not on our subnet
    # fortunately that works at boot-time
    gateway THE-REAL-GATEWAY

auto eth1
iface eth1 inet static
    address 10.1.0.31/16

auto eth2
iface eth2 inet static
    address 10.3.3.31/24

setup static names - from most trusted to untrusted networks

10.3.3.31       vrrp1
10.3.3.32       vrrp2

10.1.0.31       vrrp1
10.1.0.32       vrrp2

10.4.4.31       vrrp1
10.4.4.32       vrrp2

make sure everything responds to ping from node1 to node2 (and since there’s an answer, that goes both ways)

for host in 10.3.3.32 10.1.0.32 10.4.4.32 10.1.0.50; do
    ping -c1 $host >/dev/null && echo $host OK || echo $host NOK
done; unset host
nmap -p 22 10.1.0.50

sysprep

    mv -i /etc/sysctl.conf /etc/sysctl.conf.dist
    vi /etc/sysctl.conf

    # vrrp
    net.ipv4.ip_forward = 1
    net.ipv4.ip_nonlocal_bind = 1

    # state tracker
    #net.netfilter.nf_conntrack_tcp_be_liberal = 1
    net.netfilter.nf_conntrack_tcp_loose = 0
net.netfilter.nf_conntrack_helper = 0

    sysctl -p

nat setup & drop policy

cd /etc/
mv -i nftables.conf nftables.conf.dist
vi nftables.conf

here we go DNAT only

define nic=eth0
define gst=eth1
define sync=eth2

table inet filter
flush table inet filter
table inet filter {
    chain input {
        type filter hook input priority filter; policy accept;
    }
    chain forward {
        type filter hook forward priority filter; policy drop;

        ip protocol icmp accept

        ct state invalid log prefix "FORWARD INVALID: " drop

        iif $nic ct state new tcp flags syn accept
        iif $nic ct state established,related accept
        iif $gst ct state established accept

        log prefix "FORWARD DROP POLICY: "
    }
    chain output {
        type filter hook output priority filter; policy accept;
    }
}

table ip nat
flush table ip nat
table ip nat {
    chain postrouting {
        type nat hook postrouting priority srcnat;
    }
    chain prerouting {
        type nat hook prerouting priority dstnat;

        iif $nic tcp dport 50 dnat 10.1.0.50:22;
    }
}

but you might like to keep it even more simple and deal with both directions at once

table inet filter {
    chain input {
        type filter hook input priority filter; policy accept;
    }

    chain forward {
        type filter hook forward priority filter; policy drop;
        ip protocol icmp accept
        ct state invalid log prefix "FORWARD INVALID: " drop
        ct state established,related,new accept
        log prefix "FORWARD DROP POLICY: "
    }

    chain output {
        type filter hook output priority filter; policy accept;
    }
}
table ip nat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 10.1.0.0/16 oif eth0 snat 217.19.208.157
    }

    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        iif eth0 tcp dport 50 dnat 10.1.0.50:22
    }
}

Install

see install

VRRP setup

see routing

State tracker setup

notice external cache is disabled here – this is active/passive mode meaning we absolutely need to trigger the HA scripts as shown in the keepalived setup

cd /etc/conntrackd/
mv -i conntrackd.conf conntrackd.conf.dist
grep -vE '^#|^$' conntrackd.conf.dist > conntrackd.conf.clean
grep -vE '^#|^$' conntrackd.conf.dist > conntrackd.conf
vi conntrackd.conf

General {
    HashSize 8192
    HashLimit 65535
    LockFile /run/lock/conntrack.lock
    UNIX {
        Path /var/run/conntrackd.sock
    }
    SocketBufferSize 262142
    SocketBufferSizeMaxGrown 655355
    Systemd on

    Syslog off
    LogFile on

    NetlinkBufferSize 262142
    NetlinkBufferSizeMaxGrowth 655355
    #NetlinkBufferSize 2097152
    #NetlinkBufferSizeMaxGrowth 8388608

    Filter {
        Protocol Accept {
            TCP
            #SCTP
            #UDP
            #ICMP
        }

        Address Ignore {
            IPv4_address 127.0.0.1
            IPv6_address ::1

            # do not track cluster network
            IPv4_address 10.3.3.0/24

            # note: we need to track the VIPs

            # do not track local ip
            IPv4_address 10.1.0.31

            # do not track fake subnet
            IPv4_address 10.4.4.0/24
        }

        # this does not seem to be mandatory for the fail-over to work
        #State Accept {
        #   ESTABLISHED CLOSED TIME_WAIT CLOSE_WAIT for TCP
        #}
    }
}
Sync {
    Mode FTFW {
        DisableExternalCache off
        StartupResync on
    }

    #Mode NOTRACK {
    #   DisableExternalCache off
        #       StartupResync on
    #}

    UDP {
        IPv4_address 10.3.3.31
        IPv4_Destination_Address 10.3.3.32
        Port 3780
        Interface eth2
        Checksum on
    }
}

Ready to go

keep calm, cross fingers

debian

systemctl restart conntrackd
systemctl status conntrackd # enabled

slackware

    vi /etc/rc.d/rc.inet1

    echo -n firewall and snat...
    nft -f /etc/nftables.conf && echo done || echo FAIL

    vi /etc/rc.d/rc.local

    # keepalived starts conntrackd
    sysctl net.netfilter.nf_conntrack_tcp_be_liberal=1
    /etc/rc.d/rc.keepalived start

and check that is no error in the logs

all nodes

tail -F /var/log/conntrackd.log

Acceptance

see acceptance

Additional notes

sync the configs across the farm

vi /root/sync-vrrp

#!/bin/bash

scp /etc/nftables.conf vrrp2:/etc/
#scp /root/iptables.bash vrrp2:/root/
scp /etc/keepalived/keepalived.conf vrrp2:/etc/keepalived/

sed -r '
    s/IPv4_address 10.1.0.31/IPv4_address 10.1.0.32/;
    s/IPv4_address 10.3.3.31/IPv4_address 10.3.3.32/;
    s/IPv4_Destination_Address 10.3.3.32/IPv4_Destination_Address 10.3.3.31/;
    ' /etc/conntrackd/conntrackd.conf > /etc/conntrackd/conntrackd.conf.vrrp2
scp /etc/conntrackd/conntrackd.conf.vrrp2 vrrp2:/etc/conntrackd/conntrackd.conf

check the VIPs and states all around

vi /root/STATUS-VRRP

#!/bin/bash

echo vrrp1
pgrep -a keepalived
pgrep -a conntrackd
ip addr | grep \\.157
ip addr | grep \\.254
echo INTERNAL
conntrackd -i | grep $1
echo EXTERNAL
conntrackd -e | grep $1
echo

echo vrrp2
pgrep -a keepalived
pgrep -a conntrackd
ssh vrrp2 "ip addr | grep \\.157"
ssh vrrp2 "ip addr | grep \\.254"
echo INTERNAL
ssh vrrp2 "conntrackd -i | grep $1"
echo EXTERNAL
ssh vrrp2 "conntrackd -e | grep $1"
echo

Shooting troubles

see troubles

Resources

Setting up an HA active/passive NAT gateway

Introduction

Warnings / lessons learned

Requirements

network setup

sysprep

nat setup & drop policy

Install

VRRP setup

State tracker setup

Ready to go

debian

slackware

Acceptance

Additional notes

Shooting troubles

Resources

official

manuals

third-party guides

nice examples

some theory

troubles - startup logs

troubles - ct helpers

additional tools

alternatives