Setting up a HA active/passive NAT gateway

this is a spin-off from the fuck-martinez project, which proved to be harder than expected

Introduction

Let’s proceed without the DIY ARP ado and keep it simple with a simple group of VIPs, one on the outside and one on the inside.

Connection tracker is there to keep up with the tcp states during fail-over.

Requirements

make the HA script available

    cp /usr/doc/conntrack-tools-1.4.6/sync/primary-backup.sh /etc/conntrackd/

eventually tune it to KISS more — I had to add a delay before the flush (see troubleshooting section below)

#!/bin/bash

# (C) 2006-2011 by Pablo Neira Ayuso <pablo@netfilter.org>
# GPL2+

case "$1" in
  primary)
        /usr/sbin/conntrackd -c || logger "ERROR: failed to invoke conntrackd -c"
        sleep 1
        /usr/sbin/conntrackd -f || logger "ERROR: failed to invoke conntrackd -f"
        /usr/sbin/conntrackd -R || logger "ERROR: failed to invoke conntrackd -R"
        /usr/sbin/conntrackd -B || logger "ERROR: failed to invoke conntrackd -B"
        ;;
  backup)
        /usr/sbin/conntrackd -s || /usr/sbin/conntrackd -d
        /usr/sbin/conntrackd -t || logger "ERROR: failed to invoke conntrackd -t"
        /usr/sbin/conntrackd -n || logger "ERROR: failed to invoke conntrackd -n"
        ;;
  fault)
        /usr/sbin/conntrackd -t || logger "ERROR: failed to invoke conntrackd -t"
        ;;
  *)
        logger "ERROR: unknown state transition"
        echo "Usage: primary-backup.sh {primary|backup|fault}"
        exit 1
        ;;
esac

exit 0

also make sure you’ve filtering completely without sending TCP RSTs to the client (use DROP not REJECT). in case you cannot use DROP as a default policy but prefer to filter a specific interface only, here’s a trick.

table inet filter {
        chain input {
                type filter hook input priority filter; policy accept;

        ...

        iif $nic ct state established,related accept
        iif $nic drop

and disable tcp window tracking

    echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
    cat /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal

Keepalived setup

global_defs {
        max_auto_priority -1

        notification_email {
                support@nethence.com
        }

        notification_email_from support@nethence.com
        checker_log_all_failures
        default_interface xenbr0

        # isolate only for notify
        # (need root for conntrackd)
        #enable_script_security
        #script_user keepalive keepalive
}

vrrp_sync_group nat {
        group {
                pmr-vip1
                guest-vip1
        }

        notify_master   "/etc/conntrackd/primary-backup.sh primary"
        notify_backup   "/etc/conntrackd/primary-backup.sh backup"
        notify_fault    "/etc/conntrackd/primary-backup.sh fault"
        #notify "/var/tmp/notify.bash"
}

vrrp_instance pmr-vip1 {
        state BACKUP
        interface xenbr0
        virtual_router_id 1
        priority 1
        advert_int 1

        virtual_ipaddress {
                217.19.208.157/29
        }

        # default route remains anyhow
        notify "/var/tmp/notify.bash"
}

vrrp_instance guest-vip1 {
        state BACKUP
        interface guestbr0
        virtual_router_id 2
        priority 1
        advert_int 1

        virtual_ipaddress {
                10.1.255.254/16
        }

        notify "/var/tmp/notify.bash"
}

Connection tracker

it seems to be we need to ignore only the static ips from every node member, but NOT THE VIPs (to be confirmed). notice external cache is enabled here, so we absolutely need to trigger the HA scripts as shown above.

Sync {
        Mode FTFW {
                DisableExternalCache Off
                CommitTimeout 1800
                PurgeTimeout 5
        }

        UDP Default {
                IPv4_address @@@myip@@@
                IPv4_Destination_Address @@@otherip1@@@
                IPv4_Destination_Address @@@otherip2@@@
                Port 3780
                Interface br0
                #SndSocketBuffer 1249280
                #RcvSocketBuffer 1249280
                SndSocketBuffer 24985600
                RcvSocketBuffer 24985600
                Checksum on
        }
}

General {
        Systemd off
        HashSize 8192
        HashLimit 65535
        LogFile on
        Syslog on
        LockFile /var/lock/conntrack.lock

        UNIX {
                Path /var/run/conntrackd.ctl
        }

        NetlinkBufferSize 262142
        NetlinkBufferSizeMaxGrowth 655355

        Filter {
                Protocol Accept {
                        TCP
                }

                Address Ignore {
                        IPv4_address 127.0.0.1
                        IPv6_address ::1

                        IPv4_address 10.3.3.1
                        IPv4_address 10.3.3.2
                        IPv4_address 10.3.3.3

                        # vip
                        #IPv4_address 10.1.255.254

                        IPv4_address 10.1.255.251
                        IPv4_address 10.1.255.252
                        IPv4_address 10.1.255.253

                        # vip
                        #IPv4_address 217.19.208.157
                        #IPv4_address 217.19.208.158

                        IPv4_address 217.19.208.154
                        IPv4_address 217.19.208.155
                        IPv4_address 217.19.208.156
                }

                State Accept {
                        ESTABLISHED CLOSED TIME_WAIT CLOSE_WAIT for TCP
                }
        }
}

Ready to go

vi /etc/rc.d/rc.local

# self-verbose
/etc/rc.d/rc.keepalived start && echo OK || echo FAIL

echo -n starting conntrackd ...
conntrackd -d && echo done || echo FAIL

Acceptance testing

SSH + ping

SSH to some NATed server and ping continuously some machine of yours on the public network e.g.

    ping nethence.com

then proceed with the fail-over by killing keepalived on the node that owns the VIPs.

SSH + ISO download

SSH to some NATed server and grab a large ISO file from a close mirror e.g.

    wget https://mirror.ihost.md/slackware/slackware-iso/slackware64-15.0-iso/slackware64-15.0-install-dvd.iso

then proceed with the fail-over by killing keepalived on the node that owns the VIPs.

Troubleshooting

ERROR: failed to invoke conntrackd -c

==> do not drop the privileges in keepalived.conf

conntrack-tools[15110]: ignoring flush command, commit still in progress

==> dirty workaround: added delay of 1 second in the HA script

Resources

The netfilter.org “conntrack-tools” project https://www.nftables.org/projects/conntrack-tools/index.html

Connection Tracking System https://wiki.nftables.org/wiki-nftables/index.php/Connection_Tracking_System

netfilter connection tracking user-space daemon https://www.mankier.com/8/conntrackd

troubles

Nftables issue with ftp firewall and conntrack https://serverfault.com/questions/880551/nftables-issue-with-ftp-firewall-and-conntrack

moar

Netstat-nat https://www.tweegy.nl/projects/netstat-nat/index.html

15.0 > Network > netstat-nat (1.4.10) https://slackbuilds.org/repository/15.0/network/netstat-nat/

Setting packet connection tracking metainformation https://wiki.nftables.org/wiki-nftables/index.php/Setting_packet_connection_tracking_metainformation

Matching connection tracking stateful metainformation https://wiki.nftables.org/wiki-nftables/index.php/Matching_connection_tracking_stateful_metainformation

Conntrack helpers https://wiki.nftables.org/wiki-nftables/index.php/Conntrack_helpers

10.3. Explicit matches https://www.linuxtopia.org/Linux_Firewall_iptables/x2682.html

Setting up a HA active/passive NAT gateway

Introduction

Requirements

Keepalived setup

Connection tracker

Ready to go

Acceptance testing

SSH + ping

SSH + ISO download

Troubleshooting

Resources

official guides

other guides

some theory

nft samples

ha scripts

conntrack

troubles

moar