keepalived | conntrackd | fuck-martinez
this is a spin-off from the fuck-martinez project, which proved to be harder than expected
Let’s proceed without the DIY ARP ado and keep it simple with a simple group of VIPs, one on the outside and one on the inside.
Connection tracker is there to keep up with the tcp states during fail-over.
make the HA script available
cp /usr/doc/conntrack-tools-1.4.6/sync/primary-backup.sh /etc/conntrackd/
eventually tune it to KISS more — I had to add a delay before the flush (see troubleshooting section below)
#!/bin/bash # (C) 2006-2011 by Pablo Neira Ayuso <pablo@netfilter.org> # GPL2+ case "$1" in primary) /usr/sbin/conntrackd -c || logger "ERROR: failed to invoke conntrackd -c" sleep 1 /usr/sbin/conntrackd -f || logger "ERROR: failed to invoke conntrackd -f" /usr/sbin/conntrackd -R || logger "ERROR: failed to invoke conntrackd -R" /usr/sbin/conntrackd -B || logger "ERROR: failed to invoke conntrackd -B" ;; backup) /usr/sbin/conntrackd -s || /usr/sbin/conntrackd -d /usr/sbin/conntrackd -t || logger "ERROR: failed to invoke conntrackd -t" /usr/sbin/conntrackd -n || logger "ERROR: failed to invoke conntrackd -n" ;; fault) /usr/sbin/conntrackd -t || logger "ERROR: failed to invoke conntrackd -t" ;; *) logger "ERROR: unknown state transition" echo "Usage: primary-backup.sh {primary|backup|fault}" exit 1 ;; esac exit 0
also make sure you’ve filtering completely without sending TCP RSTs to the client (use DROP not REJECT). in case you cannot use DROP as a default policy but prefer to filter a specific interface only, here’s a trick.
table inet filter { chain input { type filter hook input priority filter; policy accept; ... iif $nic ct state established,related accept iif $nic drop
and disable tcp window tracking
echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal cat /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
global_defs { max_auto_priority -1 notification_email { support@nethence.com } notification_email_from support@nethence.com checker_log_all_failures default_interface xenbr0 # isolate only for notify # (need root for conntrackd) #enable_script_security #script_user keepalive keepalive } vrrp_sync_group nat { group { pmr-vip1 guest-vip1 } notify_master "/etc/conntrackd/primary-backup.sh primary" notify_backup "/etc/conntrackd/primary-backup.sh backup" notify_fault "/etc/conntrackd/primary-backup.sh fault" #notify "/var/tmp/notify.bash" } vrrp_instance pmr-vip1 { state BACKUP interface xenbr0 virtual_router_id 1 priority 1 advert_int 1 virtual_ipaddress { 217.19.208.157/29 } # default route remains anyhow notify "/var/tmp/notify.bash" } vrrp_instance guest-vip1 { state BACKUP interface guestbr0 virtual_router_id 2 priority 1 advert_int 1 virtual_ipaddress { 10.1.255.254/16 } notify "/var/tmp/notify.bash" }
it seems to be we need to ignore only the static ips from every node member, but NOT THE VIPs (to be confirmed). notice external cache is enabled here, so we absolutely need to trigger the HA scripts as shown above.
Sync { Mode FTFW { DisableExternalCache Off CommitTimeout 1800 PurgeTimeout 5 } UDP Default { IPv4_address @@@myip@@@ IPv4_Destination_Address @@@otherip1@@@ IPv4_Destination_Address @@@otherip2@@@ Port 3780 Interface br0 #SndSocketBuffer 1249280 #RcvSocketBuffer 1249280 SndSocketBuffer 24985600 RcvSocketBuffer 24985600 Checksum on } } General { Systemd off HashSize 8192 HashLimit 65535 LogFile on Syslog on LockFile /var/lock/conntrack.lock UNIX { Path /var/run/conntrackd.ctl } NetlinkBufferSize 262142 NetlinkBufferSizeMaxGrowth 655355 Filter { Protocol Accept { TCP } Address Ignore { IPv4_address 127.0.0.1 IPv6_address ::1 IPv4_address 10.3.3.1 IPv4_address 10.3.3.2 IPv4_address 10.3.3.3 # vip #IPv4_address 10.1.255.254 IPv4_address 10.1.255.251 IPv4_address 10.1.255.252 IPv4_address 10.1.255.253 # vip #IPv4_address 217.19.208.157 #IPv4_address 217.19.208.158 IPv4_address 217.19.208.154 IPv4_address 217.19.208.155 IPv4_address 217.19.208.156 } State Accept { ESTABLISHED CLOSED TIME_WAIT CLOSE_WAIT for TCP } } }
vi /etc/rc.d/rc.local # self-verbose /etc/rc.d/rc.keepalived start && echo OK || echo FAIL echo -n starting conntrackd ... conntrackd -d && echo done || echo FAIL
SSH to some NATed server and ping continuously some machine of yours on the public network e.g.
ping nethence.com
then proceed with the fail-over by killing keepalived on the node that owns the VIPs.
SSH to some NATed server and grab a large ISO file from a close mirror e.g.
wget https://mirror.ihost.md/slackware/slackware-iso/slackware64-15.0-iso/slackware64-15.0-install-dvd.iso
then proceed with the fail-over by killing keepalived on the node that owns the VIPs.
ERROR: failed to invoke conntrackd -c
==> do not drop the privileges in keepalived.conf
conntrack-tools[15110]: ignoring flush command, commit still in progress
==> dirty workaround: added delay of 1 second in the HA script
The netfilter.org “conntrack-tools” project https://www.nftables.org/projects/conntrack-tools/index.html
Connection Tracking System https://wiki.nftables.org/wiki-nftables/index.php/Connection_Tracking_System
Documentation http://conntrack-tools.netfilter.org/support.html
The conntrack-tools user manual http://conntrack-tools.netfilter.org/manual.html
Test Case http://conntrack-tools.netfilter.org/testcase.html
Connection tracking (conntrack) - Part 3: State and Examples https://thermalcircle.de/doku.php?id=blog:linux:connection_tracking_3_state_and_examples
PDF // Conntrack-tools: connection tracking userspace tools http://conntrack-tools.netfilter.org/files/conntrackd-nfws.pdf
PDF // Demystifying cluster-based fault-tolerant Firewalls https://web.archive.org/web/20160512183228/http://1984.lsi.us.es/~pablo/docs/intcomp09.pdf
Netfilter’s connection tracking system https://people.netfilter.org/pablo/docs/login.pdf
Explaining My Configs: nftables https://stosb.com/blog/explaining-my-configs-nftables/
Keepalived Check and Notify Scripts https://tobru.ch/keepalived-check-and-notify-scripts/
Package conntrack-tools https://www.mankier.com/package/conntrack-tools
netfilter connection tracking user-space daemon https://www.mankier.com/8/conntrackd
Nftables issue with ftp firewall and conntrack https://serverfault.com/questions/880551/nftables-issue-with-ftp-firewall-and-conntrack
Netstat-nat https://www.tweegy.nl/projects/netstat-nat/index.html
15.0 > Network > netstat-nat (1.4.10) https://slackbuilds.org/repository/15.0/network/netstat-nat/
Setting packet connection tracking metainformation https://wiki.nftables.org/wiki-nftables/index.php/Setting_packet_connection_tracking_metainformation
Matching connection tracking stateful metainformation https://wiki.nftables.org/wiki-nftables/index.php/Matching_connection_tracking_stateful_metainformation
Conntrack helpers https://wiki.nftables.org/wiki-nftables/index.php/Conntrack_helpers
10.3. Explicit matches https://www.linuxtopia.org/Linux_Firewall_iptables/x2682.html