Convergent Gateways in Da Place - part 3

part1 | part2 | part3

deal with inbound-initiated traffic

description

we now have a working poc on xen or kvm, but what about inbound-initiated traffic?

assuming DNS round-robin, the client requests arrive on varying nodes, and not necessarily the one where the service lives as a guest system. the guest systems living on different nodes have differing outbound gateways, so this would obviously bring some problems. there are two solutions for this:

1. FULL-NAT: we do not attempt to optimize the TCP responses' route and let those find the way back through the entering node — the one we are discussing here

2. CT-SYNC: we use conntrackd to synchronize the states so the answers can go right through the host gateway, just like for initiated outbound traffic in the previous pocs (but in that case, we probably need to mangle the source IP of the answer, for TCP to remain happy) — that would become part4

network requirements

we need to test three use-cases:

1. simple reverse-proxy setup e.g. for an HTTP service pointing to a precise guest system — this helps to validate the dual-ip setup on the guest bridge
2. stateful TCP connections e.g. for SSH against a precise guest system — this helps to validate FULL-NAT and CT-SYNC
3. stateless but still bi-directional UDP connections e.g. for NTP

see fuck-martinez-inbound-use-cases

FULL-NAT

on guestbr0 we differentiate node IP (e.g. 10.1.255.251) and duplicated outbound gateway IP (10.1.255.254) – and then we do full-nat instead of dnat – for the outbound packet to find its route back to where the DNAT inbound connection came from (you won’t have the issue if you are using a reverse-proxy already)

the trick is to define what destination ip you want to arp filter out, instead of using the mac address – and to carefully craft a custom subnet-wide snat rule that goes along with the port-specific dnat rules

flush ruleset

table ip nat {
    # SNAT
        chain postrouting {
                type nat hook postrouting priority srcnat;
        # node1
                ip saddr 10.5.5.0/24 oif xenbr0 snat 192.168.122.11;
                ip daddr 10.5.5.0/24 oif guestbr0 snat 10.5.5.251;
        # node2
                #ip saddr 10.5.5.0/24 oif xenbr0 snat 192.168.122.12;
                #ip daddr 10.5.5.0/24 oif guestbr0 snat 10.5.5.252;
        }

        # DNAT
        chain prerouting {
                type nat hook prerouting priority dstnat;
                iif xenbr0 tcp dport 2201 dnat 10.5.5.201:22;
                iif xenbr0 tcp dport 2202 dnat 10.5.5.202:22;
        }
}

table netdev filter {
        chain egress {
                type filter hook egress devices = { eth1.100, eth2.100 } priority -500; policy accept;
                arp saddr ip 10.5.5.254 drop
                arp daddr ip 10.5.5.254 drop
        }
}