Convergent Gateways in Da Place - Part 2

part1 | part2 | part3

description

now that the basic poc works fine, we are gonna proceed with its virtualizatoin-flavored version (be it KVM or XEN, it doesn’t matter). the difference is that the user nodes are not standalone VPCS devices, but virtualized guests instead, which live on their respective virtualization type I hypervisor, while being connected to a linux bridge on the host system.

in fact, not much changes here. the layer2 architecture basically remains the same. the only differences are:

• the eth2 link gets released as the guest lives inside the node instead of a standalone vpcs
• the guestbr0 bridge now goes on a vlan
• we extend the guestbr0 to a chain of nodes using eth1.100 + eth2.100 instead of a bare cross-over on eth1 – this means we absolutely need to enable STP on that very bridge

architecture

we have a three node cluster and nodes are connected using eth1<-->eth2 direct links across each other (STP is enabled on the internal bridges using those links, namely br0 and guestbr0). the xenbr0 bridge is perimeter and without STP. yeah sorry, we call it xenbr0 no matter what kind of hypervisor we have there.

bridge name bridge id       STP enabled interfaces
guestbr0    8000.9e08bc8d4fcd   yes     eth1.100
                            eth2.100
xenbr0      8000.020000000000   no      eth0

platform setup

we are adding GNS3 device bookworm3 with a private base MAC address within

    node3   02:00:00:00:00:00

also increase the RAM on the nodes from 256 MB to 1024 MB, as those become virtualization hosts

system setup

the poc2 specifics

we are setting up the user vlan

echo 8021q >> /etc/modules
modprobe 8021q

define vlans and the entirely different layer 2 topology e.g. on node1

vi /etc/network/interfaces

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual
iface eth1.100 inet manual 
    vlan-raw-device eth1

auto eth2
iface eth2 inet manual
iface eth2.100 inet manual 
    vlan-raw-device eth2

auto xenbr0
iface xenbr0 inet static
    bridge_ports eth0
    bridge_hw eth0
    # node1
    address 192.168.122.11/24
    gateway 192.168.122.1

auto guestbr0
iface guestbr0 inet static
    bridge_ports eth1.100 eth2.100
    bridge_stp on
    bridge_hw eth1.100
    # duplicate on all three nodes
    address 10.5.5.254/24

and here’s on the other nodes

    # node2
    address 192.168.122.12/24

    # node3
    address 192.168.122.13/24

the new node

static name resolution is always good

vi /etc/hosts

192.168.122.13  bookworm3

replicate the setup from poc1 incl. SNAT on node3

table ip nat {
    chain postrouting {
                ip saddr 10.5.5.0/24 oif xenbr0 snat 192.168.122.13;
...

table netdev filter {
    chain egress {
        arp saddr ether 02:00:00:00:00:01 drop
...

shared

• setup the (now three) nodes as KVM hosts and enable nested virtualization in there
• bootstrap a guest within

acceptance

sniff all three link between the KVM hosts and arp resolve the gateway from a nested guest

from guest1 within bookworm1

ifconfig eth0 10.5.5.201/24 up
ping -c3 10.5.5.254
ping -c3 opendns.com
arp -a

we’re all good, it still works as with poc1 – there’s no echo reply from the foreign KVM host gateways

also the users can still reach each other

from guest3 within bookworm3

ifconfig eth0 10.5.5.203/24 up
ping -c3 10.5.5.201

resources

https://wiki.debian.org/NetworkConfiguration#Howto_use_vlan_.28dot1q.2C_802.1q.2C_trunk.29_.28Etch.2C_Lenny.29

https://wiki.debian.org/BridgeNetworkConnections