benchmark network throughput e.g. rr vs lacp

draft

warning // lessons learned

it is NOT worth tuning your load-balance algo on the server side if the switch isn’t capable of doing the same – the algo bottleneck will be there

#echo layer3+4 > /sys/class/net/bond0/bonding/xmit_hash_policy

instead, you need to emulate multiple addresses (either MAC or IP)

server

iperf3 -s &
iperf3 -s -p 5202 &
iperf3 -s -p 5203 &

client

    iperf3 -c $server
    iperf3 -c $server --udp
    iperf3 -c $server --sctp

    iperf3 -c $server --parallel 3         # 1.88 vs 2.68 Gbit/s with balance-rr
    iperf3 -c $server --parallel 3 --udp   # 3.15 Gbit/s with balance-rr
    iperf3 -c $server --parallel 3 --sctp

industrialization

iperf3 -c $server -T pipe1
iperf3 -c $server -T pipe2 -p 5102
iperf3 -c $server -T pipe3 -p 5103

grab only the result

grep -e 'receiver$|sender$'

results

    balance-rr TCP 3 streams
    [SUM]   0.00-10.00  sec  2.31 GBytes  1.98 Gbits/sec  2378             sender
    [SUM]   0.00-10.04  sec  2.30 GBytes  1.97 Gbits/sec                  receiver

    balance-rr UDP 3 streams
    [SUM]   0.00-10.00  sec  3.76 MBytes  3.16 Mbits/sec  0.000 ms  0/441 (0%)  sender
    [SUM]   0.00-10.04  sec  3.76 MBytes  3.14 Mbits/sec  0.037 ms  0/441 (0%)  receiver

    balance-rr SCTP 3 streams
    [SUM]   0.00-10.00  sec  1.15 GBytes   990 Mbits/sec                  sender
    [SUM]   0.00-10.04  sec  1.15 GBytes   986 Mbits/sec                  receiver

    balance-rr TCP 3 streams after changing the TCP window size and -w 8M
    [SUM]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec  3729             sender
    [SUM]   0.00-10.04  sec  3.32 GBytes  2.84 Gbits/sec                  receiver

troubelshooting

Interestingly, when binding the endpoint to their respective interface using -B, LACP returns Retr retries to 0. That looks good. As for balance-rr that does not change anything. In fact, I got more shots with 5x the amount of normal retires when using -B.

resources

https://www.kernel.org/doc/html/latest/networking/bonding.html ==> xmit_hash_policy

https://serverfault.com/questions/569060/link-aggregation-lacp-802-3ad-max-throughput

https://chanind.github.io/2020/05/15/network-bandwidth-stress-testing-iperf.html

PDF https://community.hpe.com/hpeb/attachments/hpeb/itrc-269/5293/1/256450.pdf

https://community.hpe.com/t5/switches-hubs-and-modems/switch-assisted-load-balancing-revisited/td-p/3338736

https://cloud.garr.it/support/kb/general/bondingPerformanceTests/

https://serverfault.com/questions/626368/why-iperf-still-reports-1gbps-performance-when-using-bonding-over-two-1gbps-adap

troubles

https://arstechnica.com/civis/viewtopic.php?t=1357153

why 1gbit/s per network flow

https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

http://louwrentius.com/linux-network-interface-bonding-trunking-or-how-to-get-beyond-1-gbs.html

http://qnapsupport.net/load-balance-setup-which-is-the-best-port-trunking-option/

https://serverfault.com/questions/626368/why-iperf-still-reports-1gbps-performance-when-using-bonding-over-two-1gbps-adap

https://askubuntu.com/questions/431775/why-bonding-does-not-improve-network-speed

https://help.onapp.com/hc/en-us/articles/222048108-LACP-4-NICS-bond-Mode-4-traffic-goes-through-1-NIC-instead-of-4