Benchmarking Network Throughput



apt install iperf3

slackware, with SCTP which gets enabled at compile time without any specific flag, as long as you got the headers available,

sbopkg -i lksctp-tools
sbopkg -i iperf3

from source – same thing, about the SCTP headers,

pkill iperf3
ps aux | grep iperf3
removepkg iperf3

tar xzf 3.6.tar.gz
cd iperf-3.6/
grep -i sctp config.log
make -j16
make -j16 install


either start the server or select the nearest public one

iperf3 -s

or at startup

echo -n iperf3...
/usr/bin/iperf3 -s -D -I /var/run/ && echo done || echo FAIL


iperf3 --udp -c SERVER_ADDRESS
iperf3 --sctp -c SERVER_ADDRESS

iperf3 -P 3 -c SERVER_ADDRESS #1.88 vs 2.68 Gbit/s with balance-rr
iperf3 --udp -P 3 -c SERVER_ADDRESS #3.15 Git/s with balance-rr
iperf3 --sctp -P 3 -c SERVER_ADDRESS

use all cores

on the server,

iperf3 -s -p 5101 -D; iperf3 -s -p 5102 -D; iperf3 -s -p 5103 -D

on the client,

ksh -c "iperf3 -c $server -T s1 -p 5101 &; iperf3 -c $server -T s2 -p 5102 &; iperf3 -c $server -T s3 -p 5103" | egrep 'receiver$|sender$'
unset server

balance-rr TCP re-ordering

#net.ipv4.tcp_reordering = 3 #default
net.ipv4.tcp_reordering = 127

adjusting TCP window size

check the default values first,

sysctl net.core.wmem_max
sysctl net.core.rmem_max
sysctl net.ipv4.tcp_rmem
sysctl net.ipv4.tcp_wmem

sysctl net.ipv4.tcp_reordering

then change those,

vi /etc/sysctl.conf

net.core.wmem_max = 4194304
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 87380 4194304
#net.ipv4.tcp_reordering = 3
net.ipv4.tcp_reordering = 127

sysctl -p

iperf3 -P3 -c $server -w 8M

jumbo frames

check the current values first,

ifconfig bond0
ifconfig xenbr0
ifconfig eth0
ifconfig eth2
ifconfig eth3
#mtu 1500
#txqueuelen 1000

sysctl net.core.netdev_max_backlog

then edit your init scripts,

ifconfig bond0 mtu 9000 txqueuelen 10000
ifconfig xenbr0 mtu 9000 txqueuelen 10000
ifconfig eth0 mtu 9000 txqueuelen 10000
ifconfig eth2 mtu 9000 txqueuelen 10000
ifconfig eth3 mtu 9000 txqueuelen 10000
#ip link set

vi /etc/sysctl.conf

#net.core.netdev_max_backlog = 1000
net.core.netdev_max_backlog = 3000

sysctl -p

Note. all the NICs on that network should be switched to that MTU.


Interestingly, when binding the endpoint to their respective interface using -B, LACP returns Retr retries to 0. That looks good. As for balance-rr that does not change anything. In fact, I got more shots with 5x the amount of normal retires when using -B.


balance-rr TCP 3 streams
[SUM]   0.00-10.00  sec  2.31 GBytes  1.98 Gbits/sec  2378             sender
[SUM]   0.00-10.04  sec  2.30 GBytes  1.97 Gbits/sec                  receiver

balance-rr UDP 3 streams
[SUM]   0.00-10.00  sec  3.76 MBytes  3.16 Mbits/sec  0.000 ms  0/441 (0%)  sender
[SUM]   0.00-10.04  sec  3.76 MBytes  3.14 Mbits/sec  0.037 ms  0/441 (0%)  receiver

balance-rr SCTP 3 streams
[SUM]   0.00-10.00  sec  1.15 GBytes   990 Mbits/sec                  sender
[SUM]   0.00-10.04  sec  1.15 GBytes   986 Mbits/sec                  receiver

balance-rr TCP 3 streams after changing the TCP window size and -w 8M
[SUM]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec  3729             sender
[SUM]   0.00-10.04  sec  3.32 GBytes  2.84 Gbits/sec                  receiver


why tops 1Gbit/s per network flow?