iperf3: Uneven distribution of TCP throughput.

[stokes246810]

I am trying to use iperf3 to characterize Raspberry Pi-4 networking performance.

The setup consists of two Raspberry Pi-4s directly connected by the builtin 1 Gbe port.

I am seeing uneven/unfair distribution of TCP throughput across parallel TCP connections.

This is what I see when running 4 parallel TCP connections:

iperf3 -c pi2 -i 0 -P4

Connection 1 gets 11 Mbits/sec
Connection 2 gets 193 Mbits/sec
Connection 3 gets 719 Mbits/sec
Connection 4 gets 18 Mbits/sec
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Total of {1-4} is 941 Mbits/sec

I was expecting each of the 4 TCP connections to get ~235 Mbits/sec. {941/4}

Why are Connections 1 and 4 getting left behind?
Why is Connection 3 getting more than 3 times the expected performance?

This seems like a bug to me.

Has anyone else seen something similar?

Any pointers will be happily followed!

Thanks!

-- steve [email protected]

[swlars]

Hi,

Does the same thing happen with longer tests? It's unusual to have that kind of asymmetry. It's possible there could be a bottleneck on the Pi somewhere. Do you see the same issue when trying the test between one of the Pi's and a PC? What about different numbers of parallel connections?

Thanks!

[stokes246810]

The length of the test does not have an impact. Its seems like a connection gets a certain priority shortly after creation, and then continues to maintain that throughput rate for the life of the connection.

If I add isolcpus=1,2,3 to /boot/cmdline.txt on both client and server, then I see a fair distribution of throughput, but a lower aggregate throughput (~ 690 Mbits/sec) versus (~ 940 Mbits/sec) with no cpu isolation.

I'm now playing around with various taskset/scheduling options to see about getting a better balance of performance.

I haven't tried with a PC yet, but it's now on the list.

The number of parallel connections doesn't have an impact on the unfairness see in the default case.

[davidBar-On]

Since isolcpus impacts the result, the issue may be related to the TCP stack implementation in the multi-processor Raspberry Pi-4 SMP Linux. To test if this is the case, I suggest that you will try to limit each stream to about 200Mbps (less than 1G/4). I.e. run iperf3 -c pi2 -i 0 -P4 -b 200M.

If this test will have even data distribution then the TCP stack may cause the issue, as when iperf3 try to send it sends to all streams. Therefore a reasonable cause of the problem is that for some reason the TCP stack prioritize the streams.