iperf3 with --bidir gives low BW on either Tx or Rx

[divi2007]

hi Team,
Using iperf3 version 3.13 , with --bidir option results in lower BW when compared to unidirectional test on NIC with 25G speed
Need help on how to get Line rate with bidir option using iperf3
This is tried on Linux environment
Kernel version: 3.10.0-1160.53.1.el7

Bi-di test

[root@traget10-Clu-0 ~]# iperf3 -c 102.60.60.10 -P8 -i1 -t600 -p 9000 --bidir
[root@traget10-Clu-0 ~]# ./iperf3 -c 102.60.60.10 -P8 -i2 --bidir -t600 -p 6000
Connecting to host 102.60.60.10, port 6000
[ 5] local 102.60.60.11 port 35996 connected to 102.60.60.10 port 6000
[ 7] local 102.60.60.11 port 35998 connected to 102.60.60.10 port 6000
[ 9] local 102.60.60.11 port 36000 connected to 102.60.60.10 port 6000
[ 11] local 102.60.60.11 port 36002 connected to 102.60.60.10 port 6000
[ 13] local 102.60.60.11 port 36004 connected to 102.60.60.10 port 6000
[ 15] local 102.60.60.11 port 36006 connected to 102.60.60.10 port 6000
[ 17] local 102.60.60.11 port 36008 connected to 102.60.60.10 port 6000
[ 19] local 102.60.60.11 port 36010 connected to 102.60.60.10 port 6000
[ 21] local 102.60.60.11 port 36012 connected to 102.60.60.10 port 6000
[ 23] local 102.60.60.11 port 36014 connected to 102.60.60.10 port 6000
[ 25] local 102.60.60.11 port 36016 connected to 102.60.60.10 port 6000
[ 27] local 102.60.60.11 port 36018 connected to 102.60.60.10 port 6000
[ 29] local 102.60.60.11 port 36020 connected to 102.60.60.10 port 6000
[ 31] local 102.60.60.11 port 36022 connected to 102.60.60.10 port 6000
[ 33] local 102.60.60.11 port 36024 connected to 102.60.60.10 port 6000
[ 35] local 102.60.60.11 port 36026 connected to 102.60.60.10 port 6000
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][TX-C] 0.00-2.00 sec 527 MBytes 2.21 Gbits/sec 0 293 KBytes
[ 7][TX-C] 0.00-2.00 sec 519 MBytes 2.18 Gbits/sec 0 296 KBytes
[ 9][TX-C] 0.00-2.00 sec 521 MBytes 2.18 Gbits/sec 0 294 KBytes
[ 11][TX-C] 0.00-2.00 sec 520 MBytes 2.18 Gbits/sec 0 300 KBytes
[ 13][TX-C] 0.00-2.00 sec 520 MBytes 2.18 Gbits/sec 0 290 KBytes
[ 15][TX-C] 0.00-2.00 sec 528 MBytes 2.21 Gbits/sec 0 287 KBytes
[ 17][TX-C] 0.00-2.00 sec 529 MBytes 2.22 Gbits/sec 0 308 KBytes
[ 19][TX-C] 0.00-2.00 sec 523 MBytes 2.19 Gbits/sec 0 305 KBytes
[SUM][TX-C] 0.00-2.00 sec 4.09 GBytes 17.6 Gbits/sec 0
[ 21][RX-C] 0.00-2.00 sec 511 MBytes 2.14 Gbits/sec
[ 23][RX-C] 0.00-2.00 sec 511 MBytes 2.14 Gbits/sec
[ 25][RX-C] 0.00-2.00 sec 511 MBytes 2.14 Gbits/sec
[ 27][RX-C] 0.00-2.00 sec 511 MBytes 2.14 Gbits/sec
[ 29][RX-C] 0.00-2.00 sec 511 MBytes 2.14 Gbits/sec
[ 31][RX-C] 0.00-2.00 sec 511 MBytes 2.14 Gbits/sec
[ 33][RX-C] 0.00-2.00 sec 511 MBytes 2.14 Gbits/sec
[ 35][RX-C] 0.00-2.00 sec 504 MBytes 2.11 Gbits/sec
[SUM][RX-C] 0.00-2.00 sec 3.99 GBytes 17.1 Gbits/sec

---

Unidirectional test

[root@traget10-Clu-0 ~]# ./iperf3 -c 102.60.60.10 -P8 -i2 -t600 -p 6000
Connecting to host 102.60.60.10, port 6000
[ 5] local 102.60.60.11 port 36030 connected to 102.60.60.10 port 6000
[ 7] local 102.60.60.11 port 36032 connected to 102.60.60.10 port 6000
[ 9] local 102.60.60.11 port 36034 connected to 102.60.60.10 port 6000
[ 11] local 102.60.60.11 port 36036 connected to 102.60.60.10 port 6000
[ 13] local 102.60.60.11 port 36038 connected to 102.60.60.10 port 6000
[ 15] local 102.60.60.11 port 36040 connected to 102.60.60.10 port 6000
[ 17] local 102.60.60.11 port 36042 connected to 102.60.60.10 port 6000
[ 19] local 102.60.60.11 port 36044 connected to 102.60.60.10 port 6000
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-2.00 sec 688 MBytes 2.88 Gbits/sec 0 597 KBytes
[ 7] 0.00-2.00 sec 577 MBytes 2.42 Gbits/sec 0 574 KBytes
[ 9] 0.00-2.00 sec 702 MBytes 2.95 Gbits/sec 0 632 KBytes
[ 11] 0.00-2.00 sec 682 MBytes 2.86 Gbits/sec 0 609 KBytes
[ 13] 0.00-2.00 sec 602 MBytes 2.53 Gbits/sec 0 574 KBytes
[ 15] 0.00-2.00 sec 785 MBytes 3.29 Gbits/sec 0 796 KBytes
[ 17] 0.00-2.00 sec 708 MBytes 2.97 Gbits/sec 0 648 KBytes
[ 19] 0.00-2.00 sec 585 MBytes 2.46 Gbits/sec 0 574 KBytes
[SUM] 0.00-2.00 sec 5.21 GBytes 22.4 Gbits/sec 0

---

[ 5] 2.00-4.00 sec 960 MBytes 4.03 Gbits/sec 0 711 KBytes
[ 7] 2.00-4.00 sec 640 MBytes 2.68 Gbits/sec 0 602 KBytes
[ 9] 2.00-4.00 sec 482 MBytes 2.02 Gbits/sec 0 634 KBytes
[ 11] 2.00-4.00 sec 484 MBytes 2.03 Gbits/sec 0 621 KBytes
[ 13] 2.00-4.00 sec 984 MBytes 4.13 Gbits/sec 0 731 KBytes
[ 15] 2.00-4.00 sec 941 MBytes 3.95 Gbits/sec 0 809 KBytes
[ 17] 2.00-4.00 sec 482 MBytes 2.02 Gbits/sec 0 652 KBytes
[ 19] 2.00-4.00 sec 637 MBytes 2.67 Gbits/sec 0 628 KBytes
[SUM] 2.00-4.00 sec 5.48 GBytes 23.5 Gbits/sec 0

[davidBar-On]

Hi, the issue is probably caused by either iperf3 performance (e.g. CPU usage) and/or system resources limit, such as network buffers, CPU, etc. Two approaches that may help:

1. Try using --zerocopy and/or --bitrate /30 options. That may improve iperf3 and network stack performance.
2. If you are building iperf3 yourself, try using the mt branch - the multi-thread beta version where each stream runs in a separate thread.

[divi2007]

Thanks a lot David , will try these options and provide updates

[bmah888]

What kind of hardware are you using for these tests?

[divi2007]

hi Bruce, The HW used is Intel based 800 series Ethernet Adapters with icelake processor Regards, Divya

…

On Tue, Jun 27, 2023 at 3:24 AM Bruce A. Mah ***@***.***> wrote: What kind of hardware are you using for these tests? — Reply to this email directly, view it on GitHub <#1533 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALJTZNB4TTLN4KIK7WKUWPLXNIAJDANCNFSM6AAAAAAZNDRFLE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bmah888]

Hrm. There are some situations where iperf3 can be CPU-bound, particularly with the original single-threaded iperf3 implementation. I'm not real familiar with the Ice Lake family of CPUs, unfortunately, but single-threaded iperf3 does best on processors with fast cores, as it only uses one core on the CPU. The multi-threaded iperf3 can use multiple cores on a CPU, which can results in better results, even if those cores are slower.

So it's very possible that the two direction streams in your --bidir test are saturating the CPU.