iPerf throughout drops to zero.

[potpara]

NOTE: The iperf3 issue tracker is for registering bugs, enhancement
requests, or submissions of code. It is not a means for asking
questions about building or using iperf3. Those are best directed
towards the Discussions section for this project at
https://github.com/esnet/iperf/discussions
or to the iperf3 mailing list at [email protected].
A list of frequently-asked questions
regarding iperf3 can be found at http://software.es.net/iperf/faq.html.

Context:

• Version of iperf3: 3.1.17

• Hardware: , iPerf Server bare metal Kontron ME1100 Intel(R) Xeon(R) CPU D-1567 @ 2.10GHz, NIC Intel X552 10G

• Operating system (and distribution, if any): iPerf Client CentOS 7.9, iPerf Server Android LTE Mobile phone

Please note: iperf3 is supported on Linux, FreeBSD, and macOS.
Support may be provided on a best-effort basis to other UNIX-like
platforms. We cannot provide support for building and/or running
iperf3 on Windows, iOS, or Android.

• Other relevant information (for example, non-default compilers,
  libraries, cross-compiling, etc.):

Please fill out one of the "Bug Report" or "Enhancement Request"
sections, as appropriate. Note that submissions of bug fixes, new
features, etc. should be done as a pull request at
https://github.com/esnet/iperf/pulls

Bug Report

NA

• Expected Behavior
  Without TCP throughput drop to zero.
• Actual Behavior
  On iPerf server we run iperf3 -c 10.101.130.5 -i 1 -p 5202 -t 3600 -V
  On iPerf server we run iPerf3 -s -i 1 -p 5202
  Average RTT is high ~180ms since iPerf server is installed on LTE mobile handset and we don't limit send buffer size using -w. Average throughput rate is ~150Mbps. Throughput drops do zero either after 247sec or 488sec. From pcaps i see that at moment when throughput drop occurs iPerf client initiates in total 6 retransmission of packet with raw sequence number 1156349687 even it previously received TCP ACK for particular packet +169ms after first transmission. Due to 6 retransmission attempts throughput is equal to zero for around ~25sec and than it recovers. Why iPerf initiates retransmission after it received TCP ACK for packet? Is my overall understanding correct or I am doing something wrong? I attach segment of pcap with logged packets of interest.
• Steps to Reproduce
  Problem is reproducible in every attempt without exception.
• Possible Solution
  I am able to reduce RTT to ~50ms by limiting send buffer size to around ~500K but problem still exists. I switched off tcp-segmentation-offload, generic-segmentation-offload &generic-receive-offload on iPerf client but no improvement. Problem exists even if i change hardware platform for iPerf client.

Enhancement Request

• Current behavior

• Desired behavior

• Implementation notes
  sgi_iperf_filtered_tcp_segment.zip

[davidBar-On]

It seems that a main reason for the problem is that the Acks arrive to the client in blocks of several Acks (I assume the pcaps was taken on the client side). That causes a too long delay of receiving the Acks on the client side so it starts to re-transmit. It may be that on the server side (or somewhere between the server and the client) there is buffering of small messages for efficiency (Ack size is only 66 bytes). It would help to see the pcaps also on the server side.

A possible reason for the delay in receiving the Acks, when the re-transmit starts, is the LTE, which is an unreliable network. It may be that there are cases when the LTE bandwidth drops temporarily.

Regarding the "6 retransmission of packet with raw sequence number 1156349687". Since no Ack is received, TCP (and not iperf3) retries the retransmit. Note that the time period between the re-transmits doubles each time, which is the known TCP congestion mechanism for trying to not overload the network when there are problems.

[potpara]

Hi @davidBar-On thanks a lot for a quick response. Apologies if I didn't provide detailed enough input. Yes, pcaps were taken on client side - Kontron ME1100 server.

• In regards to 6 retransmissions behavior is completely understandable and expected, i.e after each unacknowledged transmission RTO increases which at the end leads to ~25sec long throughput drop.

• It is possible that TCP ACKs are buffered somewhere in the network, setup is quite complex and it's not easy task to find where, but I'll have a look.

• What I don't understand is why iPerf client initiates retransmission (in packet No 4293 in pcap) for packet with raw Seq No 1156349687 if it received TCP ACK for it (in packet No 3040 in pcap)?

Regards,
Rade

[davidBar-On]

Hi @potpara, regarding:

What I don't understand is why iPerf client initiates retransmission (in packet No 4293 in pcap) for packet with raw Seq No 1156349687 if it received TCP ACK for it (in packet No 3040 in pcap)?

The sequence number is the byte (octet) number of the TCP stream, not the packet number. In addition, the number in the packet sent is the first byte number in the packet, while the number in the ACK is the next expected byte number (i.e. all the bytes until this one were received). Therefore the retry in line 4293 starts from the first byte that was not acked.

[davidBar-On]

One more point. I noticed that your TCP doesn't use Selective ACK (SACK) - see for example here about SACK. It is important especially for unreliable networks, since without SACK if a packet is lost than all the packets sent after the lost packet will have to be re-transmitted.

This may be another partial explanation to the problem. Because SACK is not used, if the first send of Seq No 1156349687 in line 1066 was lost, all the packets sent later will not be acked. Therefore, only after the packet is re-transmitted and received, acks start to be received again. But now, because SACK was not used, all the packets sent after that packet has to be re-transmitted. (Note that probably this is not the main issue, since the packet is re-transmitted during 24 seconds before successfully received.)

[potpara]

Hi @davidBar-On . Thanks for quick responses, it was really useful to figure out what is going on.

• You were right about "What I don't understand is why iPerf client initiates retransmission (in packet No 4293 in pcap) for packet with raw Seq No 1156349687 if it received TCP ACK for it (in packet No 3040 in pcap)?" -- I was looking at wrong packet.

• Meanwhile I took tcpdump from LTE handset and it TCP ACKs are quite equally distributed in time, there aren't bursts of ACKs, so it must be something in between causing this behavior.

• About SACK, I checked SYN&SYN_ACK messages and I see that SACK is permitted by both hosts. I'll investigate it, thanks for turning my attention.

It turns out that there isn't anything unusual in iPerf behavior and it's fine for me to close this issue.

Thanks!

syn&syn_ack.zip
.