Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iperf3: error - unable to receive parameters from client: Bad file descriptor #1808

Open
coinhu1995 opened this issue Dec 12, 2024 · 11 comments
Open

iperf3: error - unable to receive parameters from client: Bad file descriptor #1808

coinhu1995 opened this issue Dec 12, 2024 · 11 comments

Comments

@coinhu1995
Copy link

coinhu1995 commented Dec 12, 2024 •
edited
Loading

Context

Got error when measuring bandwidth between macos (local) and ubuntu (AWS EC2 with public IP)

  • Version of iperf3:
    Client and server are the same: iperf 3.17.1+

  • Hardware:

  • Client: Apple macmini M1 - macos Sonoma 14.2.1
  • Server: AWS EC2 - Ubuntu 18.04.5 LTS

Bug Report

  • Steps to Reproduce
    Server log
ubuntu@ip-172-31-0-5:~/tmp/iperf-master$ iperf3 -s -B 0.0.0.0 -V -d
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)


State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
warning: JSON size of data read does not correspond to offered length
All threads stopped
iperf3: error - unable to receive parameters from client: Bad file descriptor
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client log

user@name iperf-master % iperf3 -c 52.77.179.243 -V -d
iperf 3.17.1+
Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Reading new State from the Server - current state is 0-Test reset
State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.17.1+"
}
Time: Thu, 12 Dec 2024 07:58:14 UTC
Connecting to host 52.77.179.243, port 5201
      Cookie: jnfff5macsu2es3oawqxbguci47gdwncu576
      TCP MSS: 1388 (default)
Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
All threads stopped
State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe
  • Possible Solution
    N/A
@davidBar-On
Copy link
Contributor

warning: JSON size of data read does not correspond to offered length

To better understand this issue, I created a branch in my private iperf3 fork which print a debug message when this problem happens. The message starts with "*** TEMP DEBUG - Parameters JSON: ..." and shows the expected Parameters JSON length and the actual length received.

Can you rebuild iperf3 for the server from this branch, run it and send the debug message output?

@coinhu1995
Copy link
Author

coinhu1995 commented Dec 13, 2024 •
edited
Loading

@davidBar-On Here is the log

iperf-issue-1808-json-parameters-length-mismatch % iperf3 -c 52.77.179.243 -V -d
iperf 3.17.1+
Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Reading new State from the Server - current state is 0-Test reset
State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.17.1+"
}
Time: Fri, 13 Dec 2024 02:05:28 UTC
Connecting to host 52.77.179.243, port 5201
      Cookie: udgeliqdmeiobpcu7jub7d573lmmogzpeg5j
      TCP MSS: 1388 (default)
Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
All threads stopped
State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe

~/tmp/iperf-issue-1808-json-parameters-length-mismatch$ iperf3 -s -B 0.0.0.0 -V -d
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG - Parameters JSON: expected 127 bytes but received 0 bytes;
warning: JSON size of data read does not correspond to offered length
All threads stopped
iperf3: error - unable to receive parameters from client: Bad file descriptor
iperf 3.17.1+
Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

@coinhu1995
Copy link
Author

coinhu1995 commented Dec 13, 2024 •
edited
Loading

@davidBar-On Additional information: I'm on the corporate network. Are there any chances that the corporate network could prevent the client from sending packets to the server?

@davidBar-On
Copy link
Contributor

davidBar-On commented Dec 13, 2024 •
edited
Loading

I'm on the corporate network. Are there any chances that the corporate network could prevent the client from sending packets to the server?

I don't know. Usually different UDP/TCP ports are blocked, but in this case some messages were already exchanged between the client and server before the error happened. If there is a content filter (which I am not sure there is), then maybe the JSON contents were filtered.

*** TEMP DEBUG - Parameters JSON: expected 127 bytes but received 0 bytes;

Receiving 0 bytes can happen either because of timeout or because the the session was closed (by the corporate network)? To better understand what happens, I added few additional debug messages (same branch on my repository). Can you build the server from the new code and send the results?

Please add the --timestamps option to both Client and Server. If the error is because of timeout, probably because the network filters the message, then the log messages timestamps may indicate that.

@coinhu1995
Copy link
Author

coinhu1995 commented Dec 16, 2024 •
edited
Loading

@davidBar-On Here is the log

Server

$ iperf3 -s -B 0.0.0.0 -V -d --timestamps
Mon Dec 16 02:50:12 2024 iperf 3.17.1+
Mon Dec 16 02:50:12 2024 Mon Dec 16 02:50:12 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Mon Dec 16 02:50:12 2024 -----------------------------------------------------------
Mon Dec 16 02:50:12 2024 Server listening on 5201 (test #1)
Mon Dec 16 02:50:12 2024 -----------------------------------------------------------
Mon Dec 16 02:50:12 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
*** TEMP DEBUG: Nread() read() returned 37, errno=0;
Mon Dec 16 02:50:49 2024 State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG: Nread() read() returned 4, errno=0;
*** TEMP DEBUG: Nread() select() 1 returned 0 (timeout), errno=0;
warning: JSON size of data read does not correspond to offered length - expected 127 bytes but received 0 bytes; errno=0
Mon Dec 16 02:50:59 2024 All threads stopped
Mon Dec 16 02:50:59 2024 iperf3: error - unable to receive parameters from client: Bad file descriptor
Mon Dec 16 02:50:59 2024 iperf 3.17.1+
Mon Dec 16 02:50:59 2024 Mon Dec 16 02:50:59 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Mon Dec 16 02:50:59 2024 -----------------------------------------------------------
Mon Dec 16 02:50:59 2024 Server listening on 5201 (test #2)
Mon Dec 16 02:50:59 2024 -----------------------------------------------------------
Mon Dec 16 02:50:59 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client

iperf3 -c 52.77.179.243 -V -d --timestamps
Mon Dec 16 09:50:48 2024 iperf 3.17.1+
Mon Dec 16 09:50:48 2024 Mon Dec 16 09:50:48 2024 Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Mon Dec 16 09:50:49 2024 Reading new State from the Server - current state is 0-Test reset
Mon Dec 16 09:50:49 2024 State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.17.1+"
}
Mon Dec 16 09:50:49 2024 Time: Mon, 16 Dec 2024 02:50:49 UTC
Mon Dec 16 09:50:49 2024 Connecting to host 52.77.179.243, port 5201
Mon Dec 16 09:50:49 2024       Cookie: 6t5wnbqf3avcmt6kjmprzox65c2dghxrdbjp
Mon Dec 16 09:50:49 2024       TCP MSS: 1388 (default)

Mon Dec 16 09:57:23 2024 Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
Mon Dec 16 09:57:23 2024 All threads stopped
Mon Dec 16 09:57:23 2024 State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
Mon Dec 16 09:57:23 2024 iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe

@davidBar-On
Copy link
Contributor

Somehow the JSON parameters message does not arrive to the server, so the read times out (note the 10 seconds difference between the client send time 50:49 and the server error at 50:59). It is probably because of some filtering in the corporate network.

I don't know what this filtering is, but one guess is that it filters text/JSON. To check that I have changed the sent JSON to zeros. If JSON is filtered, then the message will arrive to the server (but it will fail since this is not JSON). Can you rebuild and run with the new version in the branch to check?

@coinhu1995
Copy link
Author

@davidBar-On Here is the update

Server

iperf3 -s -B 0.0.0.0 -V -d --timestamps
Tue Dec 17 02:16:12 2024 iperf 3.18
Tue Dec 17 02:16:12 2024 Tue Dec 17 02:16:12 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 02:16:12 2024 -----------------------------------------------------------
Tue Dec 17 02:16:12 2024 Server listening on 5201 (test #1)
Tue Dec 17 02:16:12 2024 -----------------------------------------------------------
Tue Dec 17 02:16:12 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
*** TEMP DEBUG: Nread() read() returned 37, errno=0;
Tue Dec 17 02:16:24 2024 State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG: Nread() read() returned 4, errno=0;
*** TEMP DEBUG: Nread() select() 1 returned 0 (timeout), errno=0;
warning: JSON size of data read does not correspond to offered length - expected 124 bytes but received 0 bytes; errno=0
Tue Dec 17 02:16:34 2024 All threads stopped
Tue Dec 17 02:16:34 2024 iperf3: error - unable to receive parameters from client: Bad file descriptor
Tue Dec 17 02:16:34 2024 iperf 3.18
Tue Dec 17 02:16:34 2024 Tue Dec 17 02:16:34 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 02:16:34 2024 -----------------------------------------------------------
Tue Dec 17 02:16:34 2024 Server listening on 5201 (test #2)
Tue Dec 17 02:16:34 2024 -----------------------------------------------------------
Tue Dec 17 02:16:34 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client

iperf3 -c 52.77.179.243 -V -d --timestamps
Tue Dec 17 09:16:24 2024 iperf 3.18
Tue Dec 17 09:16:24 2024 Tue Dec 17 09:16:24 2024 Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Tue Dec 17 09:16:24 2024 Reading new State from the Server - current state is 0-Test reset
Tue Dec 17 09:16:24 2024 State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.18"
}
Tue Dec 17 09:16:24 2024 Time: Tue, 17 Dec 2024 02:16:24 UTC
Tue Dec 17 09:16:24 2024 Connecting to host 52.77.179.243, port 5201
Tue Dec 17 09:16:24 2024       Cookie: pqiaqxlgn7eajr6tbif6ozrb737khtbbokhn
Tue Dec 17 09:16:24 2024       TCP MSS: 1388 (default)
Tue Dec 17 09:17:17 2024 Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
Tue Dec 17 09:17:17 2024 All threads stopped
Tue Dec 17 09:17:17 2024 State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
Tue Dec 17 09:17:17 2024 iperf3: error - unable to send control message - port may not be available, the other side may have stopped running, etc.: Broken pipe

@davidBar-On
Copy link
Contributor

@coinhu1995, sorry, I somehow didn't push the last debug changes to github. I did it now. Can you rebuild and run again?

@coinhu1995
Copy link
Author

@davidBar-On

Server

iperf3 -s -B 0.0.0.0 -V -d --timestamps
Tue Dec 17 10:08:18 2024 iperf 3.18
Tue Dec 17 10:08:18 2024 Tue Dec 17 10:08:18 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 10:08:18 2024 -----------------------------------------------------------
Tue Dec 17 10:08:18 2024 Server listening on 5201 (test #1)
Tue Dec 17 10:08:18 2024 -----------------------------------------------------------
Tue Dec 17 10:08:18 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)
*** TEMP DEBUG: Nread() read() returned 37, errno=0;
Tue Dec 17 10:08:29 2024 State change: State set to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange (from 15-IPERF_START - waiting for a new test)
*** TEMP DEBUG: Nread() read() returned 4, errno=0;
*** TEMP DEBUG: Nread() read() returned 124, errno=0;
Tue Dec 17 10:08:29 2024 All threads stopped
Tue Dec 17 10:08:29 2024 iperf3: error - unable to receive parameters from client: Bad file descriptor
Tue Dec 17 10:08:29 2024 iperf 3.18
Tue Dec 17 10:08:29 2024 Tue Dec 17 10:08:29 2024 Linux ip-172-31-0-5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64
Tue Dec 17 10:08:29 2024 -----------------------------------------------------------
Tue Dec 17 10:08:29 2024 Server listening on 5201 (test #2)
Tue Dec 17 10:08:29 2024 -----------------------------------------------------------
Tue Dec 17 10:08:29 2024 State change: State set to 15-IPERF_START - waiting for a new test (from 0-Test reset)

Client

iperf3 -c 52.77.179.243 -V -d --timestamps                                                                           
Tue Dec 17 17:08:29 2024 iperf 3.18
Tue Dec 17 17:08:29 2024 Tue Dec 17 17:08:29 2024 Darwin CPP00114445A 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103 x86_64
Control connection MSS 1388
Tue Dec 17 17:08:29 2024 Reading new State from the Server - current state is 0-Test reset
Tue Dec 17 17:08:29 2024 State change: client received and changed State to 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
send_parameters:
{
	"tcp":	true,
	"omit":	0,
	"time":	10,
	"num":	0,
	"blockcount":	0,
	"parallel":	1,
	"len":	131072,
	"pacing_timer":	1000,
	"client_version":	"3.18"
}
*** TEMP DEBUG: set JSON string to zeros - strlen(str)=0;
*** TEMP DEBUG: JSON_write() length=124, return code=0;
Tue Dec 17 17:08:29 2024 Time: Tue, 17 Dec 2024 10:08:29 UTC
Tue Dec 17 17:08:29 2024 Connecting to host 52.77.179.243, port 5201
Tue Dec 17 17:08:29 2024       Cookie: jlkp5fpm6zywxzcuyjdjcna2q7rmljrhebzn
Tue Dec 17 17:08:29 2024       TCP MSS: 1388 (default)
Tue Dec 17 17:08:29 2024 Reading new State from the Server - current state is 9-PARAM_EXCHANGE - Client to Server Parameters Exchange
Tue Dec 17 17:08:29 2024 All threads stopped
Tue Dec 17 17:08:29 2024 State change: State set to 16-IPERF_DONE (from 9-PARAM_EXCHANGE - Client to Server Parameters Exchange)
Tue Dec 17 17:08:29 2024 iperf3: error - control socket has closed unexpectedly

@davidBar-On
Copy link
Contributor

@coinhu1995, thanks for re-running. It is almost clear now that this is a corporate network issue. Either it filters JSON, filters text messages or something else. This is because in the last test there was no timeout and all the full 124 bytes message was received by the server. There is still small probability that this is an iperf3 issue, although I cannot guess what that might be.

I suggest to ask corporate network administrators about this. If you want to evaluate this further, you can try sending different messages instead of the parameters string, to see if they are received by the server. This can be done with my branch code, in the JSON_write() function in iperf_api.c. I suggest either or both:

  • Try to change the message contents to see its effect, by changing the memset(str, 0, hsize) I inserted to set the message to zeroes. E.g. change it to memset(str, 'X', hsize) to see if they are received by the server.
  • Try sending different JSON or textual messages to see if they are received by the server (print str after str = cJSON_PrintUnformatted(json) to see how the sent JSON looks like). This is by replaceing the Nwrite(fd, str, hsize, Ptcp). For example, send emty JSON by Nwrite(fd, "{}", strlen("{}"), Ptcp), or send subset of the JSON to see if and which filed(s) may cause the problem - e.g. str="{\"tcp\":true,\"omit\":0,\"client_version\":\"3.18\"}"; Nwrite(fd, str, strlen(str), Ptcp).

If you get any interesting results, either from the tests or about the corporate network, please share.

@coinhu1995
Copy link
Author

@davidBar-On thank you. I will check with network admin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@coinhu1995 @davidBar-On