Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iperf3 - failed to open /dev/urandom #1333

Open
togoetha opened this issue Sep 17, 2024 · 3 comments
Open

iperf3 - failed to open /dev/urandom #1333

togoetha opened this issue Sep 17, 2024 · 3 comments

Comments

@togoetha
Copy link

togoetha commented Sep 17, 2024 •
edited
Loading

Doing some lightweight throughput tests for a Kubernetes network architecture that supports OSv, but running into an odd problem where iperf3 can't find /dev/urandom. The images are built with Capstan, most basic possible config requiring osv.iperf3 and start command /tools/iperf3 -s .

The image starts fine, but as soon as any client connects I get the following output:

sudo /usr/bin/qemu-system-x86_64 -vnc unix:/root/.capstan/instances/qemu/vmtest_debug_app/vnc-domain-socket    -m 1024    -smp 1    -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0    -drive file=/root/.capstan/instances/qemu/vmtest_debug_app/disk.qcow2,if=none,id=hd0,aio=threads,cache=none    -chardev stdio,mux=on,id=stdio,signal=off    -device isa-serial,chardev=stdio    -netdev tap,id=hn0,ifname=vtap3391,script=no,downscript=no   -device virtio-net-pci,netdev=hn0,id=nic1,mac=46:0c:3b:7f:c0:c7    -chardev socket,id=charmonitor,path=/root/.capstan/instances/qemu/vmtest_debug_app/osv.monitor,server=on,wait=off    -mon chardev=charmonitor,id=monitor,mode=control    -enable-kvm    -cpu host,+x2apic    -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/shm,share=on    -numa node,memdev=mem  
OSv v0.57.0-183-g9c152813
1 CPUs detected
Firmware vendor: Unknown
bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
vga: Add VGA device instance
eth0: ethernet address: 46:0c:3b:7f:c0:c7
devfs: created device vblk0.1 for a partition at offset:6291456 with size:10731126784
virtio-blk: Add blk device instances 0 as vblk0, devsize=10737418240
random: intel drng, rdrand registered as a source.
random: <Software, Yarrow> initialized
VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
BSD shrinker: event handler list found: 0x600000fed900
	BSD shrinker found: 1
BSD shrinker: unlocked, running
eth0: 10.184.66.26
Booted up in 136.05 ms
Cmdline: /tools/iperf3 -s
Running from /init/40-kubernetes: /kubernetes.so !

kubernetes: failed to open /run/kubernetes/configmaps.
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
random: device unblocked.
Accepted connection from 10.184.66.16, port 53268
iperf3: error - failed to open /dev/urandom: No such file or directory

program exited with status 1
VFS: unmounting /dev
VFS: unmounting /proc
Warning: unmount_rootfs: failed to unmount /proc, error = Invalid argument
VFS: unmounting /
Powering off.

From what I can see in the code OSv should definitely be creating the urandom device.
Is this an image/OSv issue or is the Capstan iperf package/app no longer the way to go and should I just compile iperf3 straight from source instead?

Edit: looking at the iperf app, seems like it's just a pie recompile of default iperf3. I'll just modify iperf3 source to work around this in the meantime, although that's a bit hacky.
@wkozaczuk
Copy link
Collaborator

wkozaczuk commented Sep 18, 2024 •
edited
Loading

I have just built an image with the iperf3 from apps and it seems not to work:

OSv v0.57.0-183-g9c152813
1 CPUs detected
Firmware vendor: SeaBIOS
: ignoring missing symbol fsgsbase_avail
bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
vga: Add VGA device instance
eth0: ethernet address: 52:54:00:12:34:56
devfs: created device vblk0.1 for a partition at offset:6291456 with size:530579456
virtio-blk: Add blk device instances 0 as vblk0, devsize=536870912
random: virtio-rng registered as a source.
random: intel drng, rdrand registered as a source.
random: <Software, Yarrow> initialized
VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
BSD shrinker: event handler list found: 0x600000f45800
	BSD shrinker found: 1
BSD shrinker: unlocked, running
[I/17 dhcp]: Broadcasting DHCPDISCOVER message with xid: [2035976780]
[I/17 dhcp]: Waiting for IP...
[I/191 dhcp]: Received DHCPOFFER message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15
[I/191 dhcp]: Broadcasting DHCPREQUEST message with xid: [2035976780] to SELECT offered IP: 192.168.122.15
[I/191 dhcp]: Received DHCPACK message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15
[I/191 dhcp]: Server acknowledged IP 192.168.122.15 for interface eth0random: device unblocked.
 with time to lease in seconds: 86400
eth0: 192.168.122.15
[I/191 dhcp]: Configuring eth0: ip 192.168.122.15 subnet mask 255.255.255.0 gateway 192.168.122.1 MTU 1500
Booted up in 303.56 ms
Cmdline: /tools/iperf3 -s
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.122.1, port 38372
[  5] local 192.168.122.15 port 5201 connected to 192.168.122.1 port 38386
iperf3: getsockopt - Invalid argument
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   585 MBytes  4.91 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   1.00-2.00   sec   661 MBytes  5.55 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   2.00-3.00   sec   652 MBytes  5.47 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   3.00-4.00   sec   661 MBytes  5.54 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   4.00-5.00   sec   652 MBytes  5.47 Gbits/sec                  
iperf3: getsockopt - Invalid argument
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-5.00   sec  3.14 GBytes  5.38 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
iperf3: interrupt - the server has terminated
program exited with status 1
[I/193 dhcp]: Unicasting DHCPRELEASE message with xid: [1722528183] from client: 192.168.122.15 to server: 192.168.122.1
VFS: unmounting /dev
VFS: unmounting /proc
VFS: unmounting /
Powering off.

Comparing your output with mine, I see these in mine:

VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
BSD shrinker: event handler list found: 0x600000f45800

and yours:

VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
BSD shrinker: event handler list found: 0x600000fed900

so these 3 lines are missing in yours:

VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys

This suggests these 3 filesystems, including /dev/urandom, were not mounted. I wonder why?

@togoetha
Copy link
Author

Odd indeed. I checked with run.py to see if any qemu parameters were missing, but that doesn't seem to be it. Next I reduced the build pipeline to this:

- git clone "https://github.com/cloudius-systems/capstan.git" "capstan-src" && cd "capstan-src"
- GOOS=linux GOARCH=amd64 go build -a -ldflags "-X main.VERSION='v1.0.0' -w -s" -tags netgo -v -o "dist/amd64/capstan" "github.com/cloudius-systems/capstan"
- cp "dist/amd64/capstan" "/usr/local/bin/capstan" && chmod +x "/usr/local/bin/capstan"
- cd "${CI_PROJECT_DIR}/iperf"
- git clone --recurse-submodules "https://github.com/cloudius-systems/osv.git" "osv-src" && cd osv-src
- ./scripts/build-capstan-mpm-packages kernel
- ./scripts/build-capstan-mpm-packages iperf3
- cd "${CI_PROJECT_DIR}/iperf"
- capstan package list
- cp -r "${CI_PROJECT_DIR}/common/init" "${CI_PROJECT_DIR}/iperf/capstan/."
- cp -r "${CI_PROJECT_DIR}/common/run" "${CI_PROJECT_DIR}/iperf/capstan/."
- cp "${CI_PROJECT_DIR}/common/kubernetes.so" "${CI_PROJECT_DIR}/iperf/capstan/."
- cd ${CI_PROJECT_DIR}/iperf/capstan && capstan package compose --verbose iperf

Which still gives the same result. The init/kubernetes part is just a script that checks a specific directory for config files, it runs after boot anyway. Capstan package is minimal:

name: iperf
title: iperf3 flint image
author: Tom Goethals
created: "2024-09-13T18:29:52+02:00"
require:
- osv.iperf3

For now it works with a fork of iperf3 that uses /dev/random instead of urandom, so I'll figure it out as needed.

@togoetha
Copy link
Author

Could you post the two commands used to build and run the iperf image? I tried from scratch on my desktop, but for the life of me I can't get the extra mounts in there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wkozaczuk @togoetha