high (~1s) latency when running on localhost

[andres-erbsen]

Hi,

I tried out using salsify-receiver and salsify-sender on my laptop, and encountered unexpectedly high latency.

I set a 300ms metronome and counted ticks from when I moved my hand and saw the result. I estimate 3x300ms, that is around 1s of delay. The rest of the setup was as follows:

• salsify 0061035 with ssim patched to return 8 (no x264).
• Thinkpad X1C3 webcam (Integrated C (usb-0000:00:14.0-8)) that otherwise achieves nice (~100ms?) latency in cheese
• receiver command: ./src/salsify/salsify-receiver 1234 1280 720
• sender command: ./src/salsify/salsify-sender -p YUYV --device /dev/video0 127.0.0.1 1234 1337
• Linux ashryn 5.1.4-arch1-1-ARCH #1 SMP PREEMPT Wed May 22 08:06:56 UTC 2019 x86_64 GNU/Linux
• Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz (and cpu usage according to top was around 50% for the sender and 25% for the receiver).

Similar issues were confirmed when streaming to a different laptop over ethernet (and again over wifi).

Given that the paper reports better than 300ms latency in figure 6d ("intermittent link"), I am figuring there must be something that I am doing differently. What do you think could be going on?

Thanks,
Andres

[sadjad]

Hi Andres,

Thank you for reaching out. Off the top of my head, one major problem is the input format for the sender program (YUYV). As we need input frames to be in YUV420 format to be encoded, we do a manual conversion from YUYV to YUV420 in software, which is really slow (https://github.com/excamera/alfalfa/blob/master/src/input/camera.cc#L120). Additionally, most webcams don't deliver a high frame rate when using raw formats such as YUYV (my Logitech C922 delivers 5 or 10 fps in YUYV mode). On a 3.5 GHz CPU, I remember getting no more than 5 frames per second when using this format--which translates into a ~200ms latency just for getting the image off the camera.

The most preferable input format for us is YUV420, but most webcams don't support raw output in this format at all (they have their peak performance with MJPEG). You can see the list of supported formats by running v4l2-ctl --list-formats.

If you wanna make it work, I can think of a few ways to do that:

1. Improve the performance of conversion from YUYV to YUV420 (maybe by using a library like swscale) -- this may not be useful, because the real limit is the frame rate supported by your camera (run v4l2-ctl --list-formats-ext to check for the supported frame rates).
2. Write code to get images in MJPEG format from the camera (highest frame rate possible) and decode and convert to YUV420 using some fast library.
3. Use a camera that supports raw YUV420 output (could be hard to come by).
4. (Probably the easiest) Use ffmpeg to create a virtual camera with YUV420 output. FFmpeg reads the frames in MJPEG format from the real camera and outputs them to a virtual camera in YUV420 format. Salsify then uses the virtual camera in YUV420 mode. I outlined the details of this in Always Get Camera error #78 (comment).

Please let me know if you had any other questions!

Best,
Sadjad

[andres-erbsen]

Thanks, that was it. Suggestion 4 got me down to latencies around ~150ms and ~250ms (localhost and over wifi, rough measurements by syncing my hand waving to the webcam output at 180-degree phase and counting that frequency). Thanks for your help.

[keithw]

The root cause here is that the Thinkpad's webcam can only do 10 fps in YUYV422 mode, and to get 30fps, we'd require a JPEG decoder in Salsify. (Which is doable and probably easy, but not done.) For the performance evaluation in the paper, we were using a 60fps YUV420 webcam which Salsify can handle directly.

I think our performance on these 10fps YUYV webcams should be a lot better after 2723268 , though -- because of the way we were using the V4L2 API (with only a single buffer), I think we were only getting 5fps with these webcams.

I don't think our swizzling from YUYV(422) to YUV420 is the problem -- I timed it and this takes us less than 1 ms. With the 2723268 patch, we can get 10 fps with basically no latency. (Try the real-webcam command to see what salsify is seeing directly.)

For anything performance-critical, though, the real solution is probably going to require linking in a JPEG decoder. The Thinkpad's webcam can only do 10 fps in YUYV422 mode, and 30fps in MJPEG mode. So if you want anything better than 10 fps, I think we need our own JPEG decoder in the loop. I'm skeptical that ffmpeg's virtual device is not introducing added latency...

[keithw]

Also -- when testing Salsify over localhost, you probably want to limit the speed somehow (e.g. mm-link <(echo 4) <(echo 4) to model a 3 Mbps connection, and then run salsify-sender inside that). Otherwise Salsify will see an infinitely fast connection and may max out a CPU trying to encode with quality high enough to fill the pipe.

[keithw]

You might check if this helps -- Salsify now supports MJPEG webcams directly.