Video stream lags/latency

[SystemDiagnosticss]

Video stream has big lags and latency
https://youtu.be/yXp9vstm8bA
I use mjpeg codec and firefox browser
Url: http://0.0.0.0:8080/stream?topic=/.../image_raw&width=640&height=470

I am also tested params such us quality and other type=ros_compressed.
But its not helped
What params I must use for minimal latency???

[randoms]

The latency might be introduced by this commit.
In my case, I have no latency when using mjpeg codec, and less then 1 second when using vp8 codec. (The latency might be high at beginning, but will be lower later.)
I'm not familiar with ffmepg. I tried to reduce the latency but with no success.
I think the browser cache maybe the cause in your case. The latency not only from server side but also from the client side. You may try to use Chrome and check if the latency is lower.

[freddie-ai]

Regarding latency, I have found VP8 to have a constant delay of around 2s on my machine, compared to ros_compressed which gradually develops a larger latency as time progresses.

VP9 and H264 seem quite buggy. Specifically, H.264 begins very well (with low latency) but quickly slows down until it gets stuck on one frame (this happens within about 5s of starting playback).

Would love to know if anyone else has thoughts on achieving near realtime streaming?

[msadowski]

Hi!

Did you end up finding some optimal settings for your use case? I'm currently in the similar boat - trying to find a way to stream videos with minimal delay using minimal bandwidth. Any pointers you have would be highly appreciated!

[SystemDiagnosticss]

@msadowski Hello. In my case problew was in Gazebosim. Gazibosim lags show in video. I understand it later. Web_video_server for real camera works "out of the box" - GOOD without lags.

[msadowski]

@SystemDiagnosticss: Thanks for the update! Did you play with web_video_server settings for the stream? I was trying to optimize the payload for minimal lag and bandwidth. What seems to work reasonably with my laptop camera (relatevily high res) is embedding the stream like this:
<img src="http://0.0.0.0:8080/stream?topic=/v4l/camera/image_raw&type=mjpeg&width=200&height=200&default_transport=compressed" width="400" height="300">

I tried using theora for the default transport to save bandwidth however there is a crazy lag (over 10 seconds).

Did you find anything that seems better than compressed transport?

Thanks!

[IceSentry]

Did anyone manage to use the vp8 stream? The mjpeg solution is very inefficient but the vp8 stream has so much delay it's unusable

[msadowski]

@IceSentry I did quite some experiments with it and never managed to use any of the streams that require a video element to run with a reasonable latency. I'm using only the sources that support elements right now.

[viktorku]

vp8 has always introduced latencies that build up over time. Currently for best quality I would recommend the new png method #74. It delivers noticeable improvement over mjpeg with max quality.

[akshata1100]

Hey, what's the best solution to this?

[khaledgabr77]

@msadowski did you used image_transport pkg or any transport method to publish it as a compressed image. or default_transport=compressed it's a parameter I can pass with the raw image and it will handle that.

[msadowski]

@khaledgabr77 The last time I've been looking into that was around 4 years ago so I imagine a lot could have changed. Around this time I've published a blog post: https://msadowski.github.io/ros-web-tutorial-pt3-web_video_server/ where you will find my ideas on what seemed to worked well at the time. From what I can see I was indeed using a compressed transport.

[viktorku]

Please use the lossless PNG streamer in your application (with type=png). It does not suffer from any latency even when using the highest quality because PNG encoding is faster and more lightweight than VP8. It's not a 1:1 substitute as you need to integrate it as an image instead of a video but you get higher quality and faster delivery. The VP8 compression is slow and harder to optimize especially when working with higher framerates (30+). Pair that with a publisher node under a higher workload and lags are bound to happen at some point.

@akshata1100 @khaledgabr77 @msadowski et al

[khaledgabr77]

@msadowski thanks for sharing your blog, it's was very useful.
i used image_transport and used ros_compressed type , and passed this type over the server and it's working, but the issue with lagging and delay.
i'm using 2 camera with same type ros_compressed and sending this images over the server.
the first camera working real time but the second one start working as real time and after 1 min the delay start happening.

[khaledgabr77]

@viktorku what i'm working is publish 3 camera images over the server..i'm using web_video_server to publish this data..they working as raw images but found there are a big delay and lagging. so i'm trying to compressed this images and uncompressed over the server.
i used image_transport pkg and compressed theses images over the server and passed the ros_compressed as type. still have the delay and lagging for 3 cameras.
i'm not sure PNG will solve the delay issue, but i read that h264 type is more efficient.
also, i will try using PNG and h264 and h265 . and will update with results.

[khaledgabr77]

also, when tried with ros_compressed and played with parameters like reduce the quality started with 90 then 30 until 5 and the window size..still have one camera working real time and second camera has a delay.

[omerts]

Anyone solved it yet?

[randoms]

This issue has existed for years, and reduce latency is also not an easy job. I have some updates on this issue.
The latency is from both the server side and the client side.
For the server side latency, most of it was introduced by the video encoder. Here are some points we can optimize.

1. Use hardware video encoders, for example your GPU. If you are using an Intel CPU you can also use VAAPI to encode h264 videos.
2. Optimize video encoder parameters, it depends on the type of your encoder.

For the client side latency, caches are the cause of most latency. We need try to do out best to reduce caches of our video player. Here is a very useful ffmpeg command to play the video with low delay.

ffplay -fflags nobuffer -flags low_delay -framedrop -fflags discardcorrupt  -preset ultrafast -tune zerolatency -fpsprobesize 1 -max_delay 1 -probesize 32 -analyzeduration 0 -sync ext  "http://192.168.0.104:8080/stream?topic=/camera/image_raw&type=h264"

192.168.0.104 is the IP address of robot, and /camera/image_raw is the camera topic. You need to replace them with yours.

If you are using an Intel CPU, This is our optimized version, h264 encoder is accelerated by VAAPI. If you are using RK3588, this is another optimized version. The package was rewrited with gsteamer, and added audio support. video encoders was accelerated by MPP. The latency is only about 100ms. If you are familiar with gstreamer, it's easy to modify it to support other GPUs.