sequential-track-playback-manual dropping frames #18

yanj-github · 2021-02-15T14:51:32Z

I am seeing the test "sequential-track-playback-manual playback" is dropping many frames.
Can you investigate if you are seeing this on your end as well please?

louaybassbouss · 2021-02-15T17:25:58Z

@yanj-github can you provide more details for example which content. Did you tried to play the content e.g. in dash.js player?

yanj-github · 2021-02-15T17:36:40Z

@louaybassbouss I am using https://dash.akamaized.net/WAVE/vectors/avc_sets/1/
I just managed to run more tests. It seems different set up and different trys give me different frame loose. It would be really helpful that if you can check whether you are seeing same problem on your set up please?
The best result that I have is test on a TV which I have lost only first frame.
It appears that if I try to run test on the same PC that runs test runner gives me quite many frame loose. I am guessing it might be caused by the memory limit on my PC. Is there any requirement of the PC that test runner should work on please?
I have tried on another dash player which seems it is fine.

yanj-github · 2021-02-16T11:20:07Z

I would like to get answers to the following points:

If others seeing frame lost as well? I just wanted to rule out this is not my set-up issue only.
If others seeing same issue, then are we going to allow any tolerance to the observation?

louaybassbouss · 2021-02-16T12:40:49Z

I would like to get answers to the following points:

If others seeing frame lost as well? I just wanted to rule out this is not my set-up issue only.

If others seeing same issue, then are we going to allow any tolerance to the observation?

@yanj-github we are checking

jpiesing · 2021-02-16T12:46:23Z

@yanj-github Is this a PC or a TV? If it is easy for you to modify the code, you might try reading VideoPlaybackQuality.droppedVideoFrames (and perhaps totalVideoFrames ) from
https://w3c.github.io/media-playback-quality/ and logging the results.

In principle there should be no dropped frames.

louaybassbouss · 2021-02-16T12:57:48Z

@yanj-github if you are in Browser can you try to play the content via dash.js player I prepared this URL for you which takes the avc_sets/1/ video as input. Can you check the "Dropped Frame"? I started the video chrome Version 88.0.4324.150 (Official Build) (x86_64) several time in the first run I got 18 dropped frames, in the second run 1 dropped frame and third run 0 dropped frames.

yanj-github · 2021-02-16T13:31:04Z

@yanj-github if you are in Browser can you try to play the content via dash.js player I prepared this URL for you which takes the avc_sets/1/ video as input. Can you check the "Dropped Frame"? I started the video chrome Version 88.0.4324.150 (Official Build) (x86_64) several time in the first run I got 18 dropped frames, in the second run 1 dropped frame and third run 0 dropped frames.

Thanks @louaybassbouss I will try it, I think this is exactly what I am seeing on the test runner as well. On different try dropping different number of frames. It seems I never reach 0 dropping though. I will try dash.js player later and post an update.

yanj-github · 2021-02-16T15:03:48Z

@louaybassbouss I have tried avc_sets/1/ on dash js player from your linke above 117 on 1st run 88 on second run 19 on thrid run 0 drop on 4th run. I am getting different frame drop on different run. I think this might due to the network latency. I am thinking if there is any way to do clear run by passing internet but serve content locally.
I have downloaded content to the docker and give the local url to the test runner therefore I have 3 frame lost 1st try 1 frame lost 2nd try. I tried several times but the best case is having 1st frame lost.

louaybassbouss · 2021-02-16T15:08:49Z

@louaybassbouss I have tried avc_sets/1/ on dash js player from your linke above 117 on 1st run 88 on second run 19 on thrid run 0 drop on 4th run. I am getting different frame drop on different run. I think this might due to the network latency. I am thinking if there is any way to do clear run by passing internet but serve content locally.
I have downloaded content to the docker and give the local url to the test runner therefore I have 3 frame lost 1st try 1 frame lost 2nd try. I tried several times but the best case is having 1st frame lost.

Thanks @yanj-github for the investigation yes this is the reason why we need to serve the content locally.

yanj-github · 2021-02-16T15:11:36Z

@louaybassbouss I have tried avc_sets/1/ on dash js player from your linke above 117 on 1st run 88 on second run 19 on thrid run 0 drop on 4th run. I am getting different frame drop on different run. I think this might due to the network latency. I am thinking if there is any way to do clear run by passing internet but serve content locally.
I have downloaded content to the docker and give the local url to the test runner therefore I have 3 frame lost 1st try 1 frame lost 2nd try. I tried several times but the best case is having 1st frame lost.

Thanks @yanj-github for the investigation yes this is the reason why we need to serve the content locally.

Yes, I have pretty bad internet connection here. If you can try out locally see if you seeing frame lost on 1st frame always on test runner please? I dont think we can make dash js local though.

louaybassbouss · 2021-02-16T15:16:47Z

@yanj-github You can use dash.js with local content. just copy the local url to the MPD and paste it in the dash.js url input and click on load (make sure you use dash.js via http and not https e.g. http://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html).

jpiesing · 2021-02-16T15:18:21Z

I don't understand what is going on here.
The test code should be loading the stream using XHR or fetch.
A bad internet connection should result in the test taking longer to fill the buffer up to the minimum threshold to start playing.
If loading the segments is too slow then playback will stall at some point until those segments are loaded.
I don't see why individual frames would be lost.
What am I missing?

yanj-github · 2021-02-16T15:24:00Z

I don't understand what is going on here.
The test code should be loading the stream using XHR or fetch.
A bad internet connection should result in the test taking longer to fill the buffer up to the minimum threshold to start playing.
If loading the segments is too slow then playback will stall at some point until those segments are loaded.
I don't see why individual frames would be lost.
What am I missing?

I think I have same question too; I don't understand why frames dropping on the internet connection, I thought it would rather pause.
We will test and verify with different settings and recordings as we go along and build up a picture of what seems to be happening.

jpiesing · 2021-02-16T21:16:53Z

@yanj-github if you are in Browser can you try to play the content via dash.js player I prepared this URL for you which takes the avc_sets/1/ video as input. Can you check the "Dropped Frame"? I started the video chrome Version 88.0.4324.150 (Official Build) (x86_64) several time in the first run I got 18 dropped frames, in the second run 1 dropped frame and third run 0 dropped frames.

I've played it using Chromium on Linux. No dropped frames but it repeatedly stalls - it somehow doesn't detect the buffer running low and fails to reload early enough. Chrome on Android is basically the same.
Firefox on Linux has one dropped frame but no stalls - for the same URL.

vlc plays it on the same Linux computer with no stalling but with a glitch and an error near the start -
[00007f76f001b030] mp4 demux: Fragment sequence discontinuity detected 2 != 0

vlc on android is the same as vlc on Linux.

jpiesing · 2021-02-17T08:08:10Z

@rbouqueau please could you take a quick look at avc_sets/1/?

dash.js on Chrome doesn't like it - but in different ways for different people
vlc doesn't like it - but in a different way from chrome
the DASH-IF validator generates 1 error ("The d attribute of a SegmentTimeline shall not exceed the value give by the MPD maxSegmentDuration attribute.")
Of course these may not be related and some of them may not be significant.

rbouqueau · 2021-02-17T09:20:25Z

I'm on it. I can see some dropped frames too.

The packaging is a part of the workflow I haven't touched but where I spotted issues: 1) changing the FFmpeg version produces different streams and 2) there are indeed conformance or compatibility issues as reported at cta-wave/Test-Content-Generation#21 (comment).

rbouqueau · 2021-02-17T11:04:56Z

It seems like I have no dropped frame when I stop mixing frame rates. Audio has no impact (although browser logs and conformance maxDuration warning complain about the audio).

Both Chrome and Firefox show some dropped frames.

I tried previous dash.js versions (2.3.0 and 3.1.3) and could also see frame drops.

Still working on it.

@yanj-github Could you tell me more about how the "sequential-track-playback-manual playback" test works?

yanj-github · 2021-02-17T11:23:46Z

It seems like I have no dropped frame when I stop mixing frame rates. Audio has no impact (although browser logs and conformance maxDuration warning complain about the audio).

Both Chrome and Firefox show some dropped frames.

I tried previous dash.js versions (2.3.0 and 3.1.3) and could also see frame drops.

Still working on it.

@yanj-github Could you tell me more about how the "sequential-track-playback-manual playback" test works?

I have set up test runner locally and downloaded the contents and give the "sequential-track-playback-manual playback" the local URL. So when everything is local I can see it dropping 1st frame.
Regarding to how does the test itself work, I believe @louaybassbouss will be able to help to answer this question.

jpiesing · 2021-02-17T11:40:14Z

It seems like I have no dropped frame when I stop mixing frame rates. Audio has no impact (although browser logs and conformance maxDuration warning complain about the audio).

@yanj-github This test should not be mixing frame rates. If it is then there is something wrong.

Both Chrome and Firefox show some dropped frames.

I tried previous dash.js versions (2.3.0 and 3.1.3) and could also see frame drops.

Still working on it.

@yanj-github Could you tell me more about how the "sequential-track-playback-manual playback" test works?

@rbouqueau This test is intended to play a single DASH Representation from beginning to end. That's it. Nothing clever or smart. The code just needs to read the MPD to find the first Representation and the corresponding URL, read the initialization segment & pass that to MSE and then read the media segments and pass them to MSE. Really, really simple. It's intended for testing that various options in the content and packaging work.

rbouqueau · 2021-02-17T11:56:58Z

I've left it run 50mn with a subset using only one frame rate: no dropped frame both on Firefox and Chrome.

This test is intended to play a single DASH Representation from beginning to end.

Ok. Are there any tests involving multiple framerates? If so, had they passed with previous streams generated with the Test Content Generation scripts?

FYI I'm not excluding an issue from the content, I'm still investigating.

louaybassbouss · 2021-02-17T11:59:43Z

@jpiesing @yanj-github the test only plays the first representation in the video. There is no magic in the test implementation just fetch the media segments and append them to the SourceBuffer. On JavaScript level we don't have any influence on dropping frames. BTW: This test expects an MPD with a single representation to make easier to test and compare with other players it makes sense to only consider a single representation. The audio should be in a separate MDP.

jpiesing · 2021-02-17T12:02:14Z

I've left it run 50mn with a subset using only one frame rate: no dropped frame both on Firefox and Chrome.

This test is intended to play a single DASH Representation from beginning to end.

Ok. Are there any tests involving multiple framerates? If so, had they passed with previous streams generated with the Test Content Generation scripts?

"Switching Set Playback" should be run with an MPDs including 60/30/15 (and 50/25/12.5 but we don't have that yet).

FYI I'm not excluding an issue from the content, I'm still investigating.

rbouqueau · 2021-02-17T12:14:14Z

the test only plays the first representation in the video

Ok I've refocused my tests on this then.

rbouqueau · 2021-02-17T15:16:09Z

Everything seems ok: RAP placement, timings across segment borders, segment duration, uniformity of flags&durations, sidx. I'll stop for today but I'm open to ideas or suggestions.

While doing a random test, I realized that I have dropped frames with virtually any content:

So I tried on Linux: zero dropped frames (Firefox and Chromium). Is anyone not on Windows?

dsilhavy · 2021-02-17T16:03:54Z

Some thoughts:

From my experience high load in the application/JavaScript thread can also cause rendering issues. Do you see issues with lower bitrates < 8Mbit/s
Can you check media-internals in Chrome if you see any useful output: https://developers.google.com/web/tools/chrome-devtools/media-panel

jpiesing · 2021-02-17T16:13:00Z

I'm on Linux & not seeing dropped frames (except perhaps for 1 at the start sometimes). The same on Android.

rbouqueau · 2021-02-17T17:26:39Z

@dsilhavy Thanks :)

Max bitrate is 6mbps and it even occurs with a 200kbps 15fps stream. On both Firefox and Chrome. And on dash.js 2.3.0, 3.1.3 and 3.2.0. On the default BBB content and our content. The only differentiator is the OS - why not a regression specific to Windows? (or maybe a difference in the way the dropped frames are reported on Windows?) (seems far fetched but is my best guess before having a good night of sleep)

Media-internal says nothing on this unfortunately.

rbouqueau · 2021-02-22T09:38:47Z

I gave another try and still see dropped frames (although less than on our content) for https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://dash.akamaized.net/akamai/bbb_30fps/bbb_30fps.mpd

It is on my todo-list to investigate this further this week. As always any idea is welcome :)

dsilhavy · 2021-02-22T12:22:50Z

I did a quick check as well:
I also see occasional dropped frames on my Mac in Chrome. I can see the following output in media internals which correlates with the dropped frames (timing wise):

{video_buffering_state: {reason: "DECODER_UNDERFLOW", state: "BUFFERING_HAVE_NOTHING"}}
video_buffering_state: {reason: "DECODER_UNDERFLOW", state: "BUFFERING_HAVE_NOTHING"}

My first assumption was that this is caused by a quality switch and a decoder buffer underflow. However, I see the same problem even after the quality switch has already been rendered. Maybe there are some warnings/errors during the encoding process regarding the VBV?

Edit: For means of comparison: I saw the described behavior on the highest quality 12Mbit/s

rbouqueau · 2021-02-25T15:52:47Z

@dsilhavy interesting. There were no vbv parameters so I added them: still had the dropped frames. I've also removed the color parameters: same.

I came back to the Google Chrome media-internals and see this:

00:00:28.857	info	"Truncating audio buffer which overlaps append window end. PTS 59977142us frame_end_timestamp 60000361us append_window_end 60000000us"
00:00:28.857	info	"Dropping audio frame (DTS 60000362us PTS 60000362us,60023581us) that is outside append window [0us,60000000us)."
00:00:28.857	info	"Dropping audio frame (DTS 60023582us PTS 60023582us,60046801us) that is outside append window [0us,60000000us)."
00:00:28.857	info	"Dropping audio frame (DTS 60046802us PTS 60046802us,60070021us) that is outside append window [0us,60000000us)."

These messages are related to the audio only. I wanted to make a try using a different packager.

louaybassbouss · 2021-02-25T16:12:00Z

@rbouqueau @dsilhavy does it make sense to test a video-only stream without audio?

rbouqueau · 2021-02-25T16:26:44Z

@louaybassbouss Good remark. At the beginning I suspected the framerate mix. So I reduced the set until having only one video. It reduces the probability to encounter dropped frames with falling to zero. That's when I tested back on Big Buck Bunny and saw I had dropped frames too...

My current hypothesis is that 1) the dropped frame callback has changed and 2) our content has many of them and I suspect the packaging.

dsilhavy · 2021-02-26T07:45:04Z

@rbouqueau The messages on audio overlapping the append window are caused by audio segments slightly overlapping period boundaries. dash.js maps the period boundaries to the appendWindowStart and appendWindowEnd attributes of the Sourcebuffer. That way we make sure that segments that overlap period boundaries are only partially added to the respective Sourcebuffer (matching the Period boundaries).

rbouqueau · 2021-03-01T15:13:18Z

Ok I found time to repackage with gpac and the rate of dropped frames fell: https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://www.gpac-licensing.com/downloads/CTA-Wave/gpac/manifest.mpd.

louaybassbouss · 2021-03-01T15:23:51Z

Ok I found time to repackage with gpac and the rate of dropped frames fell: https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://www.gpac-licensing.com/downloads/CTA-Wave/gpac/manifest.mpd.

@rbouqueau I cannot play the video in dash.js it seems that the CORS Headers are not set for https://www.gpac-licensing.com/downloads/CTA-Wave/gpac/manifest.mpd

rbouqueau · 2021-03-01T15:37:16Z

Copied to https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://dash.akamaized.net/WAVE/vectors/avc_sets/gpac/manifest.mpd to avoid the CORS issue

jpiesing · 2021-03-01T16:01:23Z

Copied to https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://dash.akamaized.net/WAVE/vectors/avc_sets/gpac/manifest.mpd to avoid the CORS issue

I had one dropped frame on the first run through but none on subsequent runs.

jpiesing · 2021-04-27T15:26:38Z

Copied to https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://dash.akamaized.net/WAVE/vectors/avc_sets/gpac/manifest.mpd to avoid the CORS issue

@louaybassbouss @FritzHeiden As mentioned in the April 27 meeting, please use the above content in the docker deploy image as a short-term fix until there is more content from @rbouqueau

louaybassbouss · 2021-04-28T15:19:47Z

Copied to https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://dash.akamaized.net/WAVE/vectors/avc_sets/gpac/manifest.mpd to avoid the CORS issue

@louaybassbouss @FritzHeiden As mentioned in the April 27 meeting, please use the above content in the docker deploy image as a short-term fix until there is more content from @rbouqueau

@jpiesing AVC tests are now using the new content https://dash.akamaized.net/WAVE/vectors/avc_sets/gpac/manifest.mpd

jpiesing · 2021-04-28T16:10:41Z

Copied to https://reference.dashif.org/dash.js/v3.2.0/samples/dash-if-reference-player/index.html?mpd=https://dash.akamaized.net/WAVE/vectors/avc_sets/gpac/manifest.mpd to avoid the CORS issue

@louaybassbouss @FritzHeiden As mentioned in the April 27 meeting, please use the above content in the docker deploy image as a short-term fix until there is more content from @rbouqueau

@jpiesing AVC tests are now using the new content https://dash.akamaized.net/WAVE/vectors/avc_sets/gpac/manifest.mpd

@louaybassbouss The test starts but all I get is a black page with a QR code for a few seconds then the test finishes. There are no error messages. Presumably a URL is wrong somewhere?

FritzHeiden · 2021-05-03T09:46:03Z

@jpiesing I just rebuilt the docker image and ran it in a container. For me the tests work as intended, I cannot reproduce this behavior. As the old mpd URLs do not work anymore, which may cause such behavior, to me it seems this is some sort of caching issue again.

jpiesing · 2021-05-03T09:50:14Z

@jpiesing I just rebuilt the docker image and ran it in a container. For me the tests work as intended, I cannot reproduce this behavior. As the old mpd URLs do not work anymore, which may cause such behavior, to me it seems this is some sort of caching issue again.

@FritzHeiden There's no point debugging something that is out of date. I'll wait until the new content is integrated including the .zip file & try it again then.

rbouqueau · 2021-05-03T10:05:46Z

@FritzHeiden @jpiesing Let me know if there are still invisible things. I can navigate on the server and point you to the right content.

louaybassbouss · 2021-05-03T11:23:20Z

Thanks @rbouqueau when do you think you can remove the output folder from the zip file?

louaybassbouss · 2021-05-03T11:51:47Z

Thanks @rbouqueau when do you think you can remove the output folder from the zip file?

@rbouqueau you can ignore this it is already implemented. This is the current structure of the folder structure in the zip file (example t1.zip) and it is important to keep this structure across all files. Thanks a lot 👍

yanj-github · 2021-05-26T08:22:35Z

With the latest content I dont see many dropping frames. There are some dropped at the beginning then 1-2 in the middle.

jpiesing · 2021-05-26T08:27:20Z

Frames dropped at the beginning and end are particularly critical - that's the reason for the mezzanine content having single colour frames added before and after the video.
Please can you examine the recordings and determine if the dropped frames at the beginning are a defect of the playback device or of the observation framework.
If the playback device is dropping the first or last frame then that needs to be a failure.
1-2 dropped frames in the middle should be within the tolerance. In the future, there should be a check for a/v sync being preserved after such 1-2 dropped frames but we're not doing that yet.

yanj-github · 2021-05-26T08:43:37Z

Thanks @jpiesing I can confirm this is not OF. I have checked the recording manually, it starts from frame 6 onwards. The previous recroded frame to it is a balc screen with a test runner QR code status=ready. There fore we missing green screen and strating frame 1-5.

jpiesing · 2021-05-26T08:45:04Z

Thanks @jpiesing I can confirm this is not OF. I have checked the recording manually, it starts from frame 6 onwards. The previous recroded frame to it is a balc screen with a test runner QR code status=ready. There fore we missing green screen and strating frame 1-5.

Then that is a fairly significant fail for that particular device.

jpiesing · 2021-06-08T13:36:23Z

I propose closing this issue. If some devices aren't fast enough to play 1920x1080p60 content then that's a device problem not a test problem. If devices are fast enough to play 1920x1080p60 with (say) vlc but not with MSE then that's an architecture issue.
@yanj-github Do you agree we can close this issue?

yanj-github · 2021-06-08T14:14:32Z

Yes please @jpiesing

jpiesing · 2021-06-08T14:15:40Z

Yes please @jpiesing

This was referenced Apr 27, 2021

25% dropped frames running sequential track playback on Android #35

Closed

wrong media stream being installed for sequential track playback cta-wave/dpctf-deploy#12

Closed

yanj-github mentioned this issue May 26, 2021

default values for tolerances cta-wave/device-observation-framework#17

Closed

louaybassbouss added the ready issue is ready can be closed if all agree label May 26, 2021

jpiesing closed this as completed Jun 8, 2021

sequential-track-playback-manual dropping frames #18

sequential-track-playback-manual dropping frames #18

Comments

yanj-github commented Feb 15, 2021

louaybassbouss commented Feb 15, 2021

yanj-github commented Feb 15, 2021

yanj-github commented Feb 16, 2021

louaybassbouss commented Feb 16, 2021

jpiesing commented Feb 16, 2021

louaybassbouss commented Feb 16, 2021

yanj-github commented Feb 16, 2021

yanj-github commented Feb 16, 2021

louaybassbouss commented Feb 16, 2021

yanj-github commented Feb 16, 2021

louaybassbouss commented Feb 16, 2021 • edited Loading

jpiesing commented Feb 16, 2021

yanj-github commented Feb 16, 2021

jpiesing commented Feb 16, 2021 • edited Loading

jpiesing commented Feb 17, 2021

rbouqueau commented Feb 17, 2021

rbouqueau commented Feb 17, 2021

yanj-github commented Feb 17, 2021

jpiesing commented Feb 17, 2021

rbouqueau commented Feb 17, 2021

louaybassbouss commented Feb 17, 2021

jpiesing commented Feb 17, 2021

rbouqueau commented Feb 17, 2021

rbouqueau commented Feb 17, 2021

dsilhavy commented Feb 17, 2021

jpiesing commented Feb 17, 2021

rbouqueau commented Feb 17, 2021

rbouqueau commented Feb 22, 2021

dsilhavy commented Feb 22, 2021 • edited Loading

rbouqueau commented Feb 25, 2021

louaybassbouss commented Feb 25, 2021

rbouqueau commented Feb 25, 2021

dsilhavy commented Feb 26, 2021

rbouqueau commented Mar 1, 2021

louaybassbouss commented Mar 1, 2021

rbouqueau commented Mar 1, 2021

jpiesing commented Mar 1, 2021

jpiesing commented Apr 27, 2021

louaybassbouss commented Apr 28, 2021

jpiesing commented Apr 28, 2021

FritzHeiden commented May 3, 2021

jpiesing commented May 3, 2021

rbouqueau commented May 3, 2021

louaybassbouss commented May 3, 2021

louaybassbouss commented May 3, 2021

yanj-github commented May 26, 2021

jpiesing commented May 26, 2021

yanj-github commented May 26, 2021

jpiesing commented May 26, 2021 • edited Loading

jpiesing commented Jun 8, 2021

yanj-github commented Jun 8, 2021

jpiesing commented Jun 8, 2021

louaybassbouss commented Feb 16, 2021 •

edited

Loading

jpiesing commented Feb 16, 2021 •

edited

Loading

dsilhavy commented Feb 22, 2021 •

edited

Loading

jpiesing commented May 26, 2021 •

edited

Loading