Add support to visualmetrics to identify key frames matching the given colors #2119

aosmond · 2024-05-02T18:23:41Z

This patch updates the visualmetrics.py and visualmetrics-portable.py scripts to allow it to identify and record timestamps for frames that match the given color configuration. This consists of an RGB value with fuzz (anything +/- fuzz matches) and a fraction between 0 and 1 of the required percentage of each channel from the histogram that must match it.

This is intended to be used in Firefox automation to collect metrics with respect to video playback latency.

gmierz · 2024-05-03T13:00:29Z

Hi @aosmond, interesting metric! Can you provide an example video, and command that we could test this with along with the expected output from it?

aosmond · 2024-05-03T20:57:56Z

@gmierz
I added an example zip/description here:
https://gist.github.com/aosmond/4daa941cf36670e2ab0f7ecbeb66ed00

This should yield in browsertime.json under "visualmetrics" when the poster, first video frame, second video frame and last video frame are displayed.

visualmetrics/visualmetrics-portable.py

aosmond · 2024-05-13T19:39:01Z

Updated according to your feedback @gmierz. You can see this working in conjunction with Firefox CI in my try pushes which I am just polishing up now:
https://treeherder.mozilla.org/jobs?repo=try&revision=2bbf558778ffbf707ccfed5a49d24ecc7a0e8e59

I also updated the visualmetrics scripts to accept an individual range for each channel, as well as annotating the video with the key color frame information (if available).

aosmond · 2024-05-13T23:12:48Z

The mozilla CI side is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=1896524

gmierz

r+ for the visual metrics script changes, just a nit and a small question below. Thanks for fixing up the calculation method!

lib/video/postprocessing/finetune/getTimingMetrics.js

visualmetrics/visualmetrics.py

lib/support/har/index.js

lib/video/postprocessing/finetune/getTimingMetrics.js

soulgalore

I'm wondering could this just be a change in visualmetrics-portable.py and then add a generic way to pass on parameters to that script from Browsertime, and then you could have your magic analysing it in your ci?

aosmond · 2024-05-21T12:41:30Z

I'm wondering could this just be a change in visualmetrics-portable.py and then add a generic way to pass on parameters to that script from Browsertime, and then you could have your magic analysing it in your ci?

I'm happy to make whatever changes are necessary for a merge :). But I'm a bit uncertain as to what that would look like, as I didn't make too many changes outside the python scripts, just what I thought was necessary to get the information out and annotate the videos. What parts would you like removed?

I already do a post processing phase on the raw key color data in our CI to normalize things in case it didn't get a first frame, but found a second frame, etc.

lib/video/postprocessing/finetune/getTimingMetrics.js

soulgalore · 2024-05-31T06:14:41Z

lib/support/cli.js

@@ -718,6 +723,12 @@ export function parseCommandLine() {
      describe:
        'Use the portable visual-metrics processing script (no ImageMagick dependencies).'
    })
+    .option('visualMetricsKeyColor', {
+      type: 'array',
+      nargs: 8,


Think this is missmatch to the gist you provide with how to run it? I tried with bin/browsertime.js --visualMetrics true --video true --iterations 1 --visualMetricsKeyColor poster 120 250 250 20 0.75 --visualMetricsKeyColor first_frame 250 20 20 20 0.75 --visualMetricsKeyColor second_frame 20 20 250 20 0.75 --visualMetricsKeyColor last_frame 255 255 50 50 0.75 --browser firefox --video.framerate 60 https://www.sitespeed.io (I think that is in the example) but then I get "Not enough arguments following: visualMetricsKeyColor". Can you share how I can try it + do you know a live site where I can try?

I updated my gist at https://gist.github.com/aosmond/4daa941cf36670e2ab0f7ecbeb66ed00. I don't have a live site. It's primary purpose is to measure the latency of video playback, hence the special rainbow video I made, but in theory any video that displays a non-white solid color frame at some point would work.

…n colors. This patch updates the visualmetrics.py and visualmetrics-portable.py scripts to allow it to identify and record timestamps for frames that match the given color configuration. This consists of an RGB value with fuzz (anything +/- fuzz matches) and a fraction between 0 and 1 of the required percentage of each channel from the histogram that must match it.

…playback latency. The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. This patch updates the browsertime dependency to be built on the same version as we currently use in CI, with addition of our own patches to enable this while we wait to be merged. Until such a time, and until the tests have proven stable, they have been set at tier 3. The status of the merge can be seen at: sitespeedio/browsertime#2119 Differential Revision: https://phabricator.services.mozilla.com/D210242

…playback latency. The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. This patch requires an updated browsertime dependency once the following pull request has been merged: sitespeedio/browsertime#2119 Until then, the tests should remain disabled by default. Differential Revision: https://phabricator.services.mozilla.com/D210242

…playback latency. The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. Until the tests have proven stable, they have been set at tier 3. The status of the merge can be seen at: sitespeedio/browsertime#2119 Differential Revision: https://phabricator.services.mozilla.com/D210242

…playback latency. r=perftest-reviewers,sparky The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. Until the tests have proven stable, they have been set at tier 3. The status of the merge can be seen at: sitespeedio/browsertime#2119 Differential Revision: https://phabricator.services.mozilla.com/D210242

…playback latency. r=perftest-reviewers,sparky The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. Until the tests have proven stable, they have been set at tier 3. The status of the merge can be seen at: sitespeedio/browsertime#2119 Differential Revision: https://phabricator.services.mozilla.com/D210242 UltraBlame original commit: 9495d071271ff447bbe633af76131c289e00d471

…playback latency. The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. This patch requires an updated browsertime dependency once the following pull request has been merged: sitespeedio/browsertime#2119 Until then, the tests should remain disabled by default. Differential Revision: https://phabricator.services.mozilla.com/D210242

… latency. The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. This patch requires an updated browsertime dependency once the following pull request has been merged: sitespeedio/browsertime#2119 Differential Revision: https://phabricator.services.mozilla.com/D210242

…playback latency. The video under test is a simple recording of solid colors, where the first frame is red, second frame is blue, subsequent frames alternate between green and purple, and the final frame is yellow. This allows us to identify what frame is being displayed and when in a browser agnostic way. This is achieved by updating the visualmetrics python script in browsertime to identify frames which match a given color range, called key color frames. The script will produce a list of which frames appear when that we can use to estimate the playback latency. Note that the annotated videos produced by the jobs will also be annotated to make it clear which frames it believes are key color frames. Until the tests have proven stable, they have been set at tier 3. The status of the merge can be seen at: sitespeedio/browsertime#2119 Differential Revision: https://phabricator.services.mozilla.com/D210242

aosmond force-pushed the ao_visualmetrics_video_latency branch from 51a0aad to 511421d Compare May 3, 2024 03:25

gmierz requested review from soulgalore and gmierz May 3, 2024 13:00

aosmond force-pushed the ao_visualmetrics_video_latency branch from 511421d to 4044474 Compare May 3, 2024 17:37

aosmond force-pushed the ao_visualmetrics_video_latency branch from 4044474 to ec129bb Compare May 7, 2024 00:26

gmierz reviewed May 8, 2024

View reviewed changes

visualmetrics/visualmetrics-portable.py Outdated Show resolved Hide resolved

aosmond force-pushed the ao_visualmetrics_video_latency branch from ec129bb to a179153 Compare May 13, 2024 19:26

gmierz approved these changes May 14, 2024

View reviewed changes

lib/video/postprocessing/finetune/getTimingMetrics.js Outdated Show resolved Hide resolved

visualmetrics/visualmetrics.py Outdated Show resolved Hide resolved

soulgalore reviewed May 20, 2024

View reviewed changes

lib/support/har/index.js Outdated Show resolved Hide resolved

soulgalore reviewed May 20, 2024

View reviewed changes

lib/video/postprocessing/finetune/getTimingMetrics.js Outdated Show resolved Hide resolved

soulgalore reviewed May 20, 2024

View reviewed changes

lib/video/postprocessing/finetune/getTimingMetrics.js Outdated Show resolved Hide resolved

soulgalore reviewed May 20, 2024

View reviewed changes

aosmond force-pushed the ao_visualmetrics_video_latency branch from 5048a0a to d984511 Compare May 21, 2024 12:22

aosmond force-pushed the ao_visualmetrics_video_latency branch from d984511 to 32305f9 Compare May 21, 2024 14:24

aosmond requested a review from soulgalore May 26, 2024 19:49

soulgalore reviewed May 31, 2024

View reviewed changes

lib/video/postprocessing/finetune/getTimingMetrics.js Outdated Show resolved Hide resolved

soulgalore reviewed May 31, 2024

View reviewed changes

aosmond force-pushed the ao_visualmetrics_video_latency branch from bcef641 to b189a99 Compare June 3, 2024 01:06

Merge branch 'main' into ao_visualmetrics_video_latency

fc0cb3c

aosmond requested a review from soulgalore June 3, 2024 14:36

soulgalore merged commit 1b1645a into sitespeedio:main Jun 4, 2024
13 checks passed

Stanislav1975 mentioned this pull request Aug 9, 2024

[Snyk] Upgrade browsertime from 14.5.0 to 22.6.0 Stanislav1975/sitespeed.io#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to visualmetrics to identify key frames matching the given colors #2119

Add support to visualmetrics to identify key frames matching the given colors #2119

aosmond commented May 2, 2024

gmierz commented May 3, 2024

aosmond commented May 3, 2024 •

edited

Loading

aosmond commented May 13, 2024

aosmond commented May 13, 2024

gmierz left a comment •

edited

Loading

soulgalore left a comment

aosmond commented May 21, 2024

soulgalore May 31, 2024

aosmond Jun 3, 2024

Add support to visualmetrics to identify key frames matching the given colors #2119

Add support to visualmetrics to identify key frames matching the given colors #2119

Conversation

aosmond commented May 2, 2024

gmierz commented May 3, 2024

aosmond commented May 3, 2024 • edited Loading

aosmond commented May 13, 2024

aosmond commented May 13, 2024

gmierz left a comment • edited Loading

Choose a reason for hiding this comment

soulgalore left a comment

Choose a reason for hiding this comment

aosmond commented May 21, 2024

soulgalore May 31, 2024

Choose a reason for hiding this comment

aosmond Jun 3, 2024

Choose a reason for hiding this comment

aosmond commented May 3, 2024 •

edited

Loading

gmierz left a comment •

edited

Loading