Bug 1896524 - Add desktop browsertime tests to measure video playback…

… latency.

The video under test is a simple recording of solid colors, where the
first frame is red, second frame is blue, subsequent frames alternate
between green and purple, and the final frame is yellow. This allows us
to identify what frame is being displayed and when in a browser agnostic
way.

This is achieved by updating the visualmetrics python script in
browsertime to identify frames which match a given color range, called
key color frames. The script will produce a list of which frames appear
when that we can use to estimate the playback latency. Note that the
annotated videos produced by the jobs will also be annotated to make it
clear which frames it believes are key color frames.

This patch requires an updated browsertime dependency once the following
pull request has been merged:
sitespeedio/browsertime#2119

Differential Revision: https://phabricator.services.mozilla.com/D210242

taskcluster/kinds/test/browsertime-desktop.yml

@@ -47,7 +47,7 @@ job-defaults:
                             - linux64-custom-car
                             - linux64-minidump-stackwalk
                             - linux64-fix-stacks
-                        windows10-64.*:
+                        windows.*-64.*:
                             - win64-custom-car
                             - win64-minidump-stackwalk
                             - win64-fix-stacks
@@ -65,12 +65,15 @@ job-defaults:
                         windows.*-32.*.:
                             - win32-minidump-stackwalk
                             - win32-fix-stacks
-                        windows10-64.*:
+                        windows.*-64.*:
                             - win64-minidump-stackwalk
                             - win64-fix-stacks
-                        macos.*:
+                        macosx1015.*:
                             - macosx64-minidump-stackwalk
                             - macosx64-fix-stacks
+                        macosx1400.*:
+                            - macosx64-aarch64-minidump-stackwalk
+                            - macosx64-aarch64-fix-stacks
                         default:
                             - linux64-minidump-stackwalk
                             - linux64-fix-stacks
@@ -864,6 +867,52 @@ browsertime-pageload-benchmark:
     tier: 2
     treeherder-symbol: Btime(tp6-bench)
 
+browsertime-video-playback-latency:
+    <<: *tp6-defaults
+    raptor:
+        apps: [firefox, chrome, safari]
+        test: video-playback-latency
+        run-visual-metrics: true
+        subtests:
+            by-app:
+                safari:
+                    - vpl-h264
+                default:
+                    - vpl-av1
+                    - vpl-h264
+                    - vpl-vp9
+    description: Raptor (browsertime) Video playback latency test
+    max-run-time: 4500
+    mozharness:
+        extra-options:
+            - --chimera
+            - --browsertime-no-ffwindowrecorder
+            - --extra-profiler-run
+    run-on-projects: []
+    tier: 2
+    treeherder-symbol: Btime(vpl)
+
+browsertime-video-playback-latency-profiling:
+    <<: *tp6-defaults
+    raptor:
+        apps: [firefox]
+        test: video-playback-latency
+        run-visual-metrics: true
+        subtests:
+            - av1-1080p
+            - h264-1080p
+            - vp9-1080p
+    description: Raptor (browsertime) Video playback latency test with Gecko Profiling
+    max-run-time: 4500
+    mozharness:
+        extra-options:
+            - --chimera
+            - --browsertime-no-ffwindowrecorder
+            - --gecko-profile
+    run-on-projects: []
+    tier: 2
+    treeherder-symbol: Btime-Prof(vpl)
+
 browsertime-speculative:
     raptor:
         apps: [firefox]

taskcluster/kinds/test/test-sets.yml

@@ -104,6 +104,7 @@ browsertime:
     - browsertime-throttled
     - browsertime-pageload-benchmark
     - browsertime-indexeddb
+    - browsertime-video-playback-latency
 
 browsertime-clang:
     - browsertime-tp6
@@ -119,6 +120,7 @@ browsertime-clang:
 
 browsertime-profiling:
     - browsertime-tp6-profiling
+    - browsertime-video-playback-latency-profiling
 
 browsertime-high-priority:
     - browsertime-benchmark
@@ -347,6 +349,8 @@ windows-talos:
     # - talos-h1 Bug 1487031 - Disabled for not finding actionable regressions
 
 windows-ref-hw-tests:
+    - browsertime-video-playback-latency
+    - browsertime-video-playback-latency-profiling
     - mochitest-media
 
 talos-xperf:

testing/perfdocs/generated/raptor-metrics.rst

@@ -100,6 +100,26 @@ Similar to SpeedIndex, except that it uses the contentfulness of a frame to dete
      * **Mobile**: `allrecipes <raptor.html#allrecipes-m>`__, `amazon <raptor.html#amazon-m>`__, `amazon-search <raptor.html#amazon-search-m>`__, `bild-de <raptor.html#bild-de-m>`__, `bing <raptor.html#bing-m>`__, `bing-search-restaurants <raptor.html#bing-search-restaurants-m>`__, `booking <raptor.html#booking-m>`__, `cnn <raptor.html#cnn-m>`__, `cnn-ampstories <raptor.html#cnn-ampstories-m>`__, `dailymail <raptor.html#dailymail-m>`__, `ebay-kleinanzeigen <raptor.html#ebay-kleinanzeigen-m>`__, `ebay-kleinanzeigen-search <raptor.html#ebay-kleinanzeigen-search-m>`__, `espn <raptor.html#espn-m>`__, `facebook <raptor.html#facebook-m>`__, `facebook-cristiano <raptor.html#facebook-cristiano-m>`__, `google <raptor.html#google-m>`__, `google-maps <raptor.html#google-maps-m>`__, `google-search-restaurants <raptor.html#google-search-restaurants-m>`__, `imdb <raptor.html#imdb-m>`__, `instagram <raptor.html#instagram-m>`__, `microsoft-support <raptor.html#microsoft-support-m>`__, `reddit <raptor.html#reddit-m>`__, `sina <raptor.html#sina-m>`__, `stackoverflow <raptor.html#stackoverflow-m>`__, `wikipedia <raptor.html#wikipedia-m>`__, `youtube <raptor.html#youtube-m>`__, `youtube-watch <raptor.html#youtube-watch-m>`__
 
 
+Estimated Frame Latency (Any)
+=============================
+Similar to estimatedFirstFrameLatency, except that it uses all identified frames during video playback, normalized to be an estimate of when the first frame was displayed by using the expected time offset from the video itself.
+
+
+  * **Aliases**: estimatedAnyFrameLatency
+  * **Tests using it**:
+     * **Custom**: `vpl-av1 <raptor.html#vpl-av1-c>`__, `vpl-h264 <raptor.html#vpl-h264-c>`__, `vpl-vp9 <raptor.html#vpl-vp9-c>`__
+
+
+Estimated Frame Latency (First)
+===============================
+A metric used to denote the latency on displaying the first frame of a video. Calculated by using videos of the pageload from which key frames are identified by matching a given solid RGB color with fuzz.
+
+
+  * **Aliases**: estimatedFirstFrameLatency
+  * **Tests using it**:
+     * **Custom**: `vpl-av1 <raptor.html#vpl-av1-c>`__, `vpl-h264 <raptor.html#vpl-h264-c>`__, `vpl-vp9 <raptor.html#vpl-vp9-c>`__
+
+
 First Paint
 ===========
 Denotes the first time the browser performs a paint that has content in it (in ms).