chromium测试

[xinali]

chromium测试

最近更新: 2022/06/16

感觉已经好久没有写相关的技术文章了，主要是因为做后端开发，实在是太忙了，我又对那玩意不太感兴趣，很多东西本来想写的，有时候开个头，写着写着就放弃了，最近刚转浏览器内核开发相关工作，老大分配的第一个任务就是把chromium的相关测试跑起来，所以大概花了一周多，大概熟悉了浏览器测试相关部分的内容，可能不太准确，毕竟刚刚开始接触，目前接触的内容，感觉东西可以写的有很多，后期随着内容深入不断更新:)

在经过了解chromium相关测试内容后，我个人感觉可以把测试简单分为两大类

1. 单元测试

   包含了浏览器测试(browser_tests)，模糊测试(fuzzer)等

2. 性能测试

   主要测试浏览器的一些性能，由谷歌开发的测试框架catapult负责，它里面包含了很多的内容

环境准备

基础环境

ubuntu 18.04 LTS 64
windows 10

chromium相关环境

目前chromium的源码，文件比较多，最好挂比较稳定的梯子，一次不行的话，多次尝试吧

1. 安装depot_tools

   git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
   

2. 设置环境变量

   export PATH="$PATH:/path/to/depot_tools"
   export PATH="$PATH:${HOME}/depot_tools"
   

3. 拉取源码

   mkdir ~/chromium && cd ~/chromium
   fetch --nohooks chromium
   

4. 切换分支版本

   cd ~/chromium/src
   git checkout xx.xxxx.x.xx # 可以根据你的需要切换内核版本
   gclient sync # 必须有，否则如果checkout之后编译会出错，功能主要是同步一些第三方库相关信息
   

5. 编译生成

   ./build/install-build-deps.sh
   gclient runhooks
   gn gen out/Default
   autoninja -C out/Default chrome
   

python环境

安装pyenv

1. 安装依赖

   apt-get install -y make build-essential libssl-dev zlib1g-dev \
   libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \
   libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl
   

2. 安装pyenv

   curl https://pyenv.run | bash
   

3. 添加环境变量到~/.bashrc

   export PATH="$HOME/.pyenv/bin:$PATH"
   eval "$(pyenv init --path)"
   eval "$(pyenv virtualenv-init -)"
   

4. 开始安装python

   source ~/.bashrc
   pyenv install 2.7.14
   

关于gn/ninja构建系统

通过上面的编译和生成过程，可以发现我们并不是使用比较常见的make/cmake这种构建方式，而是使用的gn先生成相关的ninja构建文件(相当于Makefile文件)，然后使用ninja编译生成可执行文件。

gn是谷歌开源的一个元构建系统（meta-build system）。这个”元构建“的意思是，gn并不直接帮你构建项目，而是帮你产生构建项目的ninja文件，然后你再用ninja去构建项目。或者你可以这么理解，gn相当于帮你生成Makefile，然后你再用make去编译构建你的项目。

这么做的原因是，ninja虽然有构建速度快的优点，但它更多是为机器解析设计的，人能看懂ninja文件，但要为项目手写ninja文件就比较繁琐。gn结合ninja，能够让我们方便的创建和维护项目，同时又能享受ninja的编译性能。

可以阅读参考文章，对该构建系统有个简单的了解，这部分我们在browser tests会遇到

性能测试

capatult

目前capatult主要有以下几个功能

Trace-view ：性能数据分析和可视化，支持 Chromium 的 trace_event 和 Linux 内核 trace (ftrace)
Telemetry ：性能测试框架
Performance Dashboard ：性能 Dashboard
systrace ： Android systrace 解析
Web Page Replay ：网页操作和录屏

telemetry(run_benchmark)

列出当前存在哪些测试指标

python src\tools\perf\run_benchmark  --browser-executable=".\src\out\Release\chrome.exe" --browser=exact list

得到的可用测试指标

Available benchmarks for release are:
  blink_perf
  blink_perf.accessibility
  blink_perf.bindings
  blink_perf.canvas
  blink_perf.css
  blink_perf.display_locking
  blink_perf.dom
  blink_perf.events
  blink_perf.image_decoder
  blink_perf.layout
  blink_perf.layout_ng
  blink_perf.owp_storage
  blink_perf.paint
  blink_perf.paint_layout_ng
  blink_perf.parser
  blink_perf.parser_layout_ng
  blink_perf.shadow_dom
  blink_perf.svg
  blink_perf_xml_http_request.BlinkPerfXMLHttpRequest
  dromaeo
  dummy_benchmark.noisy_benchmark_1                   A noisy benchmark with mean=50 & std=20.
  dummy_benchmark.stable_benchmark_1                  A low noise benchmark with mean=100 & std=1.
  generic_trace.top25
  generic_trace_ct
  heap_profiling.desktop.disabled
  heap_profiling.desktop.native
  heap_profiling.desktop.pseudo
  jetstream
  jetstream2                                          JetStream2, a combination of JavaScript and Web Assembly benchmarks.
  kraken                                              Mozilla's Kraken JavaScript benchmark.
  leak_detection.cluster_telemetry
  loading.cluster_telemetry
  loading.desktop                                      A benchmark measuring loading performance of desktop sites.
  loading.desktop_layout_ng                           A benchmark that runs loading.desktop with the layoutng flag.
  loading.mobile_layout_ng                            A benchmark that runs loading.mobile with the layoutng flag.
  media.desktop                                       Obtains media performance for key user scenarios on desktop.
  media_router.cpu_memory                             Obtains media performance for key user scenarios on desktop.
  media_router.cpu_memory.no_media_router             Benchmark for CPU and memory usage without Media Router.
  memory.cluster_telemetry
  memory.desktop                                      Measure memory usage on synthetic sites.
  memory.leak_detection
  memory.long_running_desktop_sites                   Measure memory usage on popular sites.
  multipage_skpicture_printer
  multipage_skpicture_printer_ct                      Captures mSKPs for Cluster Telemetry.
  octane                                              Google's Octane JavaScript benchmark.
  power.desktop
  rasterize_and_record_micro.partial_invalidation     Measures rasterize and record performance for partial inval. on big pages.
  rasterize_and_record_micro.top_25                   Measures rasterize and record performance on the top 25 web pages.
  rasterize_and_record_micro_ct                       Measures rasterize and record performance for Cluster Telemetry.
  rendering.cluster_telemetry                         Measures rendering performance for Cluster Telemetry.
  rendering.desktop
  repaint_ct                                          Measures repaint performance for Cluster Telemetry.
  screenshot_ct                                       Captures PNG screenshots of web pages for Cluster Telemetry. Screenshots
  skpicture_printer
  skpicture_printer_ct                                Captures SKPs for Cluster Telemetry.
  speedometer
  speedometer-future                                  Speedometer benchmark with the V8 flag --future.
  speedometer2                                        Speedometer2 Benchmark.
  speedometer2-future                                 Speedometer2 benchmark with the V8 flag --future.
  system_health.common_desktop                        Desktop Chrome Energy System Health Benchmark.
  system_health.memory_desktop                        Desktop Chrome Memory System Health Benchmark.
  tab_switching.typical_25                            This test records the MPArch.RWH_TabSwitchPaintDuration histogram.
  tracing.tracing_with_background_memory_infra        Measures the overhead of background memory-infra dumps
  tracing.tracing_with_debug_overhead
  v8.browsing_desktop
  v8.browsing_desktop-future
  v8.loading.cluster_telemetry
  v8.loading_runtime_stats.cluster_telemetry
  v8.runtime_stats.top_25                             Runtime Stats benchmark for a 25 top V8 web pages.
  webrtc                                              Base class for WebRTC metrics for real-time communications tests.
  xr.webvr.live.static                                Measures WebVR performance with live websites.
  xr.webvr.static                                     Measures WebVR performance with synthetic sample pages.
  xr.webvr.wpr.static                                 Measures WebVR performance with WPR copies of live websites.
  xr.webxr.static                                     Measures WebXR performance with synthetic sample pages.

Not supported benchmarks for release are (force run with -d):
  cros_tab_switching.typical_24                       Measures tab switching performance with 24 tabs.
  cros_ui_smoothness                                  Measures ChromeOS UI smoothness.
  heap_profiling.mobile.disabled
  heap_profiling.mobile.native
  heap_profiling.mobile.pseudo
  loading.mobile                                       A benchmark measuring loading performance of mobile sites.
  media.mobile                                        Obtains media performance for key user scenarios on mobile devices.
  orderfile.memory_mobile                             Benchmark for native code memory footprint evaluation.
  orderfile_generation.debugging                      A very short benchmark for debugging metrics collection.
  orderfile_generation.testing
  orderfile_generation.training
  orderfile_generation.variation.testing0
  orderfile_generation.variation.testing1
  orderfile_generation.variation.testing2
  orderfile_generation.variation.training
  rendering.mobile
  startup.mobile                                      Startup benchmark for Chrome on Android.
  system_health.common_mobile                         Mobile Chrome Energy System Health Benchmark.
  system_health.memory_mobile                         Mobile Chrome Memory System Health Benchmark.
  system_health.webview_startup                       Webview startup time benchmark
  v8.browsing_mobile
  v8.browsing_mobile-future
  xr.browsing.static                                  Benchmark for testing the VR Browsing Mode performance on sample pages.
  xr.browsing.wpr.smoothness                          Benchmark for testing VR browser scrolling smoothness and throughput.
  xr.browsing.wpr.static                              Benchmark for testing the VR Browsing Mode performance on WPR pages.

挑选其中一个用于测试

python src\tools\perf\run_benchmark  --browser-executable=".\src\out\Release\chrome.exe" --browser=exact blink_perf.css

如果成功的话，就会不停的弹窗开始测试。

单元测试

browser tests

编译

# 编译browser_tests
ninja -C out\Release browser_tests

查看帮助，该命令会直接开启测试

xxx\src>out\Release\browser_tests.exe -h
This program contains tests written using Google Test. You can use the
following command line flags to control its behavior:

Test Selection:
  --gtest_list_tests
      List the names of all tests instead of running them. The name of
      TEST(Foo, Bar) is "Foo.Bar".
  --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]
      Run only the tests whose name matches one of the positive patterns but
      none of the negative patterns. '?' matches any single character; '*'
      matches any substring; ':' separates two patterns.
  --gtest_also_run_disabled_tests
      Run all disabled tests too.

Test Execution:
  --gtest_repeat=[COUNT]
      Run the tests repeatedly; use a negative count to repeat forever.
  --gtest_shuffle
      Randomize tests' orders on every iteration.
  --gtest_random_seed=[NUMBER]
      Random number seed to use for shuffling test orders (between 1 and
      99999, or 0 to use a seed based on the current time).

Test Output:
  --gtest_color=(yes|no|auto)
      Enable/disable colored output. The default is auto.
  --gtest_print_time=0
      Don't print the elapsed time of each test.
  --gtest_output=(json|xml)[:DIRECTORY_PATH\|:FILE_PATH]
      Generate a JSON or XML report in the given directory or with the given
      file name. FILE_PATH defaults to test_detail.xml.

Assertion Behavior:
  --gtest_break_on_failure
      Turn assertion failures into debugger break-points.
  --gtest_throw_on_failure
      Turn assertion failures into C++ exceptions for use by an external
      test framework.
  --gtest_catch_exceptions=0
      Do not report exceptions as test failures. Instead, allow them
      to crash the program or throw a pop-up (on Windows).

Except for --gtest_list_tests, you can alternatively set the corresponding
environment variable of a flag (all letters in upper-case). For example, to
disable colored text output, you can either specify --gtest_color=no or set
the GTEST_COLOR environment variable to no.

For more information, please read the Google Test documentation at
https://github.com/google/googletest/. If you find a bug in Google Test
(not one in your own code or tests), please report it to
<[email protected]>.
IMPORTANT DEBUGGING NOTE: each test is run inside its own process.
For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with either
--single-process-tests (to run the test in one launcher/browser process) or
--single-process (to do the above, and also run Chrome in single-process mode)

列出可用测试

PS xxx\src> .\out\Release\browser_tests.exe  --gtest_list_tests
PlatformAppBrowserTest.  # TypeParam =
  RunningAppsAreRecorded  # GetParam() =
  ActiveAppsAreRecorded  # GetParam() =
  FileAccessIsSavedToPrefs  # GetParam() =
  DISABLED_FileAccessIsRestored  # GetParam() =
  LoadAndLaunchAppChromeRunning  # GetParam() =
  LoadAndLaunchAppWithFile  # GetParam() =
  CreateAndCloseAppWindow  # GetParam() =
  OnLaunchedEvent  # GetParam() =
  DisabledWindowProperties  # GetParam() =
  EmptyContextMenu  # GetParam() =
  AppWithContextMenu  # GetParam() =
  InstalledAppWithContextMenu  # GetParam() =
  AppWithContextMenuTextField  # GetParam() =
  AppWithContextMenuSelection  # GetParam() =
  AppWithContextMenuClicked  # GetParam() =
  DisallowNavigation  # GetParam() =
  DisallowBackgroundPageNavigation  # GetParam() =
  DISABLED_Iframes  # GetParam() =
  DisallowStorage  # GetParam() =
  Restrictions  # GetParam() =
  PlatformAppsOnly  # GetParam() =
  Isolation  # GetParam() =
  DISABLED_ExtensionWindowingApis  # GetParam() =
  OpenLink  # GetParam() =
  MutationEventsDisabled  # GetParam() =
  DISABLED_AppWindowRestoreState  # GetParam() =
  AppWindowAdjustBoundsToBeVisibleOnScreen  # GetParam() =
  DISABLED_ConstrainedWindowRequest  # GetParam() =
  ReloadRelaunches  # GetParam() =
  ComponentReloadLoadsLazyBackgroundPage  # GetParam() =
  PRE_PRE_ComponentAppBackgroundPage  # GetParam() =
  PRE_ComponentAppBackgroundPage  # GetParam() =
  ComponentAppBackgroundPage  # GetParam() =
  DISABLED_ComponentExtensionRuntimeReload  # GetParam() =
  DISABLED_Messaging  # GetParam() =
  DISABLED_WebContentsHasFocus  # GetParam() =
  WindowDotPrintShouldBringUpPrintPreview  # GetParam() =
  DISABLED_ClosingWindowWhilePrintingShouldNotCrash  # GetParam() =
  ReinstallDataCleanup  # GetParam() =
  AppsIgnoreDefaultZoom  # GetParam() =
  AppWindowIframe  # GetParam() =
  NewWindowWithNonExistingFile  # GetParam() =
  SandboxedLocalFile  # GetParam() =
  NewWindowAboutBlank  # GetParam() =
  TouchpadPinchSyntheticWheelEvents  # GetParam() =
  PictureInPicture  # GetParam() =
  ChromeRuntimeGetPackageDirectoryEntryApp  # GetParam() =
LoadAndLaunchPlatformAppBrowserTest.  # TypeParam =
  LoadAndLaunchAppChromeNotRunning  # GetParam() =
LoadAndLaunchExtensionBrowserTest.  # TypeParam =
  DISABLED_LoadAndLaunchExtension  # GetParam() =
AccessibilityLabelsBrowserTest.  # TypeParam =
  NewWebContents  # GetParam() =
  ExistingWebContents  # GetParam() =
  NotEnabledWithoutScreenReader  # GetParam() =
BrowserAccessibilityStateImplTest.  # TypeParam =
  TestHistograms  # GetParam() =
ImageAnnotationBrowserTest.  # TypeParam =
  AnnotateImageInAccessibilityTree  # GetParam() =
  ImagesInLinks  # GetParam() =
  ImageDoc  # GetParam() =
  ImageUrl  # GetParam() =
  NoAnnotationsAvailable  # GetParam() =
  AnnotationError  # GetParam() =
  ImageWithSrcSet  # GetParam() =
  DISABLED_AnnotationLanguages  # GetParam() =
  DoesntAnnotateInternalPages  # GetParam() =
  TutorMessageOnlyOnFirstImage  # GetParam() =
  TutorMessageOnlyOnFirstImageInLinks  # GetParam() =
InterstitialAccessibilityBrowserTest.  # TypeParam =
  TestSSLInterstitialAccessibility  # GetParam() =
AppViewTest.  # TypeParam =
  TestAppViewWithUndefinedDataShouldSucceed  # GetParam() =
  DISABLED_TestAppViewRefusedDataShouldFail  # GetParam() =
  TestAppViewGoodDataShouldSucceed  # GetParam() =
  TestAppViewMultipleConnects  # GetParam() =
  TestAppViewConnectFollowingPreviousConnect  # GetParam() =
  TestAppViewEmbedSelfShouldFail  # GetParam() =
  KillGuestWithInvalidInstanceID  # GetParam() =
  KillGuestCommunicatingWithWrongAppView  # GetParam() =
WebViewTest.  # TypeParam =
  AudibilityStatePropagates  # GetParam() =
  WebViewRespectsInsets  # GetParam() =
  AudioMutesWhileAttached  # GetParam() =
  ...

尝试某个测试

PS xxx\src> .\out\Release\browser_tests.exe PlatformAppBrowserTest.RunningAppsAreRecorded
IMPORTANT DEBUGGING NOTE: each test is run inside its own process.
For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with either
--single-process-tests (to run the test in one launcher/browser process) or
--single-process (to do the above, and also run Chrome in single-process mode).
Using sharding settings from environment. This is shard 0/1
Using 3 parallel jobs.
Still waiting for the following processes to finish:
        "\src\out\Release\browser_tests.exe" --disable-gpu-process-for-dx12-vulkan-info-collection --gtest_also_run_disabled_tests --gtest_filter=PlatformAppBrowserTest.RunningAppsAreRecorded --single-process-tests --test-launcher-output="C:\Users\Liz\AppData\Local\Temp\16840_881301891\results16840_1744069879\test_results.xml" --user-data-dir="C:\Users\Liz\AppData\Local\Temp\16840_881301891\user_data"
        "xxx\src\out\Release\browser_tests.exe" --disable-gpu-process-for-dx12-vulkan-info-collection --gtest_also_run_disabled_tests --gtest_filter=PlatformAppBrowserTest.ActiveAppsAreRecorded --single-process-tests --test-launcher-output="C:\Users\Liz\AppData\Local\Temp\16840_1934187520\results16840_1153539424\test_results.xml" --user-data-dir="C:\Users\Liz\AppData\Local\Temp\16840_1934187520\user_data"
        "xxx\src\out\Release\browser_tests.exe" --disable-gpu-process-for-dx12-vulkan-info-collection --gtest_also_run_disabled_tests --gtest_filter=PlatformAppBrowserTest.FileAccessIsSavedToPrefs --single-process-tests --test-launcher-output="C:\Users\Liz\AppData\Local\Temp\16840_642618779\results16840_340282504\test_results.xml" --user-data-dir="C:\Users\Liz\AppData\Local\Temp\16840_642618779\user_data"
   ...

web tests(blink tests)

编译

# 编译web_tests
ninja -C out\Release blink_tests

查看帮助

xxx\src> python xxx\src\third_party\blink\tools\run_web_tests.py -h
Usage: run_web_tests.py [options] [tests]

Runs Blink web tests as described in docs/testing/web_tests.md

Options:
  -h, --help            show this help message and exit

  Platform options:
    --android           Alias for --platform=android
    --platform=PLATFORM
                        Platform to use (e.g., "mac-lion")

  Configuration options:
    --debug             Set the configuration to Debug
    -t TARGET, --target=TARGET
                        Specify the target build subdirectory under src/out/
    --release           Set the configuration to Release
    --no-xvfb           Do not run tests with Xvfb

  Printing Options:
    --debug-rwt-logging
                        print timestamps and debug information for
                        run_web_tests.py itself
    --details           print detailed results for every test
    -q, --quiet         run quietly (errors, warnings, and progress only)
    --timing            display test times (summary plus per-test w/
                        --verbose)
    -v, --verbose       print a summarized result for every test (one line per
                        test)

  web-platform-tests (WPT) Options:
    --no-manifest-update
                        Do not update the web-platform-tests MANIFEST.json
                        unless it does not exist.
  ...

尝试执行

PS xxx\src> python xxx\src\third_party\blink\tools\run_web_tests.py -t Release
Using port 'win-win10'
Test configuration: <win10, x86, release>
View the test results at file://xxx\src\out\Release\layout-test-results/results.html
Using random order with seed: 1654672627
Baseline search path: win -> generic
Using Release build
Regular timeout: 6000, slow test timeout: 30000
Command line: xxx\src\out\Release\content_shell.exe --run-web-tests --ignore-certificate-errors-spki-list=Nxvaj3+bY3oVrTc+Jp7m3E3sB1n3lXtnMDCyBsqEXiY=,55qC1nKu2A88ESbFmk5sTPQS/ScG+8DD7P+2bgFA9iM=,0Rt4mT6SJXojEMHTnKnlJ/hBKMBcI4kteBlhR1eTTdk= --user-data-dir --enable-direct-write --enable-crash-reporter --crash-dumps-dir=xxx\src\out\Release\crash-dumps\reports -

Collecting tests ...

如果在特定的版本，直接执行可能会有问题，原因在于测试配置的json文件存在字段错误

修复：src/third_party/blink/web_tests/VirtualTestSuites:1089做如下修改

bases": ["external/wpt/service-workers"] 改为 "base": "external/wpt/service-workers"

重新执行上述命令即可

如果没有上述错误，直接忽略

依赖问题

首先需要安装pywin32和VCForPython27，以方便编译psutil

1. 解决报psutil依赖，直接利用安装包安装

   python setup.py install
   

2. 解决报fcntl依赖

   目前没有办法，该模块在windows系统不可用

3. 解决报numpy依赖

   pip install numpy-1.12.1-cp27-none-win_amd64.whl
   

遇到的问题

1. ERROR at //device/vr/buildflags/buildflags.gni:xx:xx: Undefined identifier

   缺少第三方库相关文件，解决办法：运行gclient sync

参考

catapult框架

gn/ninja: 谷歌的新一代项目构建系统入门

Checking out and building Chromium on Linux

pyenv

Chromium 工程源码是如何测试的

ERROR at //device/vr/buildflags/buildflags.gni:xx:xx: Undefined identifier

browser tests

web tests

chromium checkout build

[ILikeIneine]

您好，想问下在windows下运行telemetry benchmark的python版本是多少您还有印象吗？

我在windows下尝试2.7不可以，好像依赖于3.x的python，但是仍然会有很多错误。

[xinali]

您好，想问下在windows下运行telemetry benchmark的python版本是多少您还有印象吗？

我在windows下尝试2.7不可以，好像依赖于3.x的python，但是仍然会有很多错误。

我印象不深了，时间有点久了，我当时测的版本是我们用的版本，你看看你用的是哪个内核版本，那个内核用的哪个python，确定了版本应该不会有很多错误

[luckiillly]

请问一下在使用run_benchmark 测试Android X5 webview时遇到BrowserConnectionGoneException: Timed out while waiting 60s for _GetDevToolsClient的问题，这个你有遇到过吗，请问这种是什么原因，怎么解决的？

[xinali]

请问一下在使用run_benchmark 测试Android X5 webview时遇到BrowserConnectionGoneException: Timed out while waiting 60s for _GetDevToolsClient的问题，这个你有遇到过吗，请问这种是什么原因，怎么解决的？

我们不用Andriod相关的代码环境，所以这类的测试，我自己没测试过