[Reporting] Update Puppeteer to version 8.0.0 and Chromium to r856583

package.json

@@ -314,7 +314,7 @@
     "proxy-from-env": "1.0.0",
     "proxyquire": "1.8.0",
     "puid": "1.0.7",
-    "puppeteer": "npm:@elastic/[email protected]",
+    "puppeteer": "^8.0.0",
     "query-string": "^6.13.2",
     "raw-loader": "^3.1.0",
     "rbush": "^3.0.1",
@@ -586,7 +586,6 @@
     "@types/pretty-ms": "^5.0.0",
     "@types/prop-types": "^15.7.3",
     "@types/proper-lockfile": "^3.0.1",
-    "@types/puppeteer": "^5.4.1",
     "@types/rbush": "^3.0.0",
     "@types/reach__router": "^1.2.6",
     "@types/react": "^16.9.36",

x-pack/build_chromium/README.md

@@ -3,37 +3,24 @@
 We ship our own headless build of Chromium which is significantly smaller than
 the standard binaries shipped by Google. The scripts in this folder can be used
 to accept a commit hash from the Chromium repository, and initialize the build
-environments and run the build on Mac, Windows, and Linux.
+on Ubuntu Linux.
 
-## Before you begin
+## Why do we do this
 
-If you wish to use a remote VM to build, you'll need access to our GCP account,
-which is where we have two machines provisioned for the Linux and Windows
-builds. Mac builds can be achieved locally, and are a great place to start to
-gain familiarity.
+By default, Puppeteer will download a zip file containing the Chromium browser for any
+OS. This creates problems on Linux, because Chromium has a dependency on X11, which
+is often not installed for a server environment. We don't want to make a requirement
+for Linux that you need X11 to run Kibana. To work around this, we create our own Chromium
+build, using the
+[`headless_shell`](https://chromium.googlesource.com/chromium/src/+/5cf4b8b13ed518472038170f8de9db2f6c258fe4/headless)
+build target. There are no (trustworthy) sources of these builds available elsewhere.
 
-**NOTE:** Linux builds should be done in Ubuntu on x64 architecture. ARM builds
-are created in x64 using cross-compiling. CentOS is not supported for building Chromium.
-
-1. Login to our GCP instance [here using your okta credentials](https://console.cloud.google.com/).
-2. Click the "Compute Engine" tab.
-3. Find `chromium-build-linux` or `chromium-build-windows-12-beefy` and start the instance.
-4. Install [Google Cloud SDK](https://cloud.google.com/sdk) locally to ssh into the GCP instance
-5. System dependencies:
-    - 8 CPU
-    - 30GB memory
-    - 80GB free space on disk (Try `ncdu /home` to see where space is used.)
-    - git
-    - python2 (`python` must link to `python2`)
-    - lsb_release
-    - tmux is recommended in case your ssh session is interrupted
-    - "Cloud API access scopes": must have **read / write** scope for the Storage API
-6. Copy the entire `build_chromium` directory from the `headless_shell_staging` bucket. To do this, use `gsutil rsync`:
-   ```sh
-   # This shows a preview of what would change by synchronizing the source scripts with the destination GCS bucket.
-   # Remove the `-n` flag to enact the changes
-   gsutil -m rsync -n -r x-pack/build_chromium gs://headless_shell_staging/build_chromium
-   ```
+Fortunately, creating the custom builds is only necessary for Linux. When you have a build
+of Kibana for Linux, or if you use a Linux desktop to develop Kibana, you have a copy of
+`headless_shell` bundled inside. When you have a Windows or Mac build of Kibana, or use
+either of those for development, you have a copy of the full build of Chromium, which
+was downloaded from the main [Chromium download
+location](https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html).
 
 ## Build Script Usage
 
@@ -63,170 +50,72 @@ the Chromium repo to be cloned successfully. If checking out the Chromium fails
 with "early EOF" errors, the instance could be low on memory or disk space.
 
 ## Getting the Commit Hash
+
 If you need to bump the version of Puppeteer, you need to get a new git commit hash for Chromium that corresponds to the Puppeteer version.
 ```
 node x-pack/dev-tools/chromium_version.js [PuppeteerVersion]
 ```
 
-When bumping the Puppeteer version, make sure you also update the `.chromium-commit` file with the commit hash
-for the current Chromium build, so we'll be able to construct a build pipeline for each OS (coming soon!).
+When bumping the Puppeteer version, make sure you also update the `ChromiumArchivePaths.revision` variable in
+`x-pack/plugins/reporting/server/browsers/chromium/paths.ts`.
 
 ## Build args
 
 A good how-to on building Chromium from source is
 [here](https://chromium.googlesource.com/chromium/src/+/master/docs/get_the_code.md).
 
-There are documents for each OS that will explain how to customize arguments
-for the build using the `gn` tool. Those instructions do not apply for the
-Kibana Chromium build. Our `build.py` script ensure the correct `args.gn`
-file gets used for build arguments.
-
-We have an `args.gn` file per platform:
-
-- mac: `darwin/args.gn`
-- linux 64bit: `linux-x64/args.gn`
-- windows: `windows/args.gn`
-- ARM 64bit: linux-aarch64/args.gn
+We have an `linux/args.gn` file that is automatically copied to the build target directory.
 
 To get a list of the build arguments that are enabled, install `depot_tools` and run
 `gn args out/headless --list`. It prints out all of the flags and their
-settings, including the defaults.
-
-The various build flags are not well documented. Some are documented
-[here](https://www.chromium.org/developers/gn-build-configuration). 
-
-As of this writing, there is an officially supported headless Chromium build
-args file for Linux: `build/args/headless.gn`. This does not work on Windows or
-Mac, so we have taken that as our starting point, and modified it until the
-Windows / Mac builds succeeded.
+settings, including the defaults. Some build flags are documented
+[here](https://www.chromium.org/developers/gn-build-configuration).
 
 **NOTE:** Please, make sure you consult @elastic/kibana-security before you change, remove or add any of the build flags.
 
-## Building locally
-
-You can skip the step of running `<os_name>/init.sh` for your OS if you already
-have your environment set up, and the chromium source cloned.
-
-To get the Chromium code, refer to the [documentation](https://chromium.googlesource.com/chromium/src/+/master/docs/get_the_code.md).
-Install `depot_tools` as suggested, since it comes with useful scripts. Use the
-`fetch` command to clone the chromium repository. To set up and run the build,
-use the Kibana `build.py` script (in this directory).
-
-It's recommended that you create a working directory for the chromium source
-code and all the build tools, and run the commands from there:
-```sh
-mkdir ~/chromium && cd ~/chromium
-cp -r ~/path/to/kibana/x-pack/build_chromium .
-python ./build_chromium/init.sh [arch_name]
-python ./build_chromium/build.py <commit_id>
-```
-
-## VMs
-
-I ran Linux and Windows VMs in GCP with the following specs:
-
-- 8 core vCPU
-- 30GB RAM
-- 128GB hard drive
-- Ubuntu 18.04 LTS (not minimal)
-- Windows Server 2016 (full, with desktop)
-
-The more cores the better, as the build makes effective use of each. For Linux, Ubuntu is the only officially supported build target.
-
-- Linux:
-  - SSH in using [gcloud](https://cloud.google.com/sdk/)
-  - Get the ssh command in the [GCP console](https://console.cloud.google.com/) -> VM instances -> your-vm-name -> SSH -> "View gcloud command"
-  - Their in-browser UI is kinda sluggish, so use the commandline tool (Google Cloud SDK is required)
-
-- Windows:
-  - Install Microsoft's Remote Desktop tools
-  - Get the RDP file in the [GCP console](https://console.cloud.google.com/) -> VM instances -> your-vm-name -> RDP -> Download the RDP file
-  - Edit it in Microsoft Remote Desktop:
-    - Display -> Resolution (1280 x 960 or something reasonable)
-    - Local Resources -> Folders, then select the folder(s) you want to share, this is at least `build_chromium` folder
-    - Save
-
-## Initializing each VM / environment
+## Directions for Elasticians
 
-In a VM, you'll want to use the init scripts to initialize each environment.
-On Mac OS you'll need to install XCode and accept the license agreement.
+If you wish to use a remote VM to build, you'll need access to our GCP account.
 
-Create the build folder:
-
-- Mac / Linux: `mkdir -p ~/chromium`
-- Windows: `mkdir c:\chromium`
-
-Copy the `x-pack/build-chromium` folder to each. Replace `you@your-machine` with the correct username and VM name:
-
-- Mac: `cp -r x-pack/build_chromium ~/chromium/build_chromium`
-- Linux: `gcloud compute scp --recurse x-pack/build_chromium you@your-machine:~/chromium/ --zone=us-east1-b --project "XXXXXXXX"`
-- Windows: Copy the `build_chromium` folder via the RDP GUI into `c:\chromium\build_chromium`
-
-There is an init script for each platform. This downloads and installs the necessary prerequisites, sets environment variables, etc.
-
-- Mac x64: `~/chromium/build_chromium/darwin/init.sh`
-- Linux x64: `~/chromium/build_chromium/linux/init.sh`
-- Linux arm64: `~/chromium/build_chromium/linux/init.sh arm64`
-- Windows x64: `c:\chromium\build_chromium\windows\init.bat`
-
-In windows, at least, you will need to do a number of extra steps:
-
-- Follow the prompts in the Visual Studio installation process, click "Install" and wait a while
-- Once it's installed, open Control Panel and turn on Debugging Tools for Windows:
-  - Control Panel → Programs → Programs and Features → Select the “Windows Software Development Kit” → Change → Change → Check “Debugging Tools For Windows” → Change
-- Press enter in the terminal to continue running the init
-
-## Building
-
-Note: In Linux, you should run the build command in tmux so that if your ssh session disconnects, the build can keep going. To do this, just type `tmux` into your terminal to hop into a tmux session. If you get disconnected, you can hop back in like so:
-
-- SSH into the server
-- Run `tmux list-sessions`
-- Run `tmux switch -t {session_id}`, replacing {session_id} with the value from the list-sessions output
-
-To run the build, replace the sha in the following commands with the sha that you wish to build:
+**NOTE:** The builds should be done in Ubuntu on x64 architecture. ARM builds
+are created in x64 using cross-compiling. CentOS is not supported for building Chromium.
 
-- Mac x64: `python ~/chromium/build_chromium/build.py 312d84c8ce62810976feda0d3457108a6dfff9e6`
-- Linux x64: `python ~/chromium/build_chromium/build.py 312d84c8ce62810976feda0d3457108a6dfff9e6`
-- Linux arm64: `python ~/chromium/build_chromium/build.py 312d84c8ce62810976feda0d3457108a6dfff9e6 arm64`
-- Windows x64: `python c:\chromium\build_chromium\build.py 312d84c8ce62810976feda0d3457108a6dfff9e6`
+1. Login to Google Cloud Console
+2. Click the "Compute Engine" tab.
+3. Create a Linux VM:
+   - 8 CPU
+   - 30GB memory
+   - 80GB free space on disk (Try `ncdu /home` to see where space is used.)
+   - git
+   - python2 (`python` must link to `python2`)
+   - lsb_release
+   - tmux is recommended in case your ssh session is interrupted
+   - "Cloud API access scopes": must have **read / write** scope for the Storage API
+4. Install [Google Cloud SDK](https://cloud.google.com/sdk) locally to ssh into the GCP instance
 
 ## Artifacts
 
 After the build completes, there will be a .zip file and a .md5 file in `~/chromium/chromium/src/out/headless`. These are named like so: `chromium-{first_7_of_SHA}-{platform}-{arch}`, for example: `chromium-4747cc2-linux-x64`.
+The zip files and md5 files are copied to a staging bucket in GCP storage.
 
-The zip files need to be deployed to GCP Storage. For testing, I drop them into `headless-shell-dev`, but for production, they need to be in `headless-shell`. And the `x-pack/plugins/reporting/server/browsers/chromium/paths.ts` file needs to be upated to have the correct `archiveChecksum`, `archiveFilename`, `binaryChecksum` and `baseUrl`. Below is a list of what the archive's are:
-
-- `archiveChecksum`: The contents of the `.md5` file, which is the `md5` checksum of the zip file.
-- `binaryChecksum`: The `md5` checksum of the `headless_shell` binary itself.
-
-*If you're building in the cloud, don't forget to turn off your VM after retrieving the build artifacts!*
-
-## Diagnosing runtime failures
-
-After getting the build to pass, the resulting binaries often failed to run or would hang.
-
-You can run the headless browser manually to see what errors it is generating (replace the `c:\dev\data` with the path to a dummy folder you've created on your system):
-
-**Mac**
-`headless_shell --disable-translate --disable-extensions --disable-background-networking --safebrowsing-disable-auto-update --disable-sync --metrics-recording-only --disable-default-apps --mute-audio --no-first-run --disable-gpu --no-sandbox --headless --hide-scrollbars --window-size=400,400 --remote-debugging-port=9221 https://example.com/`
-
-**Linux**
-`headless_shell --disable-translate --disable-extensions --disable-background-networking --safebrowsing-disable-auto-update --disable-sync --metrics-recording-only --disable-default-apps --mute-audio --no-first-run --disable-gpu --no-sandbox --headless --hide-scrollbars --window-size=400,400 --remote-debugging-port=9221 https://example.com/`
+## Testing
+Search the Puppeteer Github repo for known issues that could affect our use case, and make sure to test anywhere that is affected.
 
-**Windows**
-`headless_shell.exe --disable-translate --disable-extensions --disable-background-networking --safebrowsing-disable-auto-update --disable-sync --metrics-recording-only --disable-default-apps --mute-audio --no-first-run --disable-gpu --no-sandbox --headless --hide-scrollbars --window-size=400,400 --remote-debugging-port=9221 https://example.com/`
+Here's the steps on how to test a Puppeteer upgrade, run these tests on Mac, Windows, Linux x64 and Linux arm64:
 
-In the case of Windows, you can use IE to open `http://localhost:9221` and see if the page loads. In mac/linux you can just curl the JSON endpoints: `curl http://localhost:9221/json/list`.
+- Make sure the Reporting plugin is fetching the correct version of the browser
+  at start-up time, and that it can successfully unzip it and copy the files to
+  `x-pack/plugins/reporting/chromium`
+- Make sure there are no errors when using the **Reporting diagnostic tool**
+- All functional and API tests that generate PDF and PNG files should pass.
+- Use a VM to run Kibana in a low-memory environment and try to generate a PNG of a dashboard that outputs as a 4MB file. Document the minimum requirements in the PR.
 
 ## Resources
 
 The following links provide helpful context about how the Chromium build works, and its prerequisites:
 
 - Tools for Chromium version information: https://omahaproxy.appspot.com/
 - https://www.chromium.org/developers/how-tos/get-the-code/working-with-release-branches
-- https://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows_build_instructions.md
-- https://chromium.googlesource.com/chromium/src/+/HEAD/docs/mac_build_instructions.md
 - https://chromium.googlesource.com/chromium/src/+/HEAD/docs/linux/build_instructions.md
 - Some build-flag descriptions: https://www.chromium.org/developers/gn-build-configuration
 - The serverless Chromium project was indispensable: https://github.com/adieuadieu/serverless-chrome/blob/b29445aa5a96d031be2edd5d1fc8651683bf262c/packages/lambda/builds/chromium/build/build.sh

x-pack/build_chromium/build.py

@@ -6,7 +6,7 @@
   md5_file,
 )
 
-# This file builds Chromium headless on Windows, Mac, and Linux.
+# This file builds Chromium headless on Linux.
 
 # Verify that we have an argument, and if not print instructions
 if (len(sys.argv) < 2):
@@ -76,7 +76,7 @@
 runcmd('rm -rf out/headless')
 runcmd('mkdir out/headless')
 
-# Copy build args/{Linux | Darwin | Windows}.gn from the root of our directory to out/headless/args.gn,
+# Copy args.gn from the root of our directory to out/headless/args.gn,
 # add the target_cpu for cross-compilation
 print('Adding target_cpu to args')
 argsgn_file_out = path.abspath('out/headless/args.gn')
@@ -89,7 +89,7 @@
 print('Compiling... this will take a while')
 runcmd('autoninja -C out/headless headless_shell')
 
-# Optimize the output on Linux x64 and Mac by stripping inessentials from the binary
+# Optimize the output on Linux x64 by stripping inessentials from the binary
 # ARM must be cross-compiled from Linux and can not read the ARM binary in order to strip
 if platform.system() != 'Windows' and arch_name != 'arm64':
   print('Optimizing headless_shell')
@@ -112,30 +112,10 @@ def archive_file(name):
   archive.write(from_path, to_path)
   return to_path
 
-# Each platform has slightly different requirements for what dependencies
-# must be bundled with the Chromium executable.
-if platform.system() == 'Linux':
-  archive_file('headless_shell')
-  archive_file(path.join('swiftshader', 'libEGL.so'))
-  archive_file(path.join('swiftshader', 'libGLESv2.so'))
-
-  if arch_name == 'arm64':
-    archive_file(path.join('swiftshader', 'libEGL.so'))
-
-elif platform.system() == 'Windows':
-  archive_file('headless_shell.exe')
-  archive_file('dbghelp.dll')
-  archive_file('icudtl.dat')
-  archive_file(path.join('swiftshader', 'libEGL.dll'))
-  archive_file(path.join('swiftshader', 'libEGL.dll.lib'))
-  archive_file(path.join('swiftshader', 'libGLESv2.dll'))
-  archive_file(path.join('swiftshader', 'libGLESv2.dll.lib'))
-
-elif platform.system() == 'Darwin':
-  archive_file('headless_shell')
-  archive_file('libswiftshader_libEGL.dylib')
-  archive_file('libswiftshader_libGLESv2.dylib')
-  archive_file(path.join('Helpers', 'chrome_crashpad_handler'))
+# Add dependencies that must be bundled with the Chromium executable.
+archive_file('headless_shell')
+archive_file(path.join('swiftshader', 'libEGL.so'))
+archive_file(path.join('swiftshader', 'libGLESv2.so'))
 
 archive.close()
 

x-pack/build_chromium/init.py

@@ -3,11 +3,9 @@
 from build_util import runcmd, mkdir
 
 # This is a cross-platform initialization script which should only be run
-# once per environment, and isn't intended to be run directly. You should
-# run the appropriate platform init script (e.g. Linux/init.sh) which will
-# call this once the platform-specific initialization has completed.
+# once per environment.
 
-# Set to "arm" to build for ARM on Linux
+# Set to "arm" to build for ARM
 arch_name = sys.argv[1] if len(sys.argv) >= 2 else 'undefined'
 build_path = path.abspath(os.curdir)
 src_path = path.abspath(path.join(build_path, 'chromium', 'src'))
@@ -23,7 +21,6 @@
 runcmd('git config --global core.compression 0')
 
 # Grab Chromium's custom build tools, if they aren't already installed
-# (On Windows, they are installed before this Python script is run)
 # Put depot_tools on the path so we can properly run the fetch command
 if not path.isdir('depot_tools'):
   print('Installing depot_tools...')