[Reporting] code cleanup for reporting browser build/install/setup utilities #98799

tsullivan · 2021-04-29T17:26:08Z

The main goals are to convert files to TypeScript, and to improve the Python build scripts. Some of the changes simply convert code to TypeScript.

x-pack/build_chromium/*: this area of code is responsible for creating the builds
x-pack/plugins/reporting/server/browsers: this area of code is responsible for downloading, unzipping, and copying the builds

Precedes #98688

tsullivan · 2021-04-29T17:52:10Z

x-pack/build_chromium/build.py

@@ -136,3 +145,6 @@ def archive_file(name):
 print('Creating ' + path.join(src_path, md5_filename))
 with open (md5_filename, 'w') as f:
  f.write(md5_file(zip_filename))
+
+runcmd('gsutil cp ' + path.join(src_path, zip_filename) + ' gs://headless_shell_staging')
+runcmd('gsutil cp ' + path.join(src_path, md5_filename) + ' gs://headless_shell_staging')


cc @spalger : this adds an auto-upload to a staging bucket as we talked about.

I think this suffices the problem we had here: #98688 (comment)

This is without adding a dry-run option to the script, as discussed. LMK if you think it needs the dry-run option. (It would be a pain since a run takes hours.)

Yeah, I think this is the best approach, then staging assets can just be gsutil mvd to the headless_shell bucket when they're verified.

Might be nice to use an environment variable to switch out the URLs to actually download from staging so we can test things before promoting them.

tsullivan · 2021-04-29T17:53:19Z

x-pack/build_chromium/darwin/args.gn

 use_libpci = false
 use_pulseaudio = false
 use_udev = false

 is_debug = false
 symbol_level = 0
 is_component_build = false
-remove_webcore_debug_symbols = true


remove_webcore_debug_symbols caused warning logs during compilation: it is unused

tsullivan · 2021-04-29T18:00:25Z

x-pack/plugins/reporting/server/browsers/chromium/paths.ts

-  archivesPath: path.resolve(__dirname, '../../../../../../.chromium'),
-  baseUrl: 'https://storage.googleapis.com/headless_shell/',
-  packages: [
+// We download zip files from a Kibana team GCS bucket named `headless_shell`


This file was almost entirely re-written because there is a lot of code that uses the object, and those contexts had no abstractions available to work with the structure of the data.

tsullivan · 2021-04-29T18:03:06Z

x-pack/plugins/reporting/server/browsers/download/checksum.ts


-import { readableEnd } from './util';


util was removed because it didn't contain anything significant and the code inside is easy enough to define in-line where it is needed.

tsullivan · 2021-04-29T19:29:25Z

x-pack/build_chromium/build.py

-f.close()
+argsgn_file_out = path.abspath('out/headless/args.gn')
+runcmd('cp ' + argsgn_file + ' ' + argsgn_file_out)
+runcmd('echo \'target_cpu="' + arch_name + '"\' >> ' + argsgn_file_out)


the f.write code was not working most of the time: seemed to be a race condition where the previous line of code would complete later than expected

tsullivan · 2021-04-29T20:27:48Z

x-pack/build_chromium/darwin/args.gn

-remove_webcore_debug_symbols = true
+
+# Please, consult @elastic/kibana-security before changing/removing this option.
+use_kerberos = false


moved these lines down to match the order of lines with ../windows/args.gn

tsullivan · 2021-04-29T20:30:16Z

x-pack/plugins/reporting/server/browsers/chromium/paths.ts

    {
-      platforms: ['darwin', 'freebsd', 'openbsd'],


Kibana doesn't support freebsd or openbsd

tsullivan · 2021-04-29T20:31:22Z

x-pack/plugins/reporting/server/browsers/download/clean.ts

-      await del(path, { force: true });
-    }
-  });
+  await Promise.all(


asyncMap was not that useful as a helper

elasticmachine · 2021-04-29T23:17:27Z

Pinging @elastic/kibana-app-services (Team:AppServices)

elasticmachine · 2021-04-29T23:17:28Z

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

spalger

Operations specific code LGTM

src/dev/chromium_version.ts

Co-authored-by: Michael Dokolin <[email protected]>

tsullivan · 2021-04-30T18:14:18Z

@elasticmachine merge upstream

tsullivan · 2021-04-30T20:19:57Z

@elasticmachine merge upstream

kibanamachine · 2021-04-30T23:25:15Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #123683 succeeded ef22598
💔 Build #123644 failed 395785a
💔 Build #123637 failed 571b1bd
💔 Build #123424 failed 222cb22
💚 Build #123394 succeeded 78a0642

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…ilities (elastic#98799) * [Reporting] code cleanup for reporting browser setup utilities * fix target_cpu * Update README.md * Update README.md * add note about target_cpu * Update paths.ts * more cleanup * Update src/dev/chromium_version.ts Co-authored-by: Michael Dokolin <[email protected]> * remove bug Co-authored-by: Michael Dokolin <[email protected]> Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # src/dev/code_coverage/ingest_coverage/__tests__/enumerate_patterns.test.js # src/dev/code_coverage/ingest_coverage/__tests__/mocks/team_assign_mock.txt # src/dev/code_coverage/ingest_coverage/__tests__/transforms.test.js

…tup utilities (#98799) (#98998) * [Reporting] code cleanup for reporting browser build/install/setup utilities (#98799) * [Reporting] code cleanup for reporting browser setup utilities * fix target_cpu * Update README.md * Update README.md * add note about target_cpu * Update paths.ts * more cleanup * Update src/dev/chromium_version.ts Co-authored-by: Michael Dokolin <[email protected]> * remove bug Co-authored-by: Michael Dokolin <[email protected]> Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # src/dev/code_coverage/ingest_coverage/__tests__/enumerate_patterns.test.js # src/dev/code_coverage/ingest_coverage/__tests__/mocks/team_assign_mock.txt # src/dev/code_coverage/ingest_coverage/__tests__/transforms.test.js * fix bad merge Co-authored-by: Kibana Machine <[email protected]>

tsullivan force-pushed the reporting/browser-install-cleanup branch 3 times, most recently from 73a0e00 to 9b502d3 Compare April 29, 2021 17:44

tsullivan mentioned this pull request Apr 29, 2021

[Reporting] Update Puppeteer to version 8.0.0 and Chromium to r856583 #98688

Merged

5 tasks

tsullivan force-pushed the reporting/browser-install-cleanup branch from 6843b65 to 0632ba1 Compare April 29, 2021 17:47

tsullivan commented Apr 29, 2021

View reviewed changes

tsullivan changed the title ~~[Reporting] code cleanup for reporting browser setup utilities~~ [Reporting] code cleanup for reporting browser build/install/setup utilities Apr 29, 2021

tsullivan commented Apr 29, 2021

View reviewed changes

[Reporting] code cleanup for reporting browser setup utilities

ceeb4d9

tsullivan force-pushed the reporting/browser-install-cleanup branch from 8b73fe0 to ceeb4d9 Compare April 29, 2021 18:24

tsullivan requested review from dokmic, tylersmalley and a team April 29, 2021 18:25

fix target_cpu

361bc1e

tsullivan commented Apr 29, 2021

View reviewed changes

tsullivan added 2 commits April 29, 2021 13:25

Update README.md

fa138c8

Update README.md

325ef7b

tsullivan commented Apr 29, 2021

View reviewed changes

add note about target_cpu

95dc744

Update paths.ts

78a0642

tsullivan marked this pull request as ready for review April 29, 2021 23:17

tsullivan requested review from a team as code owners April 29, 2021 23:17

spalger approved these changes Apr 29, 2021

View reviewed changes

more cleanup

222cb22

wayneseymour approved these changes Apr 30, 2021

View reviewed changes

dokmic reviewed Apr 30, 2021

View reviewed changes

src/dev/chromium_version.ts Outdated Show resolved Hide resolved

dokmic approved these changes Apr 30, 2021

View reviewed changes

tsullivan and others added 2 commits April 30, 2021 11:05

Update src/dev/chromium_version.ts

571b1bd

Co-authored-by: Michael Dokolin <[email protected]>

remove bug

1690b04

Merge branch 'master' into reporting/browser-install-cleanup

395785a

kibanamachine and others added 2 commits April 30, 2021 16:20

Merge branch 'master' into reporting/browser-install-cleanup

ef22598

Merge branch 'master' into reporting/browser-install-cleanup

ca531a9

tsullivan enabled auto-merge (squash) April 30, 2021 22:49

tsullivan merged commit 832d3b7 into elastic:master Apr 30, 2021

tsullivan deleted the reporting/browser-install-cleanup branch April 30, 2021 23:27

tsullivan mentioned this pull request Apr 30, 2021

[7.x] [Reporting] code cleanup for reporting browser build/install/setup utilities (#98799) #98998

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting] code cleanup for reporting browser build/install/setup utilities #98799

[Reporting] code cleanup for reporting browser build/install/setup utilities #98799

tsullivan commented Apr 29, 2021 •

edited

Loading

tsullivan Apr 29, 2021

spalger Apr 29, 2021

tsullivan Apr 29, 2021

tsullivan Apr 29, 2021 •

edited

Loading

tsullivan Apr 29, 2021

tsullivan Apr 29, 2021 •

edited

Loading

tsullivan Apr 29, 2021

tsullivan Apr 29, 2021

tsullivan Apr 29, 2021

elasticmachine commented Apr 29, 2021

elasticmachine commented Apr 29, 2021

spalger left a comment

tsullivan commented Apr 30, 2021

tsullivan commented Apr 30, 2021

kibanamachine commented Apr 30, 2021

[Reporting] code cleanup for reporting browser build/install/setup utilities #98799

[Reporting] code cleanup for reporting browser build/install/setup utilities #98799

Conversation

tsullivan commented Apr 29, 2021 • edited Loading

tsullivan Apr 29, 2021

Choose a reason for hiding this comment

spalger Apr 29, 2021

Choose a reason for hiding this comment

tsullivan Apr 29, 2021

Choose a reason for hiding this comment

tsullivan Apr 29, 2021 • edited Loading

Choose a reason for hiding this comment

tsullivan Apr 29, 2021

Choose a reason for hiding this comment

tsullivan Apr 29, 2021 • edited Loading

Choose a reason for hiding this comment

tsullivan Apr 29, 2021

Choose a reason for hiding this comment

tsullivan Apr 29, 2021

Choose a reason for hiding this comment

tsullivan Apr 29, 2021

Choose a reason for hiding this comment

elasticmachine commented Apr 29, 2021

elasticmachine commented Apr 29, 2021

spalger left a comment

Choose a reason for hiding this comment

tsullivan commented Apr 30, 2021

tsullivan commented Apr 30, 2021

kibanamachine commented Apr 30, 2021

💚 Build Succeeded

Metrics [docs]

History

tsullivan commented Apr 29, 2021 •

edited

Loading

tsullivan Apr 29, 2021 •

edited

Loading

tsullivan Apr 29, 2021 •

edited

Loading