Local build is broken

[atjn]

Hi :) I am trying to follow the guide on setting up local development of the wpt.fyi dashboard. I am running into a lot of problems and I have no idea how to proceed.

I have executed the exact commands detailed in: https://github.com/web-platform-tests/wpt.fyi/tree/f319b11328efa5312fb7f472ca9222df58e2fb9f?tab=readme-ov-file#development
The only exception is that I need to run sudo in front of the commands, otherwise Docker won't run. I cannot see how that would create the following errors, I run many other Docker containers this way with no problems.

Error 1

Git decides to crash because the directory /home/user/wpt.fyi does not have the correct owner. I fixed this temporarily by adding the following to /Makefile:

@@ -56,6 +56,7 @@ go_build: git mockgen packr2 github_action_go_setup
        make webapp_node_modules_prod
        go generate ./...
        # Check all packages without producing any output.
+       git config --global --add safe.directory /home/user/wpt.fyi
        go build -v ./...
        # Build the webapp.
        go build -v ./webapp/web

Error 2

The Google cloud emulator crashes because it cannot find env.yaml in the data_dir. I fixed this by adding the following to util/docker-dev/web_server.sh:

@@ -21,4 +21,4 @@ if [ "${DOCKER_STATUS}" != "0" ]; then
 fi
 
 info "Starting web server. Port forwarded to host: ${WPTD_HOST_WEB_PORT}"
-wptd_exec_it "\$(gcloud beta emulators datastore env-init) && util/server-watch.sh"
+wptd_exec_it "\$(gcloud config set project wptdashboard) && \$(gcloud beta emulators datastore start) && \$(gcloud beta emulators datastore env-init) && util/server-watch.sh"

Error 3

At this point it seems like the setup script runs correctly, but trying to access loclhost:8080 or localhost:8081 results in a connection reset error. I do not know how to fix that.

My operating system: Linux, Ubuntu 22.04 LTS
CPU: Modern Intel AMD64

I hope you can help me figure out what is wrong. I am happy to share any other details.

Here is the full log of what I did to see error 3 after I had applied the fixes for error 1 and 2.

Full log

atjn:~/Documents/GitHub/wpt.fyi$ sudo ./util/docker-dev/run.sh
[sudo] password for atjn: 
[  2024-05-25 23:03:18  INFO  ]  Creating docker instance for dev server. Instance name: wptd-dev-instance
[  2024-05-25 23:03:18  WARN  ]  Found existing docker instance wptd-dev-instance. Remove?
Confirm (y/N): y
[  2024-05-25 23:03:21  INFO  ]  Stopping wptd-dev-instance...
wptd-dev-instance
[  2024-05-25 23:03:31  INFO  ]  wptd-dev-instance stopped.
[  2024-05-25 23:03:31  INFO  ]  Removing wptd-dev-instance...
wptd-dev-instance
[  2024-05-25 23:03:32  INFO  ]  wptd-dev-instance removed.
[  2024-05-25 23:03:32  INFO  ]  Starting docker instance wptd-dev-instance...
8ebfed1339063c589a1f3aa8ec73044d08def38739d615e97bd425da2bd12fae
[  2024-05-25 23:03:33  INFO  ]  Setting up local user
groupadd: GID '0' already exists
useradd: UID 0 is not unique
atjn:~/Documents/GitHub/wpt.fyi$ sudo ./util/docker-dev/web_server.sh
if [[ "$(which inotifywait)" == "" ]]; then \
	sudo apt-get install -qqy --no-install-suggests inotify-tools; \
fi
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libinotifytools0:amd64.
(Reading database ... 84304 files and directories currently installed.)
Preparing to unpack .../libinotifytools0_3.14-8.1_amd64.deb ...
Unpacking libinotifytools0:amd64 (3.14-8.1) ...
Selecting previously unselected package inotify-tools.
Preparing to unpack .../inotify-tools_3.14-8.1_amd64.deb ...
Unpacking inotify-tools (3.14-8.1) ...
Setting up libinotifytools0:amd64 (3.14-8.1) ...
Setting up inotify-tools (3.14-8.1) ...
Processing triggers for libc-bin (2.31-13+deb11u7) ...
[  2024-05-25 23:04:30  INFO  ]  Building web server...
if [[ "$(which git)" == "" ]]; then sudo apt-get install -qqy --no-install-suggests git; fi
if [ "$(which mockgen)" == "" ]; then \
	go install github.com/golang/mock/mockgen; \
fi
go: downloading github.com/golang/mock v1.6.0
go: downloading golang.org/x/mod v0.8.0
go: downloading golang.org/x/tools v0.6.0
go: downloading golang.org/x/sys v0.20.0
if [ "$(which packr2)" == "" ]; then \
	go install github.com/gobuffalo/packr/v2/packr2; \
fi
go: downloading github.com/gobuffalo/packr/v2 v2.8.3
go: downloading github.com/gobuffalo/logger v1.0.7
go: downloading github.com/spf13/cobra v1.6.1
go: downloading github.com/gobuffalo/packd v1.0.2
go: downloading github.com/sirupsen/logrus v1.9.3
go: downloading github.com/markbates/oncer v1.0.0
go: downloading github.com/markbates/safe v1.0.1
go: downloading github.com/karrick/godirwalk v1.17.0
go: downloading github.com/markbates/errx v1.1.0
go: downloading github.com/rogpeppe/go-internal v1.9.0
go: downloading golang.org/x/sync v0.7.0
go: downloading golang.org/x/term v0.20.0
go: downloading github.com/spf13/pflag v1.0.5
# https://github.com/web-platform-tests/wpt.fyi/issues/3089
if [ -d "/github/workspace" ]; then \
	echo "Avoiding buildvcs error for Go 1.18+ by marking github workspace safe."; \
	git config --global --add safe.directory /github/workspace ; \
else \
	echo "Did not detect github workspace. Skipping." ; \
fi
Did not detect github workspace. Skipping.
make webapp_node_modules_prod
make[1]: Entering directory '/home/user/wpt.fyi'
if [[ "$(which curl)" == "" ]]; then sudo apt-get install -qqy --no-install-suggests curl; fi
if [[ "$(which gpg)" == "" ]]; then \
	sudo apt-get install -qqy --no-install-suggests gnupg; \
fi
if [[ "$(which node)" == "" ]]; then \
	curl -sL https://deb.nodesource.com/setup_18.x | sudo -E bash -; \
	sudo apt-get install -qqy nodejs; \
fi
cd webapp; npm install
npm WARN skipping integrity check for git dependency ssh://[email protected]/web-padawan/launchpad.git 
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated [email protected]: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated [email protected]: Package no longer supported. Contact [email protected] for more info.
npm WARN deprecated [email protected]: Multer 1.x is affected by CVE-2022-24434. This is fixed in v1.4.4-lts.1 which drops support for versions of Node.js before 6. Please upgrade to at least Node.js 6 and version 1.4.4-lts.1 of Multer. If you need support for older versions of Node.js, we are open to accepting patches that would fix the CVE on the main 1.x release line, whilst maintaining compatibility with Node.js 0.10.
npm WARN deprecated [email protected]: This package has been deprecated in favour of @sinonjs/samsam
npm WARN deprecated [email protected]: This package is unmaintained. Use @sinonjs/formatio instead
npm WARN deprecated [email protected]: This package is unmaintained. Use @sinonjs/formatio instead
npm WARN deprecated [email protected]: Deprecated due to CVE-2021-21366 resolved in 0.5.0
npm WARN deprecated [email protected]: This package has been deprecated in favour of @sinonjs/samsam
npm WARN deprecated [email protected]: Please use the native JSON object instead of JSON 3
npm WARN deprecated [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated [email protected]: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: Please migrate to Workbox: https://developers.google.com/web/tools/workbox/guides/migrations/migrate-from-sw
npm WARN deprecated [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated [email protected]: Please migrate to Workbox: https://developers.google.com/web/tools/workbox/guides/migrations/migrate-from-sw
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: no longer maintained
npm WARN deprecated [email protected]: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.

> postinstall
> cpy 'node_modules/@browser-logos/{chrome,chrome-beta,chrome-canary,chrome-dev,chromium,deno,edge,edge-beta,edge-canary,edge-dev,firefox,firefox-beta,firefox-nightly,geckoview,node.js,safari,servo,uc}/*_64x64.png' static && cpy 'node_modules/@browser-logos/firefox-developer-edition/*_64x64.png' static --rename=firefox-dev_64x64.png && cpy 'node_modules/@browser-logos/safari/*_64x64.png' static --rename=safari-beta_64x64.png && cpy 'node_modules/@browser-logos/safari-technology-preview/*_64x64.png'  static --rename=safari-dev_64x64.png && cpy 'node_modules/@browser-logos/safari-technology-preview/*_64x64.png' static --rename=safari-preview_64x64.png && cpy 'node_modules/@browser-logos/webkit-nightly/*_64x64.png' static --rename=wktr_64x64.png


added 1328 packages, and audited 1409 packages in 18s

108 packages are looking for funding
  run `npm fund` for details

48 vulnerabilities (5 low, 10 moderate, 19 high, 14 critical)

To address issues that do not require attention, run:
  npm audit fix

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
npm notice 
npm notice New minor version of npm available! 10.5.0 -> 10.8.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.8.0>
npm notice Run `npm install -g [email protected]` to update!
npm notice 
cd webapp; npm prune --production
npm WARN config production Use `--omit=dev` instead.

removed 1 package, and audited 81 packages in 2s

found 0 vulnerabilities
make[1]: Leaving directory '/home/user/wpt.fyi'
go generate ./...
go: downloading github.com/google/go-github/v47 v47.1.0
go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible
go: downloading golang.org/x/oauth2 v0.20.0
go: downloading github.com/gorilla/mux v1.8.1
go: downloading github.com/deckarep/golang-set v1.8.0
go: downloading cloud.google.com/go/datastore v1.17.0
go: downloading google.golang.org/api v0.180.0
go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13
go: downloading github.com/stretchr/testify v1.9.0
go: downloading cloud.google.com/go/storage v1.41.0
go: downloading github.com/taskcluster/taskcluster-lib-urls v13.0.1+incompatible
go: downloading github.com/taskcluster/taskcluster/v44 v44.23.4
go: downloading github.com/google/uuid v1.6.0
go: downloading cloud.google.com/go v0.113.0
go: downloading github.com/phayes/freeport v0.0.0-20220201140144-74d24b5ae9f5
go: downloading github.com/samthor/nicehttp v1.0.0
go: downloading cloud.google.com/go/cloudtasks v1.12.8
go: downloading cloud.google.com/go/logging v1.10.0
go: downloading cloud.google.com/go/secretmanager v1.13.0
go: downloading github.com/gomodule/redigo v1.9.2
go: downloading github.com/gorilla/handlers v1.5.2
go: downloading github.com/gorilla/securecookie v1.1.2
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240513163218-0867130af1f8
go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda
go: downloading github.com/tebeka/selenium v0.9.9
go: downloading google.golang.org/grpc v1.64.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/google/go-querystring v1.1.0
go: downloading golang.org/x/crypto v0.23.0
go: downloading github.com/googleapis/gax-go/v2 v2.12.4
go: downloading google.golang.org/protobuf v1.34.1
go: downloading cloud.google.com/go/compute/metadata v0.3.0
go: downloading cloud.google.com/go/iam v1.1.8
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/felixge/httpsnoop v1.0.4
go: downloading github.com/blang/semver v3.5.1+incompatible
go: downloading golang.org/x/net v0.25.0
go: downloading go.opencensus.io v0.24.0
go: downloading go.opentelemetry.io/otel v1.24.0
go: downloading go.opentelemetry.io/otel/trace v1.24.0
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240513163218-0867130af1f8
go: downloading cloud.google.com/go/longrunning v0.5.7
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading github.com/go-logr/logr v1.4.1
go: downloading go.opentelemetry.io/otel/metric v1.24.0
go: downloading github.com/go-logr/stdr v1.2.2
go: downloading golang.org/x/text v0.15.0
go: downloading google.golang.org/api v0.180.0
go: downloading cloud.google.com/go/auth v0.4.1
go: downloading cloud.google.com/go/auth/oauth2adapt v0.2.2
go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0
go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0
go: downloading golang.org/x/time v0.5.0
go: downloading github.com/google/s2a-go v0.1.7
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2
go: downloading github.com/golang/protobuf v1.5.4
go: downloading github.com/taskcluster/taskcluster/v44 v44.23.4
go: downloading github.com/taskcluster/httpbackoff/v3 v3.1.0
go: downloading github.com/cenkalti/backoff/v3 v3.2.2
go: downloading github.com/tent/hawk-go v0.0.0-20161026210932-d341ea318957
go: downloading github.com/orcaman/writerseeker v0.0.0-20200621085525-1d3f536ff85e
go: downloading github.com/taskcluster/slugid-go v1.1.0
go: downloading github.com/johncgriffin/overflow v0.0.0-20211019200055-46fa312c352c
go: downloading github.com/fatih/camelcase v1.0.0
go: downloading github.com/pborman/uuid v1.2.1
# Check all packages without producing any output.
git config --global --add safe.directory /home/user/wpt.fyi
go build -v ./...
net/http/internal/testcert
internal/sysinfo
github.com/dgryski/go-farm
github.com/web-platform-tests/wpt.fyi/api/query/cache/lru
github.com/tebeka/selenium/log
github.com/golang/mock/gomock
github.com/pmezard/go-difflib/difflib
github.com/davecgh/go-spew/spew
net/http/httptest
github.com/web-platform-tests/wpt.fyi/api/query
testing
github.com/web-platform-tests/wpt.fyi/shared/metrics
github.com/web-platform-tests/wpt.fyi/api/screenshot
github.com/phayes/freeport
github.com/web-platform-tests/wpt.fyi/util
github.com/web-platform-tests/wpt.fyi/api/azure/mock_azure
github.com/web-platform-tests/wpt.fyi/api/checks/mock_checks
github.com/web-platform-tests/wpt.fyi/api/manifest/mock_manifest
github.com/web-platform-tests/wpt.fyi/api/query/cache/backfill/mock_backfill
github.com/stretchr/testify/assert
github.com/web-platform-tests/wpt.fyi/webapp/packrd
github.com/web-platform-tests/wpt.fyi/api/receiver/mock_receiver
github.com/web-platform-tests/wpt.fyi/api/taskcluster/mock_taskcluster
github.com/web-platform-tests/wpt.fyi/api
github.com/web-platform-tests/wpt.fyi/api/query/cache/index
github.com/web-platform-tests/wpt.fyi/api/query/cache/query
gopkg.in/yaml.v2
github.com/blang/semver
archive/zip
github.com/web-platform-tests/wpt.fyi/api/query/cache/monitor
github.com/web-platform-tests/wpt.fyi/api/query/cache/poll
github.com/web-platform-tests/wpt.fyi/api/query/cache/backfill
github.com/web-platform-tests/wpt.fyi/api/query/test
github.com/web-platform-tests/wpt.fyi/shared/sharedtest
github.com/tebeka/selenium/internal/zip
github.com/tebeka/selenium/chrome
github.com/tebeka/selenium/firefox
github.com/web-platform-tests/wpt.fyi/api/query/cache/service
github.com/tebeka/selenium
github.com/samthor/nicehttp
github.com/web-platform-tests/wpt.fyi/webdriver
github.com/web-platform-tests/wpt.fyi/webapp
github.com/web-platform-tests/wpt.fyi/webapp/web
# Build the webapp.
go build -v ./webapp/web
[  2024-05-25 23:05:54  INFO  ]  Starting web server. Port forwarded to host: 8080
Updated property [core/project].
Executing: /usr/lib/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator create --project_id=wptdashboard /root/.config/gcloud/emulators/datastore
[datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.CloudDatastore$CreateAction$1 apply
[datastore] INFO: Provided project_id to Cloud Datastore emulator create command, which is no longer necessary.
[datastore] Created new Cloud Datastore project in '/root/.config/gcloud/emulators/datastore'.
Executing: /usr/lib/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=localhost --port=8081 --store_on_disk=True --allow_remote_shutdown --consistency=0.9 /root/.config/gcloud/emulators/datastore
[datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.CloudDatastore$FakeDatastoreAction$9 apply
[datastore] INFO: Provided --allow_remote_shutdown to start command which is no longer necessary.
[datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.CloudDatastore$StartAction apply
[datastore] INFO: Running CLOUD_DATASTORE emulator.
[datastore] May 25, 2024 9:05:56 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub <init>
[datastore] INFO: Local Datastore initialized:
[datastore] 	Type: High Replication
[datastore] 	Storage: /root/.config/gcloud/emulators/datastore/WEB-INF/appengine-generated/local_db.bin
[datastore] May 25, 2024 9:05:57 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub load
[datastore] INFO: The backing store, /root/.config/gcloud/emulators/datastore/WEB-INF/appengine-generated/local_db.bin, does not exist. It will be created.
[datastore] API endpoint: http://localhost:8081
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore] 
[datastore]   export DATASTORE_EMULATOR_HOST=localhost:8081
[datastore] 
[datastore] Dev App Server is now running.
[datastore] 
[datastore] The previous line was printed for backwards compatibility only.
[datastore] If your tests rely on it to confirm emulator startup,
[datastore] please migrate to the emulator health check endpoint (/). Thank you!
[datastore] The health check endpoint for this emulator instance is http://localhost:8081/