Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INFRA] Parallelize the Buildkite test job on PR #7197

Merged
merged 12 commits into from
Sep 20, 2023
Merged

[INFRA] Parallelize the Buildkite test job on PR #7197

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
1d24b58
Refactoring test Bash script for parallelization.
1Copenut Sep 18, 2023
f5442c7
Splitting test tasks linting and unit testing.
1Copenut Sep 18, 2023
69fd82c
Moved agents declaration into individual steps.
1Copenut Sep 18, 2023
2fb5747
Was missing the double ampersand to run Yarn commands.
1Copenut Sep 18, 2023
0902f9a
Restoring Cypress install to all commands.
1Copenut Sep 18, 2023
7967c91
Adjusting conditional array adds to be one string each.
1Copenut Sep 19, 2023
3c4493c
Removed double quote literals for bash -c command.
1Copenut Sep 19, 2023
4d04d63
Updating all bash strings after successful lint run. Adding one Cypre…
1Copenut Sep 19, 2023
d2a29f4
Adding Cypress runs for React 16 and 17.
1Copenut Sep 19, 2023
2818174
Refactoring test pipeline to switch case for readability.
1Copenut Sep 19, 2023
779c584
Labeling steps for task recognition.
1Copenut Sep 19, 2023
09fe4e8
Updating comment about branch filter.
1Copenut Sep 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 38 additions & 3 deletions .buildkite/pipelines/pipeline_pull_request_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,44 @@
# 🏠/.buildkite/pipelines/pipeline_pull_request_test.yml

steps:
- agents:
- command: .buildkite/scripts/pipeline_test.sh
label: ":typescript: Linting"
agents:
provider: "gcp"
command: .buildkite/scripts/pipeline_test.sh
env:
TEST_TYPE: 'lint'
if: build.branch != "main" # We're skipping testing commits in main for now to maintain parity with previous Jenkins setup
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My only conceit on this PR is to make the Docker image smaller, because each command is run by a separate agent with no knowledge of other agents. I'm seeing several opportunities to reduce the size and speed up our Docker image caching. I'll roll that work into removing the Puppeteer step from Dockerfile later this week or next.

- command: .buildkite/scripts/pipeline_test.sh
label: ":jest: Unit tests"
agents:
provider: "gcp"
env:
TEST_TYPE: 'unit'
if: build.branch != "main" # We're skipping testing commits in main for now to maintain parity with previous Jenkins setup
- command: .buildkite/scripts/pipeline_test.sh
label: ":cypress: Cypress tests on React 16"
agents:
provider: "gcp"
env:
TEST_TYPE: 'cypress:16'
if: build.branch != "main" # We're skipping testing commits in main for now to maintain parity with previous Jenkins setup
artifact_paths:
- "cypress/react16/screenshots/**/*.png"
- command: .buildkite/scripts/pipeline_test.sh
label: ":cypress: Cypress tests on React 17"
agents:
provider: "gcp"
env:
TEST_TYPE: 'cypress:17'
if: build.branch != "main" # We're skipping testing commits in main for now to maintain parity with previous Jenkins setup
artifact_paths:
- "cypress/react17/screenshots/**/*.png"
- command: .buildkite/scripts/pipeline_test.sh
label: ":cypress: Cypress tests on React 18"
agents:
provider: "gcp"
env:
TEST_TYPE: 'cypress:18'
if: build.branch != "main" # We're skipping testing commits in main for now to maintain parity with previous Jenkins setup
artifact_paths:
- "cypress/screenshots/**/*.png"
- "cypress/react18/screenshots/**/*.png"
57 changes: 45 additions & 12 deletions .buildkite/scripts/pipeline_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,48 @@

set -euo pipefail

docker run \
-i --rm \
--env GIT_COMMITTER_NAME=test \
--env GIT_COMMITTER_EMAIL=test \
--env HOME=/tmp \
--user="$(id -u):$(id -g)" \
--volume="$(pwd):/app" \
--workdir=/app \
docker.elastic.co/eui/ci:5.3 \
bash -c "/opt/yarn*/bin/yarn \
&& yarn cypress install \
&& NODE_OPTIONS=\"--max-old-space-size=2048\" npm run test-ci"
DOCKER_OPTIONS=(
-i --rm
--env GIT_COMMITTER_NAME=test
--env GIT_COMMITTER_EMAIL=test
--env HOME=/tmp
--user="$(id -u):$(id -g)"
--volume="$(pwd):/app"
--workdir=/app
docker.elastic.co/eui/ci:5.3
)

case $TEST_TYPE in
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally I used if / elif / fi blocks but it was getting hard to read. This felt cleaner and more scalable.

lint)
echo "[TASK]: Running linters"
DOCKER_OPTIONS+=(bash -c "/opt/yarn*/bin/yarn && yarn cypress install && NODE_OPTIONS=\"--max-old-space-size=2048\" yarn lint")
;;

unit)
echo "[TASK]: Running unit tests"
DOCKER_OPTIONS+=(bash -c "/opt/yarn*/bin/yarn && yarn cypress install && NODE_OPTIONS=\"--max-old-space-size=2048\" yarn test-unit")
;;

cypress:16)
echo "[TASK]: Running Cypress tests against React 16"
DOCKER_OPTIONS+=(bash -c "/opt/yarn*/bin/yarn && yarn cypress install && NODE_OPTIONS=\"--max-old-space-size=2048\" yarn test-cypress --react-version 16")
;;

cypress:17)
echo "[TASK]: Running Cypress tests against React 17"
DOCKER_OPTIONS+=(bash -c "/opt/yarn*/bin/yarn && yarn cypress install && NODE_OPTIONS=\"--max-old-space-size=2048\" yarn test-cypress --react-version 17")
;;

cypress:18)
echo "[TASK]: Running Cypress tests against React 18"
DOCKER_OPTIONS+=(bash -c "/opt/yarn*/bin/yarn && yarn cypress install && NODE_OPTIONS=\"--max-old-space-size=2048\" yarn test-cypress")
;;

*)
echo "[ERROR]: Unknown task"
echo "Exit code: 1"
exit 1
;;
esac

docker run "${DOCKER_OPTIONS[@]}"
Loading