Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Production Deploy #3457

Merged
merged 103 commits into from
Nov 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
026fb66
Update workers test data
arkid15r Sep 12, 2023
10ab7d4
Update post downloads summary job
arkid15r Sep 12, 2023
21f453d
Update build badge status
arkid15r Sep 13, 2023
f702947
Merge pull request #3383 from AlexsLemonade/ark/3379-fix-smasher-tests
arkid15r Sep 13, 2023
48a7649
Merge pull request #3384 from AlexsLemonade/ark/update-build-status-b…
arkid15r Sep 13, 2023
dbc6464
Merge pull request #3385 from AlexsLemonade/ark/3382-weekly-stats-update
arkid15r Sep 13, 2023
1085869
Implement data request tracking
arkid15r Sep 15, 2023
2ba36c0
Update common/foreman scripts
arkid15r Sep 15, 2023
928acef
Extend using of staging images cache
arkid15r Sep 15, 2023
ece29d6
Merge pull request #3388 from AlexsLemonade/ark/update-sh-scripts
arkid15r Sep 21, 2023
40e8463
Merge pull request #3389 from AlexsLemonade/ark/3380-use-docker-cache…
arkid15r Sep 21, 2023
6294e5f
Use 1Password as a secrets storage
arkid15r Sep 21, 2023
e54872e
Merge branch 'dev' into ark/3361-tracking-data-requests
arkid15r Sep 25, 2023
cac1ead
Address PR review comments
arkid15r Sep 25, 2023
bc365b2
Merge pull request #3387 from AlexsLemonade/ark/3361-tracking-data-re…
arkid15r Sep 26, 2023
85b7774
Merge branch 'dev' into ark/1password-integration
arkid15r Sep 26, 2023
439899b
Customize Sentry environment tag
arkid15r Sep 26, 2023
3cd73b0
Merge pull request #3391 from AlexsLemonade/ark/1password-integration
arkid15r Sep 27, 2023
3374b30
show taxonomy id on exception
davidsmejia Sep 27, 2023
67efe9b
add species taxonomy id
davidsmejia Sep 27, 2023
d3c19d0
add helpers for transcriptome index
davidsmejia Sep 27, 2023
95a578b
switch to http from ftp
davidsmejia Sep 27, 2023
cd44faa
use requests and helper
davidsmejia Sep 27, 2023
a41a069
add fallbacks for different filenames
davidsmejia Sep 27, 2023
d476f8e
add missing session variable
davidsmejia Sep 27, 2023
4213147
fetch bacteria fungi by assembly assign species_taxonomy_id
davidsmejia Sep 27, 2023
bada128
update tests for all orgaisms
davidsmejia Sep 27, 2023
f3d53e8
delimiter not delimeter
davidsmejia Sep 27, 2023
a743e17
delete and recreate survey transcriptome cassettes
davidsmejia Sep 27, 2023
7e98e1f
Update docker files: use heredocs syntax
arkid15r Sep 28, 2023
82324dc
Move smasher test jobs to larger runners
arkid15r Sep 28, 2023
c3c4a43
Update GH workflows: add test_base to test_api_base dependencies
arkid15r Sep 28, 2023
da4847c
fix typos
davidsmejia Sep 28, 2023
51fd6d1
only check if the last check failed
davidsmejia Sep 28, 2023
2febe1a
update return type
davidsmejia Sep 28, 2023
09337d7
remove variable and switch comparison order
davidsmejia Sep 28, 2023
5c0622a
remove variable and switch comparison order
davidsmejia Sep 28, 2023
1e61329
update util and add test
davidsmejia Sep 28, 2023
78f1410
Merge pull request #3399 from AlexsLemonade/ark/move-smasher-jobs-to-…
arkid15r Sep 28, 2023
a18fd44
Merge branch 'dev' into ark/make-test-api-depend-on-base
arkid15r Sep 28, 2023
b3e5f66
Merge pull request #3400 from AlexsLemonade/ark/make-test-api-depend-…
arkid15r Sep 28, 2023
bb96ad9
adds filter_dicts_on_key
davidsmejia Sep 29, 2023
b54ed9e
use filter_dicts_on_key
davidsmejia Sep 29, 2023
ccb5535
add test for filter_dicts_on_key
davidsmejia Sep 29, 2023
f105c2f
format comments
davidsmejia Sep 29, 2023
baa1b99
use list comprehension
davidsmejia Sep 29, 2023
4869107
update docstring
davidsmejia Sep 29, 2023
77741b2
assertIsNone
davidsmejia Sep 29, 2023
7ca18f6
Merge branch 'dev' of https://github.com/AlexsLemonade/refinebio into…
davidsmejia Sep 29, 2023
5139f7e
Merge pull request #3397 from AlexsLemonade/davidsmejia/3381-strain-m…
davidsmejia Oct 2, 2023
6304213
Merge pull request #3398 from AlexsLemonade/ark/docker-heredocs-refac…
arkid15r Oct 2, 2023
e7acc18
Update scripts: set platform and system version
arkid15r Oct 2, 2023
a1f59e6
Run agilent tests after affymetrix image is ready
arkid15r Oct 3, 2023
7ab86ce
Merge pull request #3402 from AlexsLemonade/ark/optimize-gh-tests
arkid15r Oct 6, 2023
c2a9cee
Untrack dev.tfvars
arkid15r Oct 6, 2023
1369e34
Merge pull request #3401 from AlexsLemonade/ark/update-scripts
arkid15r Oct 6, 2023
dac43c9
Update SENTRY_DSN configuration
arkid15r Oct 6, 2023
ee0de0a
Merge pull request #3407 from AlexsLemonade/ark/update-sentry-dsn-config
davidsmejia Oct 9, 2023
9c6afc8
Fix SENTRY_DSN env name mismatch
arkid15r Oct 9, 2023
d937e57
Merge pull request #3409 from AlexsLemonade/ark/update-sentry-dsn
davidsmejia Oct 10, 2023
06f1610
Merge pull request #3403 from AlexsLemonade/ark/untrack-infrastructur…
arkid15r Oct 10, 2023
15d7142
Use STAGE variable for Sentry environment tag
arkid15r Oct 11, 2023
760fdc3
Merge pull request #3412 from AlexsLemonade/ark/3410-fix-stage-enviro…
davidsmejia Oct 12, 2023
2b5a81d
Update downloaders image: fix rvm.io keys issue
arkid15r Oct 12, 2023
548dc82
Rename Sample::genotype to Sample::genetic_information
arkid15r Oct 12, 2023
2da4ec9
Merge pull request #3414 from AlexsLemonade/ark/update-downloader-image
arkid15r Oct 13, 2023
c0e7ff1
Update deprecated AWS elastic IP address configuration
arkid15r Oct 16, 2023
b3adda4
Explicitly remove existing API containers
arkid15r Oct 16, 2023
3007d7d
Merge pull request #3418 from AlexsLemonade/ark/update-terraform-config
arkid15r Oct 16, 2023
06511fa
Merge pull request #3419 from AlexsLemonade/ark/update-api-instance-u…
arkid15r Oct 16, 2023
f5d4b39
Merge pull request #3415 from AlexsLemonade/ark/3408-change-sample-ge…
arkid15r Oct 17, 2023
1b22efb
Fix deploy issues
arkid15r Oct 17, 2023
517b276
Merge pull request #3422 from AlexsLemonade/ark/fix-deploy-issues
arkid15r Oct 17, 2023
3d79f3c
Add STAGE to surveyor batch environment
arkid15r Oct 17, 2023
40e34d6
Add Sample::developmental_stage field support
arkid15r Oct 18, 2023
56333e6
Merge pull request #3423 from AlexsLemonade/ark/add-stage-to-surveyor…
arkid15r Oct 19, 2023
81b6e60
Fix downloaders/no-op tests
arkid15r Oct 20, 2023
0f114da
Update pip requirements
arkid15r Oct 20, 2023
a5555cb
Merge pull request #3429 from AlexsLemonade/ark/fix-downloaders-no-op…
arkid15r Oct 20, 2023
c926647
Merge pull request #3430 from AlexsLemonade/ark/update-python-depende…
arkid15r Oct 20, 2023
4692b48
Split downloaders/no_op test jobs
arkid15r Oct 20, 2023
087618e
Merge pull request #3431 from AlexsLemonade/ark/split-downloaders-no-…
arkid15r Oct 23, 2023
e34095d
Merge pull request #3425 from AlexsLemonade/ark/3306-sample-developme…
arkid15r Oct 23, 2023
adf0b40
Add API server SSH connection check
arkid15r Oct 25, 2023
d0b17b9
Merge pull request #3441 from AlexsLemonade/ark/3437-api-server-wont-…
arkid15r Oct 25, 2023
c95a4eb
Add deploy box cleanup instructions
arkid15r Oct 26, 2023
504b31f
Merge pull request #3442 from AlexsLemonade/ark/post-deploy-cleanup
arkid15r Oct 26, 2023
273afb3
Fix no-op tests
arkid15r Oct 27, 2023
14e1835
Merge pull request #3446 from AlexsLemonade/ark/3436-fix-no-op-tests
arkid15r Oct 28, 2023
e51c389
Add restart flag to API docker container
arkid15r Oct 30, 2023
8fbf245
Merge pull request #3447 from AlexsLemonade/ark/add-api-container-res…
arkid15r Oct 30, 2023
8d67102
bump up staging instance
davidsmejia Oct 31, 2023
d95a8e0
Merge pull request #3448 from AlexsLemonade/davidsmejia/scale-up-staging
davidsmejia Oct 31, 2023
6f928b6
update nginx log location
davidsmejia Nov 3, 2023
488d32b
serve static files from var/www/static_volumes
davidsmejia Nov 3, 2023
47ce019
Merge pull request #3453 from AlexsLemonade/davidsmejia/3452-tmp-to-var
davidsmejia Nov 3, 2023
b111556
map static to /var/www/volumes_static on deploy
davidsmejia Nov 6, 2023
1a78ca1
Merge pull request #3454 from AlexsLemonade/davidsmejia/3452-var-on-r…
davidsmejia Nov 6, 2023
7a7277c
ensure /var/www/volumes_static exists on container restart
davidsmejia Nov 7, 2023
111a821
combine mkdir + chmod commands
davidsmejia Nov 7, 2023
401c869
Merge pull request #3455 from AlexsLemonade/davidsmejia/3452-ensure-a…
davidsmejia Nov 7, 2023
a9efff4
scale up worker volume
davidsmejia Nov 9, 2023
07d3759
Merge pull request #3456 from AlexsLemonade/davidsmejia/scale-workers…
davidsmejia Nov 9, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/scripts/post_deploy_cleanup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ set -e
ssh -o StrictHostKeyChecking=no \
-o ServerAliveInterval=15 \
-i infrastructure/data-refinery-key.pem \
"ubuntu@${DEPLOY_IP_ADDRESS}" \
"ubuntu@${DEPLOY_BOX_IP}" \
"cd refinebio && git clean -f"
26 changes: 12 additions & 14 deletions .github/scripts/remote_deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,41 +9,39 @@
# It has been written with the intention of being run from GitHub Actions as
# part of our CI/CD process. It therefore assumes that the following
# environment variables will be set:
# - DEPLOY_IP_ADDRESS -- The IP address of the instance to run the deploy on.
# - DEPLOY_BOX_IP -- The IP address of the instance to run the deploy on.
# - CI_TAG -- The tag that was pushed to GitHub to trigger the deploy.
# Will be used as the version for the system and the tag for Docker images.
# - DOCKER_ID -- The username that will be used to log into Dockerhub.
# - DOCKER_PASSWD -- The password that will be used to log into Dockerhub.
# - OPENSSL_KEY -- The OpenSSl key which will be used to decrypt the SSH key.
# - DOCKER_USERNAME -- The username that will be used to log into Dockerhub.
# - DOCKER_PASSWORD -- The password that will be used to log into Dockerhub.
# - AWS_ACCESS_KEY_ID -- The AWS key id to use when interacting with AWS.
# - AWS_SECRET_ACCESS_KEY -- The AWS secret key to use when interacting with AWS.

echo "$INSTANCE_SSH_KEY" >infrastructure/data-refinery-key.pem
echo "$DEPLOY_BOX_SSH_PRIVATE_KEY" >infrastructure/data-refinery-key.pem
chmod 600 infrastructure/data-refinery-key.pem

run_on_deploy_box() {
# shellcheck disable=SC2029
ssh -o StrictHostKeyChecking=no \
-o ServerAliveInterval=15 \
-i infrastructure/data-refinery-key.pem \
ubuntu@"${DEPLOY_IP_ADDRESS}" \
ubuntu@"${DEPLOY_BOX_IP}" \
"cd refinebio && $1"
}

# Create file containing local env vars that are needed for deploy.
rm -f env_vars
cat >>env_vars <<EOF
export CI_TAG='$CI_TAG'
export DOCKER_ID='$DOCKER_ID'
export DOCKER_PASSWD='$DOCKER_PASSWD'
export OPENSSL_KEY='$OPENSSL_KEY'
export DOCKER_USERNAME='$DOCKER_USERNAME'
export DOCKER_PASSWORD='$DOCKER_PASSWORD'
export AWS_ACCESS_KEY_ID='$AWS_ACCESS_KEY_ID'
export AWS_SECRET_ACCESS_KEY='$AWS_SECRET_ACCESS_KEY'
export TF_VAR_database_password='$DATABASE_PASSWORD'
export TF_VAR_django_secret_key='$DJANGO_SECRET_KEY'
export TF_VAR_raven_dsn='$RAVEN_DSN'
export TF_VAR_raven_dsn_api='$RAVEN_DSN_API'
export TF_VAR_engagementbot_webhook='$ENGAGEMENTBOT_WEBHOOK'
export TF_VAR_sentry_dsn='$SENTRY_DSN'
export TF_VAR_slack_webhook_url='$SLACK_WEBHOOK_URL'
export TF_VAR_ssh_public_key='$SSH_PUBLIC_KEY'
EOF

# And checkout the correct tag.
Expand All @@ -56,12 +54,12 @@ run_on_deploy_box "bash .github/scripts/verify_tag.sh $CI_TAG"
# Copy the necessary environment variables over.
scp -o StrictHostKeyChecking=no \
-i infrastructure/data-refinery-key.pem \
-r env_vars ubuntu@"$DEPLOY_IP_ADDRESS":refinebio/env_vars
-r env_vars ubuntu@"$DEPLOY_BOX_IP":refinebio/env_vars

# Along with the ssh key iself, which the deploy script will use.
scp -o StrictHostKeyChecking=no \
-i infrastructure/data-refinery-key.pem \
-r infrastructure/data-refinery-key.pem ubuntu@"$DEPLOY_IP_ADDRESS":refinebio/infrastructure/data-refinery-key.pem
-r infrastructure/data-refinery-key.pem ubuntu@"$DEPLOY_BOX_IP":refinebio/infrastructure/data-refinery-key.pem

echo "Building new images"
# Output to the docker update log.
Expand Down
6 changes: 3 additions & 3 deletions .github/scripts/slackpost_deploy.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

if [[ $ENGAGEMENTBOT_WEBHOOK == "" ]]; then
echo "No webhook url. Set ENGAGEMENTBOT_WEBHOOK in the environment variables if you want to be notified of deploys on slack"
if [[ $SLACK_WEBHOOK_URL == "" ]]; then
echo "No webhook url. Set SLACK_WEBHOOK_URL in the environment variables if you want to be notified of deploys on slack"
exit 0
fi

Expand Down Expand Up @@ -36,4 +36,4 @@ escapedText=$(echo "$text" | sed 's/"/\"/g' | sed "s/'/\'/g")

json="{\"channel\": \"$channel\", \"username\":\"$username\", \"icon_emoji\":\":tada:\", \"attachments\":[{\"color\":\"danger\" , \"text\": \"$escapedText\"}]}"

curl -s -d "payload=$json" "$ENGAGEMENTBOT_WEBHOOK"
curl -s -d "payload=$json" "$SLACK_WEBHOOK_URL"
6 changes: 3 additions & 3 deletions .github/scripts/slackpost_end_to_end.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

if [[ $ENGAGEMENTBOT_WEBHOOK == "" ]]; then
echo "No webhook url. Set ENGAGEMENTBOT_WEBHOOK in the environment variables if you want to be notified of deploys on slack"
if [[ $SLACK_WEBHOOK_URL == "" ]]; then
echo "No webhook url. Set SLACK_WEBHOOK_URL in the environment variables if you want to be notified of deploys on slack"
exit 0
fi

Expand All @@ -26,4 +26,4 @@ escapedText=$(echo "$text" | sed 's/"/\"/g' | sed "s/'/\'/g")

json="{\"channel\": \"$channel\", \"username\":\"$username\", \"icon_emoji\":\":tada:\", \"attachments\":[{\"color\":\"danger\" , \"text\": \"$escapedText\"}]}"

curl -s -d "payload=$json" "$ENGAGEMENTBOT_WEBHOOK"
curl -s -d "payload=$json" "$SLACK_WEBHOOK_URL"
Loading