Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 upload #150

Merged
merged 11 commits into from
Oct 1, 2024
Merged

S3 upload #150

merged 11 commits into from
Oct 1, 2024

Conversation

timetinytim
Copy link
Contributor

Added a job to upload assets to an S3 bucket on helm chart creation. Tested this in an isolated situation outside of a chart, and the job successfully uploads to exoscale, so the actual job definition works. Haven't tested it in the actual chart itself yet.

One note: this uploads the entire /public folder. Not sure if this is the "correct" thing to do or not, or if just /public/assets should be uploaded. Let me know if this should be changed.

@timetinytim timetinytim requested a review from renchap September 23, 2024 08:58
@timetinytim timetinytim self-assigned this Sep 23, 2024
@timetinytim
Copy link
Contributor Author

After some tweaks and testing, I updated how the transfer is done. Setting --transfers to something higher than the default of 4 does significantly increases speed, and --fast-list helps speed things up a little bit, though only marginally in this case. With the current configuration, and the moment:

Transferring the whole /public folder (just for comparison's sake)
Initial copy: ~30s
Re-run (i.e. checking existing files): ~60s

Transferring just /public/assets and /public/packs
Initial copy: ~17s
Re-run (i.e. checking existing files): ~22s

@renchap Let me know if this works. I think this is much more reasonable than it was before, and good enough to merge.

values.yaml Outdated
# that are renamed. As the pods are getting redeployed, and old/new pods are
# present simultaneously, there is a change that old asset files are
# requested from pods that don't have them anymore. Uploading asset files to
# S3 in this manner solves this potential conflict.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • add that you would need to update your CDN / proxy to send request to /assets and /packs to this bucket?

@timetinytim timetinytim merged commit 9f5d8ee into main Oct 1, 2024
8 checks passed
@timetinytim timetinytim deleted the s3-upload branch October 1, 2024 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants