Storage Utilization Optimiaztion

[BlazeIsClone]

The current configuration requires us to have 2x amount of storage capacity as we compress files on the origin server itself before transporting them to the runner.

Solution

We can Rsync transport files to the runner then within the runner we can move the files to the object storage. Once the files are in the bucket we can zip them there.

Limitations

Again the GitHub runner should provide support for handling the required storage capacity. At this point in time, it's approximately 30GiBs

[BlazeIsClone]

Multipart Upload Files to S3 and Zip:

We can use the aws s3 cp command with the --multipart-upload option to multipart upload large files to S3. For example:

Zip the Uploaded Files in S3 Bucket:
aws s3 cp largefile.txt s3://your-bucket/largefile.txt --multipart-upload

To zip the uploaded files in the S3 bucket, you can use the aws s3 cp command with the --exclude and --include options to specify the files you want to include in the zip, and then pipe the output to a zip command.

aws s3 cp s3://your-bucket/ s3://your-bucket/ --exclude "*" --include "*.txt" --recursive | xargs -I {} zip -j {}.zip {}

This command will zip all .txt files in the S3 bucket and store them with a .zip extension.

[BlazeIsClone]

Marking this as release v3.0.0