feat(ci): Mirror NCBI datasets results to Hetzner object storage to reduce NIH load and get independence from external service

[corneliusroemer]

Mirror NCBI datasets zip folders to Hetzner object storage to reduce load on NIH servers and uncouple our CI from external services.

Right now all Loculus/PPX organisms are covered, we can easily add Taxon IDs for Genspectrum ones @anna-parker

It scrapes once a day. The zip folder can be downloaded at:

https://hel1.your-objectstorage.com/loculus-public/mirror/{TAXONID}.zip

Where TAXONID is the taxonid we scrape, e.g. 10244 for mpox.

As a next step, we can optionally use these paths to download the zip folders in ingest.

What happens if I overwrite an object and the upload fails midway?

• Single PUT: The old object remains unchanged; no partial overwrite.
• Multipart Upload: Incomplete parts stay temporarily; must complete or abort the upload.

What happens if someone downloads while I overwrite an object?

• They get the old version until the new upload is fully completed.
• No partial or mixed content; updates are atomic.

Does S3 allow partial writes when overwriting?

• No, an object is only replaced when the upload fully succeeds.

[theosanderson]

Nice!