Write an archiver for NREL Cambium expansions of Standard Scenarios

[krivard]

Motivation and context:

NREL selects a subset of their Standard Scenarios (see #561) for expansion and deeper analysis using Cambium, a specialized modeling and analysis tool. The primary contribution of this dataset is hourly long-run marginal emissions rates.

The results are available via the Scenario Viewer and could be downloaded using the same/similar code we use for Standard Scenarios, but the datasets are much larger -- 1s-10s of GB zipped.

• Description: https://www.nrel.gov/analysis/cambium.html
• Viewer: https://scenarioviewer.nrel.gov/?project=0f92fe57-3365-428a-8fe8-0afc326b3b43&mode=browse&layout=Default

Requirements for archiving

To be archived on Zenodo, a dataset must be:

• published under an open license that permits reuse and redistribution
• less than 50Gb in size (when zipped) - this is the sketchy bit; we might have to split the archive into 5-year batches
• relevant to energy modelling and research

Checklist for archive creation

Based on the README documentation on creating a new archive:

• Define the dataset's metadata
• Implement archiver interface
• Test archiver locally
• Test uploading to Zenodo
• Manually review archive before publication
• Finalize archive (only core Catalyst developers can complete this step)
• Automate archiving

Links to published archives:

Include a link to the published sandbox archive for review.

[krivard]

Challenges so far:

• The server keeps timing out. Not sure if this is a soft IP-based rate-limit or actual server load/unreliability. Boosting to start backoff at 60 seconds helps a little but still fails some times.
• The data is too big: just two years would put us over the Zenodo limit, so we'd have to separate them annually, which seems annoying:

Report	Size in GB (zipped)
Cambium 2020 Total	17.61
Cambium 2021 Total	28.36
Cambium 2022 Total	36.41
Cambium 2023 Total	6.58
Grand Total	88.96

• An analysis of file sizes suggests the ALL - ALL - ALL files are not easier-to-handle packages of everything, but are instead ...whatever the opposite of microdata is. They're aggregates that can't be disaggregated.
• An analysis of .zip contents of the 2023 files suggests there is no overlap in files between zip files; every file name only appears once. 😞