Write an archiver for NREL Standard Scenarios

[cmgosnell]

Motivation and context:

"NREL's Standard Scenarios are a suite of forward-looking scenarios of the U.S. power sector that are updated annually to support and inform energy analysis. "

Main page
The data is downloadable on this scenario viewer
These are also archived on this atb archive page but these links go to the scenario viewer (but it could be a good way to access the years.

Requirements for archiving

To be archived on Zenodo, a dataset must be:

• published under an open license that permits reuse and redistribution
• less than 50Gb in size (when zipped)
• relevant to energy modelling and research

Checklist for archive creation

Based on the README documentation on creating a new archive:

• Define the dataset's metadata
• Implement archiver interface
• Test archiver locally
• Test uploading to Zenodo
• Manually review archive before publication
• Finalize archive (only core Catalyst developers can complete this step)
• Automate archiving

Links to published archives:

Include a link to the published sandbox archive for review.

[krivard]

Okay, it looks like this can be most easily done by reverse-engineering their API, since literally everything in the Viewer is loaded by javascript and not by HTML. Thankfully, the API is pretty straightforward, with no credentials or anything -- though that does mean we should try to be polite and not hammer their server with requests.

Pseudocode:

project_records = GET https://scenarioviewer.nrel.gov/api/projects/
for scenario_project in project_records where scenario_project["name"] starts with "Standard Scenarios":
	project_uuid = scenario_project["uuid"]
        project_year = scenario_project["name"].match("Standard Scenarios (\d{4})").group(1)
	file_list_resp = POST project_uuid={project_uuid} https://scenarioviewer.nrel.gov/api/file-list/
	for file_record in file_list_resp["files"] where file_record["file_type"] == "CSV":
		file_data = POST -L file_ids={file_record["id"]} project_uuid={project_uuid} https://scenarioviewer.nrel.gov/api/download/
		# or alternately
		download_url = (POST file_ids={file_record["id"]} project_uuid={project_uuid} https://scenarioviewer.nrel.gov/api/download/).header("Location")

[krivard]

This doc has explanations of all this year's scenarios. Do we care for archiver purposes?

https://www.nrel.gov/docs/fy25osti/92256.pdf