Finalize list of datasets for pilot lessons

[e-belfer]

APIs:

• CEMS
• EIA's API

Webscraping:

• what do we want to do here? Teach how to use inspect? Show a demo with a headless browser? Maybe looking at state data could be helpful here, or scraping an ISO site. Data should be complementary to whatever we grab from EIA.

XML:

JSON:

Parquet:

Cloud buckets:

Option 1:

• query EIA API in XML and JSON - hm, looks like some of the data is missing
• click over to webpage and find some download links -> download them
• grab Parquet from cloud bucket using Catalyst data

Questions to answer:

• how long does it take to provide an EIA API key? If not instantaneous, will be a problem.

Notes on EIA API registration process:
Form is here. Verification email instant. Requires email login and verification. May as well get people to do this. Can't share a key because "forgot my key" just sends the same key.

• Add API registration to the setup docs for the lesson
• But also link them in the part of the episode where we do this cause people will inevitably forget

For the lesson before the API:

• two different Json and XML files left locally (get from API) - are they the same, or do they span different time periods?
• get Parquet from S3 - oh, different variables!
• go back to the API - were variables missed? oh no, this data isn't complete / I'm missing some vars!