From 005247b6ad72c3e2efb290cdebf3c3fe7a5357ce Mon Sep 17 00:00:00 2001 From: Zane Selvans Date: Mon, 4 Nov 2024 11:31:43 -0600 Subject: [PATCH] Tweaks and edits --- docs/data-sources.md | 93 ++++++++++++++++++++++++++++++-------------- docs/index.md | 2 +- mkdocs.yml | 9 +++-- 3 files changed, 71 insertions(+), 33 deletions(-) diff --git a/docs/data-sources.md b/docs/data-sources.md index ed8828d..ed81417 100644 --- a/docs/data-sources.md +++ b/docs/data-sources.md @@ -7,41 +7,72 @@ - Inevitably choices have to be made in the process, and some of the original detail gets lost. - The benefit is that you can start with data that’s much closer to analysis ready and build upon it rather than starting from scratch. -## International / Global data (OECD, IEA, IRENA, World Bank) +## Worldwide Energy Data - Most expansive datasets available. - Necessarily relatively shallow. - Recent moves toward free and open access. - Typically sourced from national agency reporting -- Good context, but not our focus. +- Good for context, but not Catalyst's focus. + +### OECD + +### IEA + +### IRENA + +### World Bank + +### [Global Energy Monitor](https://globalenergymonitor.org/) + +- Worldwide coverage, industry as well as electricity generation +- Also doing primary data collection in the Global South + +### [Climate Trace Data Downloads](https://climatetrace.org/data) + +### [Our World In Data](https://ourworldindata.org/) + +- Experimental [OWID Python API](https://docs.owid.io/projects/etl/api/) ## Open Source / Data Projects -- [Public Utility Data Liberation Project](https://catalystcoop-pudl.readthedocs.io/) (PUDL) (Catalyst Cooperative) - - Primarily focused on US federal electricity system data - - Mix of operational and financial/economic data -- [Open Grid Emissions Initiative](https://docs.singularity.energy/docs/open-grid-emissions/about_ogei-about-the-open-grid-emissions-initiative) (Singularity Energy) - - Uses PUDL, EPA and other data to estimate hourly emissions intensity by location -- [GridStatus](https://www.gridstatus.io/) - - Software behind the API is open source, but setting up the whole tech stack isn’t trivial - - Consolidates data from organized markets into a common API - - Can’t provide full granularity available in all data sources, because they vary considerably. - - Has to provide least-common-denominator data in the name of uniformity - - Large benefit in ease of use. - - Not actually free and open data – for higher volume usage a $500/mo subscription is required. -- [PyPSA Meets Earth](https://github.com/pypsa-meets-earth/earth-osm) - - Extracts electricity system data from the Open Street Map and converts the format / cleans it up for use in modeling. +### [Public Utility Data Liberation Project](https://catalystcoop-pudl.readthedocs.io/) (PUDL) (Catalyst Cooperative) + +- Primarily focused on US federal electricity system data +- Mix of operational and financial/economic data + +### [Open Grid Emissions Initiative](https://docs.singularity.energy/docs/open-grid-emissions/about_ogei-about-the-open-grid-emissions-initiative) (Singularity Energy) + +- Uses PUDL, EPA and other data to estimate hourly emissions intensity by location + +### [GridStatus](https://www.gridstatus.io/) + +- Software behind the API is open source, but setting up the whole tech stack isn’t trivial +- Consolidates data from organized markets into a common API +- Can’t provide full granularity available in all data sources, because they vary considerably. +- Has to provide least-common-denominator data in the name of uniformity +- Large benefit in ease of use. +- Not actually free and open data – for higher volume usage a $500/mo subscription is required. + +### [PyPSA Meets Earth](https://github.com/pypsa-meets-earth/earth-osm) + +- Extracts electricity system data from the Open Street Map and converts the format / cleans it up for use in modeling. ## NGOs -- [Global Energy Monitor](https://globalenergymonitor.org/) - - Worldwide coverage, industry as well as power - - Also doing primary data collection in the Global South -- [Climate Trace Data Downloads](https://climatetrace.org/data) -- [RMI Utility Transition Hub](https://utilitytransitionhub.rmi.org/) -- [CarbonPlan Datasets](https://carbonplan.org/data) -- [Our World In Data](https://ourworldindata.org/) (new [experimental Python API](https://docs.owid.io/projects/etl/api/)) -- [WattTime](https://watttime.org/data-science/data-signals/) marginal emissions & human health impacts by time and location (NGO, but not open data) +### [RMI Utility Transition Hub](https://utilitytransitionhub.rmi.org/) + +### [CarbonPlan Datasets](https://carbonplan.org/data) + +- Carbon capture and sequestration project evaluations +- Carbon offset project evaluations +- Wildfire emissions estimates +- Downscaled climate model outputs + +### [WattTime](https://watttime.org/data-science/data-signals/) + +- Marginal emissions & human health impacts by time and location +- Non-profit NGO, but the data is no longer open. ## National Labs @@ -54,14 +85,18 @@ ## Academic -- Static, self-archived datasets associated with peer reviewed publications, e.g. +### Static archives + +Datasets associated with a publication that have been archived for future reference, but are not actively maintained. + +- [10+ years of locational marginal pricing data scraped from ISO/RTO websites](https://zenodo.org/records/3368397) - - [Shaping photovoltaic array output to align with changing wholesale electricity price profiles](https://zenodo.org/records/3368397) (on Zenodo) +### Curated resources -- Semi-automated dataset updates, e.g. +Research datasets with at least an attempt at ongoing maintenance. - - [Power Sector Carbon Index](https://emissionsindex.org/) (Carnegie Mellon University) - - [GridEmissions](https://gridemissions.jdechalendar.su.domains/#/) (Stanford University) +- [Power Sector Carbon Index](https://emissionsindex.org/) (Carnegie Mellon University) +- [GridEmissions](https://gridemissions.jdechalendar.su.domains/#/) (Stanford University) ```python import pandas as pd diff --git a/docs/index.md b/docs/index.md index efb0331..79a3950 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,4 +1,4 @@ -# Open Energy Data For All (OED4A) +# Data Liberation Engineering This site contains a curated list of publicly available US energy system data, and provides some guidelines on what to look for when evaluating a data source for potential diff --git a/mkdocs.yml b/mkdocs.yml index c77f3a9..f229488 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,11 +1,11 @@ --- -site_name: Catalyst MkDocs Material Test +site_name: Data Liberation Engineering site_url: https://docs.catalyst.coop site_description: A top level documentation index for Catalyst Cooperative (TEST). repo_name: catalyst-cooperative/catalyst-cooperative.github.io repo_url: https://github.com/catalyst-cooperative/catalyst-cooperative.github.io edit_uri: edit/main/docs/ -copyright: "Copyright © 2024 Catalyst Cooperative" +copyright: 'CC-BY-4.0 © 2024 Catalyst Cooperative' theme: name: material @@ -15,7 +15,7 @@ theme: - content.action.view - search.suggest icon: - repo: fontawesome/brands/github + repo: fontawesome/brands/git-alt palette: - media: "(prefers-color-scheme: light)" scheme: default @@ -70,6 +70,9 @@ extra: - icon: fontawesome/brands/kaggle link: https://www.kaggle.com/catalystcooperative name: Catalyst Cooperative on Kaggle + # - icon: fontawesome/solid/face-smile-beam + # link: https://www.huggingface.co/catalyst-cooperative + # name: Catalyst Cooperative on Hugging Face - icon: fontawesome/brands/x-twitter link: https://twitter.com/CatalystCoop name: Catalyst Cooperative on Twitter