Skip to content

Commit

Permalink
Tweaks and edits
Browse files Browse the repository at this point in the history
  • Loading branch information
zaneselvans committed Nov 4, 2024
1 parent e4b5310 commit 005247b
Show file tree
Hide file tree
Showing 3 changed files with 71 additions and 33 deletions.
93 changes: 64 additions & 29 deletions docs/data-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,41 +7,72 @@
- Inevitably choices have to be made in the process, and some of the original detail gets lost.
- The benefit is that you can start with data that’s much closer to analysis ready and build upon it rather than starting from scratch.

## International / Global data (OECD, IEA, IRENA, World Bank)
## Worldwide Energy Data

- Most expansive datasets available.
- Necessarily relatively shallow.
- Recent moves toward free and open access.
- Typically sourced from national agency reporting
- Good context, but not our focus.
- Good for context, but not Catalyst's focus.

### OECD

### IEA

### IRENA

### World Bank

### [Global Energy Monitor](https://globalenergymonitor.org/)

- Worldwide coverage, industry as well as electricity generation
- Also doing primary data collection in the Global South

### [Climate Trace Data Downloads](https://climatetrace.org/data)

### [Our World In Data](https://ourworldindata.org/)

- Experimental [OWID Python API](https://docs.owid.io/projects/etl/api/)

## Open Source / Data Projects

- [Public Utility Data Liberation Project](https://catalystcoop-pudl.readthedocs.io/) (PUDL) (Catalyst Cooperative)
- Primarily focused on US federal electricity system data
- Mix of operational and financial/economic data
- [Open Grid Emissions Initiative](https://docs.singularity.energy/docs/open-grid-emissions/about_ogei-about-the-open-grid-emissions-initiative) (Singularity Energy)
- Uses PUDL, EPA and other data to estimate hourly emissions intensity by location
- [GridStatus](https://www.gridstatus.io/)
- Software behind the API is open source, but setting up the whole tech stack isn’t trivial
- Consolidates data from organized markets into a common API
- Can’t provide full granularity available in all data sources, because they vary considerably.
- Has to provide least-common-denominator data in the name of uniformity
- Large benefit in ease of use.
- Not actually free and open data – for higher volume usage a $500/mo subscription is required.
- [PyPSA Meets Earth](https://github.com/pypsa-meets-earth/earth-osm)
- Extracts electricity system data from the Open Street Map and converts the format / cleans it up for use in modeling.
### [Public Utility Data Liberation Project](https://catalystcoop-pudl.readthedocs.io/) (PUDL) (Catalyst Cooperative)

- Primarily focused on US federal electricity system data
- Mix of operational and financial/economic data

### [Open Grid Emissions Initiative](https://docs.singularity.energy/docs/open-grid-emissions/about_ogei-about-the-open-grid-emissions-initiative) (Singularity Energy)

- Uses PUDL, EPA and other data to estimate hourly emissions intensity by location

### [GridStatus](https://www.gridstatus.io/)

- Software behind the API is open source, but setting up the whole tech stack isn’t trivial
- Consolidates data from organized markets into a common API
- Can’t provide full granularity available in all data sources, because they vary considerably.
- Has to provide least-common-denominator data in the name of uniformity
- Large benefit in ease of use.
- Not actually free and open data – for higher volume usage a $500/mo subscription is required.

### [PyPSA Meets Earth](https://github.com/pypsa-meets-earth/earth-osm)

- Extracts electricity system data from the Open Street Map and converts the format / cleans it up for use in modeling.

## NGOs

- [Global Energy Monitor](https://globalenergymonitor.org/)
- Worldwide coverage, industry as well as power
- Also doing primary data collection in the Global South
- [Climate Trace Data Downloads](https://climatetrace.org/data)
- [RMI Utility Transition Hub](https://utilitytransitionhub.rmi.org/)
- [CarbonPlan Datasets](https://carbonplan.org/data)
- [Our World In Data](https://ourworldindata.org/) (new [experimental Python API](https://docs.owid.io/projects/etl/api/))
- [WattTime](https://watttime.org/data-science/data-signals/) marginal emissions & human health impacts by time and location (NGO, but not open data)
### [RMI Utility Transition Hub](https://utilitytransitionhub.rmi.org/)

### [CarbonPlan Datasets](https://carbonplan.org/data)

- Carbon capture and sequestration project evaluations
- Carbon offset project evaluations
- Wildfire emissions estimates
- Downscaled climate model outputs

### [WattTime](https://watttime.org/data-science/data-signals/)

- Marginal emissions & human health impacts by time and location
- Non-profit NGO, but the data is no longer open.

## National Labs

Expand All @@ -54,14 +85,18 @@

## Academic

- Static, self-archived datasets associated with peer reviewed publications, e.g.
### Static archives

Datasets associated with a publication that have been archived for future reference, but are not actively maintained.

- [10+ years of locational marginal pricing data scraped from ISO/RTO websites](https://zenodo.org/records/3368397)

- [Shaping photovoltaic array output to align with changing wholesale electricity price profiles](https://zenodo.org/records/3368397) (on Zenodo)
### Curated resources

- Semi-automated dataset updates, e.g.
Research datasets with at least an attempt at ongoing maintenance.

- [Power Sector Carbon Index](https://emissionsindex.org/) (Carnegie Mellon University)
- [GridEmissions](https://gridemissions.jdechalendar.su.domains/#/) (Stanford University)
- [Power Sector Carbon Index](https://emissionsindex.org/) (Carnegie Mellon University)
- [GridEmissions](https://gridemissions.jdechalendar.su.domains/#/) (Stanford University)

```python
import pandas as pd
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Open Energy Data For All (OED4A)
# Data Liberation Engineering

This site contains a curated list of publicly available US energy system data, and
provides some guidelines on what to look for when evaluating a data source for potential
Expand Down
9 changes: 6 additions & 3 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
site_name: Catalyst MkDocs Material Test
site_name: Data Liberation Engineering
site_url: https://docs.catalyst.coop
site_description: A top level documentation index for Catalyst Cooperative (TEST).
repo_name: catalyst-cooperative/catalyst-cooperative.github.io
repo_url: https://github.com/catalyst-cooperative/catalyst-cooperative.github.io
edit_uri: edit/main/docs/
copyright: "Copyright © 2024 Catalyst Cooperative"
copyright: '<a href="https://creativecommons.org/licenses/by/4.0/">CC-BY-4.0</a> &copy; 2024 Catalyst Cooperative'

theme:
name: material
Expand All @@ -15,7 +15,7 @@ theme:
- content.action.view
- search.suggest
icon:
repo: fontawesome/brands/github
repo: fontawesome/brands/git-alt
palette:
- media: "(prefers-color-scheme: light)"
scheme: default
Expand Down Expand Up @@ -70,6 +70,9 @@ extra:
- icon: fontawesome/brands/kaggle
link: https://www.kaggle.com/catalystcooperative
name: Catalyst Cooperative on Kaggle
# - icon: fontawesome/solid/face-smile-beam
# link: https://www.huggingface.co/catalyst-cooperative
# name: Catalyst Cooperative on Hugging Face
- icon: fontawesome/brands/x-twitter
link: https://twitter.com/CatalystCoop
name: Catalyst Cooperative on Twitter
Expand Down

0 comments on commit 005247b

Please sign in to comment.