Extract VCE renewable generation profiles and remove deprecated `gsutil` from workflows #3893

e-belfer · 2024-10-02T17:00:24Z

Overview

Closes #3880.

What problem does this address?

Extracts VCE renewable generation profile data into raw dagster assets. Also fixes a nightly build failure due to the deprecation of gsutil (IAM permissions were updated manually and need to be added in terraform, but this is out of scope for this PR).

What did you change?

Add a module for vcerare extraction and specify valid run settings
Add vcerare to the fast and full ETL runs
Swap gsutil with gcloud storage

To-do list

Give feedback

Swap fake DOI once Archive the raw vceregen data on Zenodo pudl-archiver#434 merged
Extract the tiny lat/lon/fips CSV
Check in column_maps folder or add option to return no error when folder does not exist
Swap out ancient gsutil for gcloud storage in zenodo-cache-sync
Options

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Give feedback

Review the PR yourself and call out any questions or issues you have.
For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
Options

src/pudl/extract/vceregen.py

…ative/pudl into extract-vceregen

For more information, see https://pre-commit.ci

…ative/pudl into extract-vceregen

jdangerx

The gsutil/gcloud storage changes look good to me!

jdangerx · 2024-10-16T19:00:26Z

docker/Dockerfile

@@ -15,7 +15,7 @@ RUN apt-get update && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

-# Configure gsutil authentication
+# Configure gcloud authentication


Sort of surprised this is needed, it's writing to boto.cfg which sounds like it's for AWS. But no need to mess with that now I guess.

jdangerx · 2024-10-16T19:07:49Z

docker/gcp_pudl_etl.sh

@@ -298,13 +298,13 @@ if [[ $ETL_SUCCESS == 0 ]]; then
    # If running a tagged release, ensure that outputs can't be accidentally deleted
    # It's not clear that an object lock can be applied in S3 with the AWS CLI
    if [[ "$GITHUB_ACTION_TRIGGER" == "push" && "$BUILD_REF" == v20* ]]; then
-        gsutil -m -u catalyst-cooperative-pudl retention temp set "gs://pudl.catalyst.coop/$BUILD_REF/*" 2>&1 | tee -a "$LOGFILE"
+        gcloud storage --billing-project="catalyst-cooperative-pudl" objects update "gs://pudl.catalyst.coop/$BUILD_REF/*" --temporary-hold 2>&1 | tee -a "$LOGFILE"


nit: seems like we could use "$GCP_BILLING_PROJECT" instead of "catalyst-cooperative-pudl" here.

aesharpe

This looks good! a few comments about format etc.

I have one overarching question about the dataset name. Technically it is called the "VCE RARE Power Dataset" but we refer to it here as the "VCE RARE generation profiles". I'm not sure how important the last few words after VCE RARE are...I like the way we currently have it because "generation profiles" feels more descriptive than "power dataset" but maybe we should just use the official name. I don't feel strongly about this, just wanted to flag.

aesharpe · 2024-10-16T19:06:54Z

src/pudl/extract/extractor.py

+    def _load_column_maps(self, column_map_pkg: str) -> dict:
+        """Create a dictionary of all column mapping CSVs to use in get_column_map()."""
+        column_dict = {}
+        for res_path in importlib.resources.files(column_map_pkg).iterdir():
+            # res_path is expected to end with ${page}.csv
+            if res_path.suffix == ".csv":
+                column_map = self._load_csv(column_map_pkg, res_path.name)
+                column_dict[res_path.stem] = column_map
+        return column_dict
+


is this just a formality seeing as we don't have any column maps for this dataset?

This is moving the existing load step into a separate method in the base class, so that i'm able to subclass and change it in the VCE extractor.

aesharpe · 2024-10-16T19:08:09Z

src/pudl/extract/vcerare.py

+This dataset has 1,000s of columns, so we don't want to manually specify a rename on
+import because we'll pivot these to a column. We adapt the standard extraction
+infrastructure to simply read in the data.


Maybe add "in the PUDL pipeline" after "pivot these to a column"

aesharpe · 2024-10-16T19:09:40Z

src/pudl/extract/vcerare.py

+The drive also contains one more file: vce_county_lat_long_fips_table.csv. This file is
+not partitioned, so we always read it in regardless of the partitions configured for the
+run.


one more CSV file (because there is also the readme)

aesharpe · 2024-10-16T19:13:42Z

src/pudl/extract/vcerare.py

+
+
+def raw_vcerare_asset_factory(part: str) -> AssetsDefinition:
+    """An asset factory for VCE hourly renewable generation profiles."""


Same name comment

aesharpe · 2024-10-16T19:14:17Z

src/pudl/extract/vcerare.py

+
+    @asset(**asset_kwargs)
+    def _extract(context, raw_vcerare__all_dfs):
+        """Extract raw GridPath RA Toolkit renewable energy generation profiles.


This seems like an accidental copy-over from the gridpath extractor.

Sure is, good catch!

aesharpe · 2024-10-16T19:17:24Z

src/pudl/extract/vcerare.py

+
+
+class VCEMetadata(GenericMetadata):
+    """Special metadata class for VCE RARE renewable generation profiles."""


the RE in rare stands for renewable energy so I think we can take out the word renewable here

aesharpe · 2024-10-16T19:17:47Z

src/pudl/extract/vcerare.py

+
+
+class Extractor(CsvExtractor):
+    """Extractor for VCE renewable generation profiles."""


Should call it by RARE name here

aesharpe · 2024-10-16T19:21:07Z

src/pudl/settings.py

@@ -401,6 +401,19 @@ class EiaAeoSettings(GenericDatasetSettings):
    years: list[int] = data_source.working_partitions["years"]


+class VCERARESettings(GenericDatasetSettings):


I think the way we've named the other sources is with Pascal case (VceRareSettings)

aesharpe · 2024-10-17T02:42:41Z

src/pudl/package_data/vcerare/__init__.py

@@ -0,0 +1 @@
+"""CSV file extraction maps for VCE RARE generation profiles."""


Just a note for consistency if you want to change this to say "Power Dataset" to match the others.

aesharpe · 2024-10-17T02:43:44Z

src/pudl/settings.py

    phmsagas: PhmsaGasSettings | None = None
    nrelatb: NrelAtbSettings | None = None
    gridpathratoolkit: GridPathRAToolkitSettings | None = None
+    vcerare: VCERareSettings | None = None


another nit, but I think even the VCE can be lowercased to VceRare... to match the others.

aesharpe · 2024-10-17T07:10:08Z

src/pudl/extract/vcerare.py

+
+
+@asset(required_resource_keys={"datastore", "dataset_settings"})
+def raw_vcegen__lat_lon_fips(context) -> pd.DataFrame:


I think this should be raw_vcerare__

Sure should be, I'll update now

aesharpe and others added 5 commits September 30, 2024 15:57

Add source metadata for vceregen

9fadfff

Add profiles to vceregen dataset name

4036479

Remove blank line in description

d8b992e

Stash WIP of extraction

82be4d4

Extract VCE tables to raw dask dfs

e044e93

e-belfer self-assigned this Oct 2, 2024

e-belfer added the vcerare Pertaining to Vibrant Clean Energy's Resource Adequacy Renewable Energy Power datasets label Oct 2, 2024

e-belfer linked an issue Oct 2, 2024 that may be closed by this pull request

Write an extractor for the vceregen data #3880

Closed

e-belfer added 3 commits October 2, 2024 13:28

Clean up warnings and restore EIA 176

57ad9a7

Revert to pandas concatenation

b922328

Add latlonfips

53934f2

zaneselvans reviewed Oct 2, 2024

View reviewed changes

src/pudl/extract/vceregen.py Outdated Show resolved Hide resolved

aesharpe added the new-data Requests for integration of new data. label Oct 3, 2024

Base automatically changed from vceregen-metadata to main October 3, 2024 19:32

e-belfer and others added 9 commits October 8, 2024 15:36

Remove dask, coerce dtypes on read-in

3677f78

override load_column_maps behavior

c870b63

Merge branch 'main' into extract-vceregen

4a4511d

Merge branch 'main' into extract-vceregen

b6b5e6c

Update DOI to sandbox and temporarily xfail DOI test

291ba7d

Update regen to rare

3eaebe6

Merge branch 'main' into extract-vceregen

b324123

Merge branch 'extract-vceregen' of https://github.com/catalyst-cooper…

120451d

…ative/pudl into extract-vceregen

[pre-commit.ci] auto fixes from pre-commit.com hooks

5b98e60

For more information, see https://pre-commit.ci

e-belfer marked this pull request as ready for review October 16, 2024 13:46

Merge branch 'main' into extract-vceregen

9f6204e

e-belfer requested a review from aesharpe October 16, 2024 13:46

e-belfer added 4 commits October 16, 2024 10:02

Update gsutil in zenodo-cache-sync

77d47a4

Merge branch 'extract-vceregen' of https://github.com/catalyst-cooper…

adfff81

…ative/pudl into extract-vceregen

Add back user project

6dea332

Update project path

7554a36

e-belfer added 2 commits October 16, 2024 14:24

Update project to billing project

9ada9f5

Update dockerfile to replace gsutil with gcloud storage

4158afd

e-belfer requested a review from jdangerx October 16, 2024 18:50

e-belfer added bug Things that are just plain broken. nightly-builds Anything having to do with nightly builds or continuous deployment. labels Oct 16, 2024

e-belfer changed the title ~~Extract VCE renewable generation profiles~~ Extract VCE renewable generation profiles and remove deprecated gsutil from workflows Oct 16, 2024

jdangerx approved these changes Oct 16, 2024

View reviewed changes

aesharpe reviewed Oct 16, 2024

View reviewed changes

Update dataset name, use GCP_BILLING_PROJECT var

7f45bff

aesharpe reviewed Oct 17, 2024

View reviewed changes

Rename fips csv and update init

e16a4df

e-belfer enabled auto-merge October 17, 2024 12:25

e-belfer added this pull request to the merge queue Oct 17, 2024

Merged via the queue into main with commit 40dd5cc Oct 17, 2024
19 checks passed

e-belfer deleted the extract-vceregen branch October 17, 2024 13:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extract VCE renewable generation profiles and remove deprecated `gsutil` from workflows #3893

Extract VCE renewable generation profiles and remove deprecated `gsutil` from workflows #3893

e-belfer commented Oct 2, 2024 •

edited

Loading

To-do list

To-do list

jdangerx left a comment

jdangerx Oct 16, 2024

jdangerx Oct 16, 2024

e-belfer Oct 16, 2024

aesharpe left a comment

aesharpe Oct 16, 2024

e-belfer Oct 16, 2024

aesharpe Oct 16, 2024

aesharpe Oct 16, 2024

aesharpe Oct 16, 2024

aesharpe Oct 16, 2024

e-belfer Oct 16, 2024

aesharpe Oct 16, 2024

aesharpe Oct 16, 2024

aesharpe Oct 16, 2024

aesharpe Oct 17, 2024

aesharpe Oct 17, 2024

aesharpe Oct 17, 2024

e-belfer Oct 17, 2024



		def raw_vcerare_asset_factory(part: str) -> AssetsDefinition:
		"""An asset factory for VCE hourly renewable generation profiles."""



		class VCEMetadata(GenericMetadata):
		"""Special metadata class for VCE RARE renewable generation profiles."""



		class Extractor(CsvExtractor):
		"""Extractor for VCE renewable generation profiles."""

		@@ -401,6 +401,19 @@ class EiaAeoSettings(GenericDatasetSettings):
		years: list[int] = data_source.working_partitions["years"]


		class VCERARESettings(GenericDatasetSettings):

		@@ -0,0 +1 @@
		"""CSV file extraction maps for VCE RARE generation profiles."""



		@asset(required_resource_keys={"datastore", "dataset_settings"})
		def raw_vcegen__lat_lon_fips(context) -> pd.DataFrame:

Extract VCE renewable generation profiles and remove deprecated gsutil from workflows #3893

Extract VCE renewable generation profiles and remove deprecated gsutil from workflows #3893

Conversation

e-belfer commented Oct 2, 2024 • edited Loading

Overview

What problem does this address?

What did you change?

To-do list

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

jdangerx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aesharpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Extract VCE renewable generation profiles and remove deprecated `gsutil` from workflows #3893

Extract VCE renewable generation profiles and remove deprecated `gsutil` from workflows #3893

e-belfer commented Oct 2, 2024 •

edited

Loading