Start Airtable automation #366
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… need to adjust the offshore wind extraction code to work with json
- Still need to create logic for updating the settings file - Need to figure out how to better handle airtable dtype and missing column issue Airtable API does not return column in records if there aren't any values. This is problematic if an entire column is missing values.:
…mate-partners/deployment-gap-model into init-airtable-automation
15 tasks
bendnorman
added
github_actions
…mate-partners/deployment-gap-model into init-airtable-automation
bendnorman
commented
Sep 27, 2024
bendnorman
commented
Sep 27, 2024
bendnorman
commented
Sep 27, 2024
jdangerx
approved these changes
Nov 14, 2024
|
|
||
| env: | ||
| API_KEY_GOOGLE_MAPS: ${{ secrets.API_KEY_GOOGLE_MAPS }} | ||
| GITHUB_REF: ${{ github.ref_name }} # This is changed to dev if running on a schedule |
There was a problem hiding this comment.
We'll get to remove this logic if we end up removing dev - but for now this seems OK.
| - name: Display env variables | ||
| run: | | ||
| echo "Workspace directory: $GITHUB_WORKSPACE" \ | ||
| echo "Google credentials path: $GOOGLE_GHA_CREDS_PATH" \ |
There was a problem hiding this comment.
non-blocking: Haven't set up WIF yet, yeah? We're just using an access key?
There was a problem hiding this comment.
Not yet :/ I'll add it to the list of infra improvements.
| "gs://dgm-archive/synapse/offshore_wind/offshore_wind_locations_2024-09-10.csv" | ||
| ) | ||
| # get the latest version of the offshore wind data from the candidate yaml file | ||
| projects_uri = "airtable/Offshore Wind Locations Synapse Version/Projects.json" |
There was a problem hiding this comment.
nit: if these get immediately fed into es and overwritten, should we just inline these values on lines 127 and 128?
| dataset_ref = client.dataset(dataset_id) | ||
|
|
||
| # get all parquet files in the bucket/{version} directory | ||
| blobs = output_bucket.list_blobs(prefix=f"{version}/{destination_blob_prefix}") |
There was a problem hiding this comment.
non-blocking: You could avoid the if statement on line 87 if you add a match_glob="*.parquet" here, I think.
| for blob in blobs: | ||
| if blob.name.endswith(".parquet"): | ||
| # get the blob filename without the extension | ||
| table_name = blob.name.split("/")[-1].split(".")[0] |
There was a problem hiding this comment.
non-blocking: could make it a Path then use Path.stem here.
…mate-partners/deployment-gap-model into init-airtable-automation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements Phase I of the Airtable automation project
Specifically this PR:
ExtractionSettingswhich acts as the interface between the archives and the ETL (kind of like theDatastorein PUDL). It loads archive version numbers from a settings file and can grab the latest version numbers from GCS.run-full-build.ymlaction ran the ETL when new commits were pushed to a branch and would update GCS and BQ if was started onmainordev. I decided to make this action just be CI and create a new action responsible for CD.run-full-build.ymlaction ran ondevor a tag was pushed. Each BQ table has aversionlabel which points to the name of the directory in GCS (dev,v2024.03.05...). Now that we'll have new outputs every day, I decided to name each new output with a UUID and use that for the BQ table version number. TheOutputMetadataclass collects metadata about the outputs like git sha of the code, date created, etc. and saves it as a yaml file in the output directory on GCS.dbcp.cli.pyscript