Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iss 228 add std clean dbt stages to census taskflow #231

Merged
merged 2 commits into from
Dec 29, 2024

Conversation

MattTriano
Copy link
Owner

Changes:

  • Implements a standardized and clean stage model generation taskflow for census datasets. Closes Implement standardized and clean stage dbt model generation for Census-taskflows #228
  • Modifies Census taskflow to run the above taskflow even if the freshness check finds the local data is fresh (previously this would end the DAG run).
    • Note: This should be adopted in the other taskflows; it's much more convenient.
  • Modifies the Census taskflow to also check that a local table for the data exists before short-circuiting an update (previously the Census taskflow only checks the census dataset metadata table and concluded the local dataset was fresh if the dataset metadata table indicated we'd already pulled the latest data).
    • This makes it possible to drop a data_raw.{census_dataset_name} (say, if you changed the columns in a new dataset), delete the corresponding dbt models, and rerun the DAG and now it will pull and recreate the table rather than just erroneously short-circuiting.
  • Adds a pipeline for a Census dataset vintage focused on housing units and occupancy.

@MattTriano MattTriano merged commit 351b4e6 into main Dec 29, 2024
1 check passed
@MattTriano MattTriano deleted the iss_228_add_std_clean_dbt_stages_to_census_taskflow branch December 29, 2024 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement standardized and clean stage dbt model generation for Census-taskflows
1 participant