Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support BigQuery Dataframes on dbt-BigQuery #753

Open
3 tasks done
amychen1776 opened this issue Jan 28, 2025 · 0 comments
Open
3 tasks done

[Feature] Support BigQuery Dataframes on dbt-BigQuery #753

amychen1776 opened this issue Jan 28, 2025 · 0 comments
Labels
feature:python-models Issues related to python models pkg:dbt-bigquery Issue affects dbt-bigquery

Comments

@amychen1776
Copy link
Contributor

amychen1776 commented Jan 28, 2025

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

We want to support BigQuery DataFrames on the dbt-bigquery adapter as an alternative way to execute python models on dbt-bigquery. The benefit of this is users can now choose to either run python models on a service external to BigQuery (Dataproc) or directly on BigQuery using the python API.

  • Both the table and incremental materializations will be supported as part of the implementation. Which incremental strategies will be dependent on what can be supported via DataFrames but the desire is to support all the existing BQ incremental strategies along with on_schema_change inputs if possible.
  • Users will be able to input a new config field submission_method to define where the python model is executed (see dbt-databricks for example).
  • The initial feature ship will be under a behavior flag to ensure a seamless user experience with no regressions. Once users have it set to true, they will be able to pass in the submission method and execute on BigQuery Dataframes.

Describe alternatives you've considered

No response

Who will this benefit?

All BigQuery users who want to keep their data on the BigQuery service still use Python models.

Are you interested in contributing this feature?

This will be a joint effort with the BigQuery and dbt Labs team.

Anything else?

No response

@amychen1776 amychen1776 added feature:python-models Issues related to python models pkg:dbt-bigquery Issue affects dbt-bigquery labels Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature:python-models Issues related to python models pkg:dbt-bigquery Issue affects dbt-bigquery
Projects
None yet
Development

No branches or pull requests

1 participant