Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] update dbt-bigquery retry options #864

Open
3 tasks done
jschintz-nytimes opened this issue Mar 4, 2025 · 0 comments
Open
3 tasks done

[Feature] update dbt-bigquery retry options #864

jschintz-nytimes opened this issue Mar 4, 2025 · 0 comments
Labels
triage:product In Product's queue type:enhancement New feature request

Comments

@jschintz-nytimes
Copy link

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Expose 2 new parameters for the bigquery retry functionality - give google some more time to spin up or find resources:

  1. base_retry_delay
  2. exponential_sleep_multiplier

https://cloud.google.com/storage/docs/retry-strategy
(I did not find a similar page specific to BQ)

Bigquery has issues with jobs and resources not being available.
About 1/1000 bq jobs randomly seems to fail. Retrying the job solves most of those issues.
But , sometimes google just needs some more time.
Configure a delay and stop the random job failures.

Describe alternatives you've considered

Manually restarting the job an hour later normally solves everything.

Who will this benefit?

Any user of dbt-bigquery with a large n of models.

"The Business". Stakeholders don't seem to care that google had an intermittent issue. SLA's are brutal.

Are you interested in contributing this feature?

Sure!

Anything else?

No response

@jschintz-nytimes jschintz-nytimes added triage:product In Product's queue type:enhancement New feature request labels Mar 4, 2025
@jschintz-nytimes jschintz-nytimes changed the title [Feature] update to dbt-bigquery rety options [Feature] update to dbt-bigquery retry options Mar 4, 2025
@jschintz-nytimes jschintz-nytimes changed the title [Feature] update to dbt-bigquery retry options [Feature] update dbt-bigquery retry options Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triage:product In Product's queue type:enhancement New feature request
Projects
None yet
Development

No branches or pull requests

1 participant