Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change full-refresh behaviour to drop target table after successful run #327

Open
1 task done
lutzkuen opened this issue Jul 14, 2023 · 2 comments
Open
1 task done
Labels
enhancement New feature or request

Comments

@lutzkuen
Copy link

lutzkuen commented Jul 14, 2023

Describe the feature

Right now when you run a full-refresh with dbt-trino it will drop the target table and attempt to re-create it. We have adopted a practice of manually taking backups before doing full-refresh of large tables. If users try to access the data while it is refreshing they get an error.
My proposal is to create the new table with the __dbt_tmp suffix and only after successful creation drop the target table and rename the __dbt_tmp table.
This way data stays available to users and no manual rollback is needed if the full refresh does not succeed.

Describe alternatives you've considered

Manual workaround is easy but annoying

Who will benefit?

Users and Developers will benefit as there is less margin for error and better data availability

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@lutzkuen lutzkuen added the enhancement New feature or request label Jul 14, 2023
@hovaesco
Copy link
Contributor

Hi @lutzkuen, what is the target connector you're using? We are working on adding support for CREATE OR REPLACE in the engine which could help with the issue and it could replace creating temp tables.

@lutzkuen
Copy link
Author

We are using dbt-trino with both starburst and Trino on Iceberg. When create or replace is going to be supported by the engine that would make a much easier solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants