Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow partials during plan / run #3792

Open
mattiasthalen opened this issue Feb 5, 2025 · 3 comments
Open

Allow partials during plan / run #3792

mattiasthalen opened this issue Feb 5, 2025 · 3 comments

Comments

@mattiasthalen
Copy link

Sometimes I need to run partial intervals for all models, e.g., during initial plan.
Instead of setting this for all models, I would like to do it as an option.
sqlmesh plan --allow-partials
sqlmesh run --allow-partials

@tobymao
Copy link
Contributor

tobymao commented Feb 6, 2025

i think we could add this for run, but can you explain more about your use case? would love to learn more about how your using this and why you need allow partials, this will help us build context to how folks are using sqlmesh

@mattiasthalen
Copy link
Author

I face the issue when initialising a project. It does the first load of all data into snapshots (SCD2), then the downstream models read from those with incremental by time range on _sqlmesh__loaded_at, a field I add during snapshot. That opens the scenario of the snapshots being created 10:04, and the downstream models are still in this partial interval.

@tobymao
Copy link
Contributor

tobymao commented Feb 6, 2025

sorry, i'm confused, that's why sqlmesh doesn't allow partials. the incremental by time range shouldn't load any data that isn't complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants