Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Iceberg support for view materialization #341

Closed
3 tasks done
dataders opened this issue Nov 1, 2024 · 1 comment
Closed
3 tasks done

[Feature] Iceberg support for view materialization #341

dataders opened this issue Nov 1, 2024 · 1 comment
Labels
type:enhancement New feature request

Comments

@dataders
Copy link
Contributor

dataders commented Nov 1, 2024

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-adapters functionality, rather than a Big Idea better suited to a discussion

Describe the feature

with dbt-labs/dbt-snowflake#321, dbt-snowflake now supports Iceberg table format with table_format="iceberg" for table, incremental, dynamic, and snapshot materializations. Other adapters already support Iceberg table format, and will hopefully soon standardize on the same config.

However, there is now a spec for Iceberg views. While not yet supported in any of the adapters that we support (except for Vanilla Spark), I expect that we'll see them eventually.

Perhaps this means that table_format might also soon be an option we'd want to extend to the view materialization. Maybe it's view_format, but I think it makes more sense to be consistent in config naming so that table_format may be set for a group of tables & views that a user desires to write to an Iceberg catalog.

{{
  config(
    materialized="view",
    table_format="iceberg"
)}}

Describe alternatives you've considered

No response

Who will this benefit?

from @Fokko:

With Iceberg having view support now, I think dbt and Iceberg are going to be a great fit and is going to open up the analytics landscape where you can do the heavy lifting with one engine and do the final analytics with another
The version of views currently is still bound to the engine that makes them (it is just SQL that's being stored), but we're transitioning into a intermediate-representation like Substrait, but that's going to need a bit more time/work

Are you interested in contributing this feature?

No response

Anything else?

No response

@dataders dataders added type:enhancement New feature request triage:product In Product's queue labels Nov 1, 2024
@amychen1776 amychen1776 removed the triage:product In Product's queue label Nov 4, 2024
@amychen1776
Copy link
Contributor

While I love this issue - since no current warehouses currently support iceberg views, to keep our repo clean I think it makes sense to close this until this is a functionality that actually exists and users can request support for this.

@amychen1776 amychen1776 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:enhancement New feature request
Projects
None yet
Development

No branches or pull requests

2 participants