2023 High level roadmap for the dbt Labs Adapters teams - including dbt Cloud, and the future of dbt-spark

[Fleid]

Disclaimers:

• As a nod to DRY, I'm stealing some of what @dataders wrote here. Thanks Anders!
• I know this is partially a discussion about dbt Cloud, happening in the dbt core repository. But I wanted a place to have a discussion on a topic that is of interest to a lot of us, a bit more permanent than a slack thread. So here we are!

Introduction, literally

First who am I to talk about this topic? I'm Florian, the product manager for Adapters at dbt Labs. My scope is adapters in core (the default materializations, dbt-postgres, dbt-redshift, dbt-snowflake, dbt-bigquery and dbt-spark) and what we call the cloud adapters, meaning how and which plugin we support in dbt Cloud (adding to the previous list dbt-databricks and soon dbt-trino).

Just to be clear, I am not responsible for all the good stuff happening there - that's primarily your contributions and the contributions of our stellar development team - but rather that if something you want is not being prioritized, I'm the one to blame! Well @jtcohen6 and @dbeatty10 are also to blame. Don't worry about us though: when things go wrong, our solution is to point at one another spiderman-style. It has worked pretty well so far ;)

Roadmap

After that preamble, let's get down to it.

I'm writing this post following a series of asks for us to support dbt-athena in dbt Cloud. It is currently one of our most-used community adapters. But at this point in time, it doesn't look like we will be able to support it in Cloud in 2023. There is just too much to do for us, before we can make it happen.

What is that? Well here's what we're thinking for 2023. Some of it is started, some planned, some is aspirational:

dbt core

Please see here for the full story

• We're catching up on PR reviews and bug solving, and will be ramping up on enhancement requests
• We're tackling streaming via materialized views, materialized tests, and following work aiming for a nice end-to-end story by Coalesce in October. More on that soon via a blog post that I'll link here when it's published in the coming weeks
• A series of independent work items aiming at major version compatibility for adapters (if you have dbt-redshift 2.0 installed, it works with any 2.x version of core) with things like profile rationalization, dependency snapshots, better credential management etc. This is not about achieving that in 2023, but making significant progress towards it
• And a list of cross warehouse features that need some love : caching, leveraging clones, dynamic data masking, etc

dbt Cloud

• New adapters
  • Starburst & Trino adapter (in public preview now — GA at the end of next month!)
  • Azure Synapse (no real timeline yet — just end of the year)
• Improvement on existing adapters
  • Catching up on existing profile gaps (BQ, Redshift)
  • Adding support for SSO/OAuth where available (Redshift soon, Databricks, Starburst/Trino)
• Evolving the platform supporting adapters in dbt Cloud, paving the way for a Bring-Your-Own adapter model

Not enough time

We have a lot to tackle, as we will also be supporting @jtcohen6 and @MichelleArk initiatives around multi-project deployments. So I'm pretty sure we won't be able to do everything. There will be some hard cuts in that list. To minimize regrets, please be vocal in the comments below in terms of priorities, or you see something missing.

One area where we are considering scaling back our involvement is dbt-spark. Now that dbt-databricks has joined the party, I have to question the return on investment of "actively" maintaining dbt-spark. Not only does that repository needs a lot of attention, but Spark is often a challenging platform to work with, and we don't have the bandwidth to build and maintain an expertise of it in-house. All of that for a small user base, highly capable of figuring things out for themselves. I'm not definitive on the topic yet, here also I'd appreciate your feedback and ideas. Be loud if you think it's a bad idea.

Thanks for taking the time to read this, I'm looking forward to reading your thoughts below ;)

[paulmech]

Hi @Fleid - I noticed you've linked to the older Athena dbt adapter; dbt docs own warehouse setup for Athena links to the newer, community maintained adapter.

The newer dbt athena community maintained adapter is the only one that I believe supports modern versions of dbt-core, as well as many of the latest features of the underlying service.

I appreciate this discussion may be more about requests by dbtCloud users to have an Athena adapter available in that product. However, continuing to link to the outdated dbt-athena adapter makes it difficult for all new users to find the latest community maintained Athena adapter, within search engine results. If it's appropriate, maybe you could update the link to point to the latest dbt athena adapter

[dbeatty10]

If it's appropriate, maybe you could update the link to point to the latest dbt athena adapter

Thanks for that call-out @paulmech ! That was an accidental oversight, and I've updated the link per your suggestion (on behalf of @Fleid). Please let us know if this comes up anywhere else, and we'll get it updated.

[Fleid]

I was actively telling myself to not make that mistake, and still ended up messing my link... I need vacations ;)

[paulmech]

Hi @Fleid - (and @dataders - thanks for the prompt on Slack to comment on the discussion)

I couldn't see it written on the dbt website, but it seems implicit that verifying a dbt adapter means that it will be hosted under the dbt-labs GitHub organization, and to some extent, supported by dbt-labs adapter team.

This seems to correspond to this discussion and your roadmap, in the sense that there is a desire to de-list dbt-spark, and add verification for synapse, trino/starburst etc.

I am curious if is permissible that an adapter can be both

1. independent and community maintained
2. a verified adapter endorsed (but not maintained or supported) by dbt-labs

This definition is important, because elsewhere in the community, projects look for user support and sometimes to the endorsement of dbt-labs as a guide for whether to support data warehouses:

• Lighthouse support for Athena - Support For AWS Athena As Source lightdash/lightdash#3554 (comment)
• Evidence-Dev closed and un-merged PR for Athena Support - Athena support evidence-dev/evidence#473

I understand the risk for dbt-labs in relation to dbtCloud, of including an adapter that goes unmaintained.

However, it'd be sad to see Athena held back from awesome products like Lightdash, when it already works so well and delivers a lot of community value through its use with dbt-core.

I don't speak for the Athena community adapter maintainers but would happily help out to support the adapter's verification in its current form.

[nicor88]

Good points @paulmech !!
Somehow verifying-a-new-adapter doesn't make clear what is needed to have an adapter verified - e.g. why dbt-trino is verified (only because there is starburst, therefore a company, behind?)

@dataders / @Fleid It will be good to make clear the criteria needed to have the adapter eligible for verification.
Specifically, what do we need to do to have dbt-athena-community verified? It's important to remark that dbt-athena was forked from the original Tomme/dbt-athena to avoid dependencies from a single, but rather have a stronger community-based approach to the adapter.

[Fleid]

I couldn't see it written on the dbt website, but it seems implicit that verifying a dbt adapter means that it will be hosted under the dbt-labs GitHub organization, and to some extent, supported by dbt-labs adapter team.

That's a misconception ;)
When an adapter is verified, dbt Labs doesn't become a maintainer of the adapter. The repository is not moved under our GitHub org.

It basically means that we have a story figured out in terms of support escalation - between dbt Labs and whoever maintains that adapter. Which makes the adapter eligible to be made available in Cloud. I think that at the moment, this implies that the maintainer is an organization capable of providing enterprise-grade support. But I may be mistaken.

I'll let @dataders give more flavor to that answer, he wrote a large part of the verification program ;)