Skip to content
This repository has been archived by the owner on May 17, 2024. It is now read-only.

Support for dynamic alias names in dbt #529

Closed
btello opened this issue Apr 26, 2023 · 6 comments · Fixed by #652
Closed

Support for dynamic alias names in dbt #529

btello opened this issue Apr 26, 2023 · 6 comments · Fixed by #652
Assignees
Labels
--dbt Issues/features related to the dbt integration enhancement New feature or request

Comments

@btello
Copy link

btello commented Apr 26, 2023

Is your feature request related to a problem? Please describe.
In our environment, we leverage dynamic alias names using dbt's generate_alias_name macro:

https://docs.getdbt.com/docs/build/custom-aliases#generate_alias_name

For example, let's say we have a model named foo. In dev the model gets aliased to stage__foo and in production the alias is just foo.

A good breakdown of why we implemented things this way as well as the actual code we're using in generate_alias_name can be found on this blog post:

https://medium.com/data-manypets/how-to-customise-dbts-model-naming-for-easy-developing-on-production-1238559a939a

The problem is that when I run data-diff --dbt it uses the dev version of the alias for both the dev and prod targets and ultimately this means I can't use data-diff in my environment (which I very much want to do :-) ).

Example:

> data-diff --dbt
Running with data-diff=0.7.4

dbxyz.stage.stage__foo <> dbxyz.dbt_dev.stage__foo
New model or no access to prod table.

Describe the solution you'd like

I would like to be able to use data-diff in an environment where I have customized dynamic alias names as described above.

Describe alternatives you've considered

I could add a static custom alias to each of my models but I've got several hundred models and this solution would just be very error prone.

Additional context
None

@btello btello added the enhancement New feature or request label Apr 26, 2023
@dlawin dlawin added the --dbt Issues/features related to the dbt integration label May 8, 2023
@followingell
Copy link

+1 for this

@dlawin
Copy link
Contributor

dlawin commented Jul 5, 2023

Hi @btello, there is support for dynamic model names now via the --state flag. We could also add support via configuration, but it's unlikely to be a big priority now that --state is available.

https://docs.datafold.com/development_testing/open_source#configure-your-dbt-project
Screenshot 2023-07-05 at 10 22 48 AM

@followingell
Copy link

followingell commented Jul 5, 2023

@dlawin Thanks for the reply!

After testing --state doesn't work for my implemetaion, details below 🙂
For our dev target we utilise the {{ node.name }} (AKA model file name) as the alias and for the prod target we use the custom_alias_name.

Is there a way that I can compare the same model across dev and prod given the above setup? I believe the package takes the alias and applies it to both targets regardless of actual database behaviour:

  • Actual dev target database identifier: dev_catalog.specific_user_dev_schema.node_name.

  • Actual prod target database identifier: prod_catalog.project_schema.custom_alias_name

  • Datafold dev identifier: dev_catalog.specific_user_dev_schema.node_name

  • Datafold prod identifier: prod_catalog.project_schema.node_name

@dlawin dlawin removed the triage label Jul 5, 2023
@dlawin dlawin self-assigned this Jul 5, 2023
@dlawin
Copy link
Contributor

dlawin commented Jul 5, 2023

It seems using model.alias in both manifests is not always accurate, we should use relation_name instead, that seems to be the true representation of the model's deployed name

@dlawin
Copy link
Contributor

dlawin commented Aug 2, 2023

Hey @btello @followingell this should be resolved in versions >=0.8.0 via use of the --state flag (we were inadvertently using the alias from the "dev" manifest)

Duplicate issue: #646

Can you reopen this issue if there are any remaining problems?

@dlawin dlawin closed this as completed Aug 2, 2023
@dlawin dlawin linked a pull request Aug 2, 2023 that will close this issue
@followingell
Copy link

@dlawin Finally got around to testing this and it's working great, thank you!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
--dbt Issues/features related to the dbt integration enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants