[Feature] dbt hooks

[JCZuurmond]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Dbt hooks are (optional) pre and post commands a user defines which run before and after dbt commands, respectively. For example:

• pre-run
• post-run
• pre-test
• post-test
• pre-seed
• post-seed
• ...

This allows a user to trigger custom processes during a dbt execution, and therefor integrate a custom workflow with dbt's workflow.

Describe alternatives you've considered

To break up your execution plan in steps by using selectors and then running another process in between:

1. dbt build -s 'project.staging'
2. <run hook>
3. dbt build -s 'project.marts'
4. <run hook>
5. ...

However, this does not allow you to leverage your pre and post commands with the powerful dbt node selection mechanism. Moreover, in this alternative, the job that triggers dbt needs to be aware about what happens inside your dbt project, where I like to have simple dbt triggers like dbt build or dbt build -s source:my_source+ that are unaware about what happens inside your dbt project.

Who will this benefit?

Advanced dbt users that want to have workflows that are tightly integrated with dbt's execution.

Are you interested in contributing this feature?

Always - if I find the time

Anything else?

pre- and post-hook

The pre- and post-hook that already exists in dbt are related, though, not the same. These hooks allow you to run a SQL query before or after a dbt model is ran. This feature requests wants to run a python function, which is more flexible.

Git hooks

This request is inspired by git hooks. Commonly used and known through the pre-commit package. Git hooks allow you to define pre and post command for git commands like commit, push and pull. Have a look in the .git/hooks/ folder!

[ChenyuLInx]

Hey @JCZuurmond Thanks for writing this great feature request!! It is definitely on track with the features we are thinking about! There's this custom user-defined task #2381 that's still on our radar! It will be a while(we are pretty far from there) before being able to support it/ having a stable interface for it. But this sounds a lot like what you are after!

For running custom Python code, there are a few security concerns for dbt Cloud we have, along the line of:

• allowing user to run custom untrusted code on dbt cloud
• allowing users to hit arbitrary APIs/endpoints, e.g. by adding the requests module to dbt's Jinja context

The guardrail, for now, is that we want dbt to just issue commands but not move any data around.

In the meantime, we think you will probably need to hack this together by something like:

• a custom script that parses dbt metadata artifacts to figure out which nodes were just run, information about them, etc
• a deployment mechanism to execute that script alongside dbt steps — either via Airflow/Dagster/etc, or that script also hits the dbt Cloud admin API

[jdbodyfelt]

It would be cool to implement a REST send within these - instant "dbt webhooks"!

[JCZuurmond]

The programmatic invocation of the latest release (v1.5) might be an approach to add hooks.