Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "@asset" to decorate a function as a DAG and an asset #41325

Merged
merged 29 commits into from
Nov 14, 2024

Conversation

uranusjr
Copy link
Member

@uranusjr uranusjr commented Aug 8, 2024

Why

As part of AIP-75

What

Add an asset decorator which can be used to decorate a function and create a DAG and an asset.

@asset(uri="s3://bucket/object", schedule=None)
def asset2_producer(self, context, asset1_producer):
    print(self)
    print(context["inlet_events"][asset1_producer])

In the example above, it create a DAG(dag_id="asset2_producer", schedule=None, ...) and an Asset(name="asset2_producer", uri="s3://bucket/object"). Inside the function, self can be used to access the asset with name asset2_producer, context is just like context in a normal task and asset1_producer is used to access an asset with name=asset1_producer

Closes: #42314

@uranusjr uranusjr force-pushed the asset-decorator branch 2 times, most recently from f0ea15c to 017cdd9 Compare August 9, 2024 10:47
@Lee-W Lee-W mentioned this pull request Sep 23, 2024
2 tasks
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 24, 2024
@uranusjr uranusjr removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 25, 2024
@Lee-W Lee-W force-pushed the asset-decorator branch 3 times, most recently from 91fb531 to b6bb01b Compare October 25, 2024 05:41
@Lee-W Lee-W force-pushed the asset-decorator branch 4 times, most recently from 2b7dbda to cc11458 Compare October 30, 2024 11:28
airflow/decorators/assets.py Outdated Show resolved Hide resolved
@Lee-W
Copy link
Member

Lee-W commented Oct 30, 2024

@uranusjr I tried to make the @asset work and added an example DAG and some unit tests. The next thing I will work on is accessing self and context. But would be great if you could take a look in advance in case I'm doing something wrong. Thanks!

@Lee-W Lee-W force-pushed the asset-decorator branch 5 times, most recently from 6257830 to 0ef189e Compare November 1, 2024 10:21
@Lee-W Lee-W marked this pull request as ready for review November 1, 2024 10:37
@Lee-W Lee-W changed the title [WIP] Implement asset definition creating a DAG Add "@asset" to decorate a function as a DAG and an asset Nov 1, 2024
@Lee-W Lee-W force-pushed the asset-decorator branch 2 times, most recently from 3ac1a6a to a7e4dcc Compare November 4, 2024 01:34
airflow/assets/__init__.py Outdated Show resolved Hide resolved
airflow/assets/__init__.py Outdated Show resolved Hide resolved
Lee-W and others added 22 commits November 14, 2024 16:07
The validate_identifier validator already checks the length, so we don't
need an extra one doing that.
@uranusjr
Copy link
Member Author

GitHub seems to have changed something and now I can’t approve my own PR. So consider this as a manual approval. ✅

@Lee-W
Copy link
Member

Lee-W commented Nov 14, 2024

GitHub seems to have changed something and now I can’t approve my own PR. So consider this as a manual approval. ✅

Thanks! Let me merge it now

@Lee-W Lee-W merged commit b3362f8 into apache:main Nov 14, 2024
52 checks passed
@Lee-W Lee-W deleted the asset-decorator branch November 14, 2024 09:14
@potiuk
Copy link
Member

potiuk commented Nov 14, 2024

GitHub seems to have changed something and now I can’t approve my own PR. So consider this as a manual approval. ✅

They are really experimenting with Approvals it seems......

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement skeleton @asset decorator
4 participants