Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move Asset user facing components to task_sdk #43773

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

Move Asset user facing components to task_sdk #43773

wants to merge 23 commits into from
+892 −732

Conversation

Lee-W
Copy link
Member

@Lee-W Lee-W commented Nov 7, 2024 •
edited
Loading

Why

As part of AIP-72

Closes: #43619

What

  • Move "Asset", "AssetAll", "AssetAny", "Dataset", "Model", "@asset" from Airflow Core to task_sdk
  • Import Asset from task_sdk for common.compat provider
  • Fix ImportError handling in common.compat provider
    • Add a temporary workaround for this error for the following providers (which should be removed after common.compat provider change in this PR has been released)
      • providers/openlineage
      • providers/google
      • providers/amazon

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-W Lee-W added the legacy api Whether legacy API changes should be allowed in PR label Nov 7, 2024
@Lee-W Lee-W force-pushed the move-asset-to-task-sdk branch 6 times, most recently from c7ecee5 to 5c8d6e2 Compare November 7, 2024 10:02
@Lee-W Lee-W marked this pull request as ready for review November 7, 2024 11:40
@Lee-W Lee-W force-pushed the move-asset-to-task-sdk branch 3 times, most recently from f0ec326 to 324efc0 Compare November 15, 2024 10:27
@Lee-W
feat(task_sdk): Move asset from Airflow core to task_sdk
0c21f9c
@Lee-W
feat(task_sdk): Move assets.metadata to task_sdk.definitions.asset
18f9cc7
@Lee-W
fix(providers/amazon): fix common.compat provider ImportError handling
7e4735f
@Lee-W
fix(providers/google): fix common.compat provider ImportError handling
c1506f7
@Lee-W
fix(providers/openlineage): fix common.compat provider ImportError ha…
7d10ad6 
…ndling
@Lee-W
fix(provider/common/compat): fix common.compat provider ImportError h…
366325a 
…andling
@Lee-W
feat(task_sdk): expose Model
9b871f8
@Lee-W
docs(nesfragements): update how asset module should be imported
d05660c
@Lee-W
fix(task_sdk): fix 2_10 compatibility
dd45e4a
@Lee-W
feat(common.compat): use version to decide how to import assets inste…
2631b65 
…ad of exception
@Lee-W
feat(providers/common.compat): use airflow version instead of excepti…
5dfb55f 
…on to return compat method
@Lee-W
refactor(providers/common/compat): extract airflow version to __init__
b67c7de
@Lee-W
fix(providers): use version compare to decide whether to import asset
52201a7
@Lee-W
feat(decorators/asset): move @asset to task_sdk
8705448
@Lee-W
refactor(asset): rename _AssetAliasCondition as AssetAliasCondition
c93d9a2
@Lee-W
feat(task_sdk): make airflow.sdk.definitions.decoratos a package
ba285c9
@Lee-W
Revert "feat(task_sdk): make airflow.sdk.definitions.decoratos a pack…
5e809bf 
…age"

This reverts commit 324efc0.
@Lee-W
feat(task_sdk): move asset related logic in airflow.sdk.definitions.d…
1fc9da5 
…ecorators to airflow.sdk.definitions.asset.*
@Lee-W
refactor(task_sdk): move @asset to airflow.sdk.definitions.asset.deco…
79b8c0d 
…rators
@Lee-W
test(providers/amazon): remove unnecessary compat handling
149d660
@Lee-W
test(providers/google): remove unnecessary compat handling
d75b3b5
@Lee-W
test(openlineage): remove unnecessary compat handling
01426cd
@Lee-W
fix(provider/openlineage): fix how asset compat is handled
eaec593
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@uranusjr uranusjr uranusjr left review comments

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin bolkedebruin is a code owner

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@mobuchowski mobuchowski Awaiting requested review from mobuchowski mobuchowski is a code owner

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

@jscheffl jscheffl Awaiting requested review from jscheffl jscheffl is a code owner

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:API Airflow's REST/HTTP API area:lineage area:providers area:Scheduler including HA (high availability) scheduler area:serialization area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK area:webserver Webserver related Issues kind:documentation legacy api Whether legacy API changes should be allowed in PR provider:common-compat
Projects
AIP-72 - Task Execution Interface and SDK
Status: In Progress
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Move Asset user-facing components to task_sdk
2 participants
@Lee-W @uranusjr