Standard provider python operator #42081
Conversation
gopidesupavan
requested review from
potiuk,
ashb,
jedcunningham,
eladkal,
o-nikolas,
mobuchowski and
hussein-awala
as code owners
boring-cyborg
bot
added
area:dev-tools
area:providers
provider:amazon-aws
tests/system/providers/apache/kafka/example_dag_event_listener.py
Outdated
Show resolved
Hide resolved
|
cc: @romsharon98 |
gopidesupavan
force-pushed
the
standard-provider-python-operator
branch
from
e873eaa
to
58d1db9
Compare
gopidesupavan
force-pushed
the
standard-provider-python-operator
branch
from
5f2d337
to
3cbf146
Compare
|
Some of the tests are failing due to compatibility issues. It appears that the changes related to the https://github.com/apache/airflow/actions/runs/11308098417/job/31450685352?pr=42081#step:11:12372 |
|
@potiuk It appears that this PR needs to be initiated from the Apache repository, or it should be someone with committer privileges. The PR involves modifications in the Here it is removing https://github.com/apache/airflow/actions/runs/11308098408/job/31450591466#step:4:219 For Any thoughts please ? |
looks like the tests are going through fine now.
looks like tests are running..now.. |
|
Compat tests 2.10.1 failing because python operator has changes related to airflow 3.0.0, should i update back changes to support in 2.10.X? |
It looks like it's more of "unit test compatibility" than backporting it. I think in this case simply tests should be fixed to be able to handle past providers. We do not need to backport this change, but we need to make sure that the assertions work when |
thanks jarek, When installing Airflow 2.10.1, the Python operator is included as part of the core package. Should assertions be checked against the core Python operator (i.e., airflow.operators.python) or against airflow.providers.standard.operators.python? Opting for the former would require code changes due to the adjustments already made for Airflow 3. Do you agree that if Airflow 2.10.1 is installed, validation tests should be conducted on airflow.operators.python, as long as the tests remain compatible? |
ah i got it what you mean by 'assertions work', need to make some adjustments in the python operator tests related to new feature updates. hope that works.. |
Yes. Some examples and detailed explanation on how to do this "test_compatibility" are here: https://github.com/apache/airflow/blob/main/contributing-docs/testing/unit_tests.rst#implementing-compatibility-for-provider-tests-for-older-airflow-versions |
gopidesupavan
force-pushed
the
standard-provider-python-operator
branch
from
2db922d
to
3e743df
Compare
|
One failure related to spell check this will fix #43538 |
|
Are we good to merge this ? 😄 |
|
Go ahead and do the honors @gopidesupavan -> That's an interesting one as first(?) serious one to merge as a committer.. Don't be shy. We can always revert it if it breaks things ;) |
woohoooo merged thank you all for the help and review here, really appreciate it 😄 |
* add core.time provider * add source-date-epoch * change core to essentials * revert external task sensor location * add provider to airflow_providers_bug_report list * change new provider name to standard * add integration * revert hatch_build * move examples back to airflow core * change sensors example dags paths * remove init * revert howto docs * change provider as not-ready * Move python operator to standard provider * fix test failures * move python operator inside standard/operators * rebase and update python imports * keep standard provider in hatch_build * update selective checks and dependencies * update imports and dependency file * fix compat import errors * fix selective_checks and imports from compat * remove imports from compat for pythonoperator * update run_tests.py python branching test paths * add setup method in python operator tests * check imports in python operator tests * fix imports and move python virtual env scripts inside standard provider * update selective checks tests and imports in test_python * move python_virtualenv tests inside standard provider * fix pre-commit static checks * use compat provider for standard provider imports * fix _SERIALIZERS ModuleNotFoundError and static checks * import _SERIALIZERS for AIRFLOW_2_10 or higher versions * fix ruff format * fix static checks in test_dag * fix test_provide_context_does_not_fail failure: add TypeError to pytest assert --------- Co-authored-by: romsharon98 <[email protected]>
Moving python operators/sensors to standard provider
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.