Redirect old location module imports to standard provider #43610
Conversation
|
@ashb If you feel this is not correct :), please suggest any alternatives or if there is more optimised way to workout these. |
|
Added tests to check if imports are working or not :) |
gopidesupavan
force-pushed
the
redirect-core-module-references-to-standard-provider
branch
from
6ff618f
to
57337f6
Compare
gopidesupavan
force-pushed
the
redirect-core-module-references-to-standard-provider
branch
from
8f923be
to
c6eaf74
Compare
Co-authored-by: Amogh Desai <[email protected]>
romsharon98
force-pushed
the
redirect-core-module-references-to-standard-provider
branch
from
c6eaf74
to
969da8d
Compare
|
Should we somehow check if "all" moved imports are handled? Not sure if we can do it easily though. |
--------- Co-authored-by: Amogh Desai <[email protected]>
…ache#43610)" There is a very interesting breaking change introduced in Python 3.11 that will likely mean that we should not use MetaPathFinder for old standard provider classes redirection. There was a change introduced in Python 3.11 that caused that module might not be found in some cases (for example when unit test patches the path) when the module is loaded as a different module (i.e. a.b differs from sys.modules['a.b']) This is tracked in CPython via: python/cpython#117860 This causes standard operator's tests fail in Python 3.11 and 3.12 when the providers/tests/standard/test_module_redirect_finder.py is executed before - i.e. the standard modules are loaded as old modules. This reverts commit 5de2e73.
…3610)" (#43946) There is a very interesting breaking change introduced in Python 3.11 that will likely mean that we should not use MetaPathFinder for old standard provider classes redirection. There was a change introduced in Python 3.11 that caused that module might not be found in some cases (for example when unit test patches the path) when the module is loaded as a different module (i.e. a.b differs from sys.modules['a.b']) This is tracked in CPython via: python/cpython#117860 This causes standard operator's tests fail in Python 3.11 and 3.12 when the providers/tests/standard/test_module_redirect_finder.py is executed before - i.e. the standard modules are loaded as old modules. This reverts commit 5de2e73.
Yeah have ran basic python operator but that is on 3.9 versions. now that MethPathFinder not supported in 3.11 and 3.12, so will check if we can do anything :) |
Just to document: MetaPath solution suffers from bug introduced in Python 3.11 python/cpython#117860 which is affecting unittest mock imports but also apparently goes beyond just unit tests - for example affecting Apprise 1.8 PrefectHQ/prefect#13314 after they implemented refactors internally. Maybe we can come back (@ashb ?) to the old ways of doing it - we had implemented deprecation utils in v1-> v2 migration that we can apply again: https://github.com/apache/airflow/blob/v2-10-test/airflow/operators/__init__.py - they did the job, the only problem was that the old "package" will have remain there - with an Also - I think such WDYT @gopidesupavan @ashb -> I think while the MetaPath was a nice idea, but it's a bit too risky. |
|
After discussion in Slack @gopidesupavan -> maybe you could (at least for now - attempt to follow the PEP-562 approach and redo the change and we can revisit it later). Ideally simple script to generate moved class dicts would be cool to have so that we can re-apply it as we move new classes. |
Yeah agree, better to avoid metapath. let me get the changes with PEP 562 |
|
BTW. This is a bit of an assumption that PEP 562 is going to be "ok" - it should work as this really creates aliases to objects in a different package, and we did not have problems with Airflow 2 deprecations even on airflow 2.11 and 2.12 - so it's likely to work (but let's see :).) |
|
And BTW. when you create a PR - you can assign "all versions" and "full tests needed" labels to your PR @gopidesupavan before you create it - this way it will run on all versions of Python/Databases and K8S (and will trigger all tests) |
--------- Co-authored-by: Amogh Desai <[email protected]>
…ache#43610)" (apache#43946) There is a very interesting breaking change introduced in Python 3.11 that will likely mean that we should not use MetaPathFinder for old standard provider classes redirection. There was a change introduced in Python 3.11 that caused that module might not be found in some cases (for example when unit test patches the path) when the module is loaded as a different module (i.e. a.b differs from sys.modules['a.b']) This is tracked in CPython via: python/cpython#117860 This causes standard operator's tests fail in Python 3.11 and 3.12 when the providers/tests/standard/test_module_redirect_finder.py is executed before - i.e. the standard modules are loaded as old modules. This reverts commit 5de2e73.
We have moved the operators/sensors/hooks/utils to standard provider from core area location.
This change is to support in ariflow 3.0.0 when user tries to import from old location,
it redirects to standard provider.
So far the following operators/sensors/hooks/utils has been moved to standard provider.
#41564
#42506
#42081
#42252
related: #43582
related: #43641
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.