-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move Pod*Exceptions to separate module to avoid unnecessary slow imports in CLI #45759
Conversation
7997f2f
to
c98b7d9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice one! Love the timing results
c98b7d9
to
4444c0d
Compare
I applied "full tests needed" and rebased it - I think that one should run full suite of tests due to potential compatibility issues. |
Looks like the only failure is unrelated one (already failing on parent commit, same failure as here: #45727 (comment) - |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh, I love these kinds of PRs :)
Added an issue for the flaky stuff #45774 |
It looks like another "caplog issue" |
Co-authored-by: Igor Kholopov <[email protected]>
Co-authored-by: Igor Kholopov <[email protected]>
Co-authored-by: Igor Kholopov <[email protected]>
Pod*Exceptions has been moved from core to providers back in de92a81, but kept in
airflow/exceptions.py
under try-case for backwards compatibility and allow for usage of an older k8s provider with a newer core airflow.This resulted in 2 unintended side-effects:
airflow
resulted in importingkubernetes
client. It is effectively the most expensive import out of all CLI does for any command, even though it is used by very few commands. Here are the timings of a trivialairflow dag-processor --help
pod_generator.py
, the import fromexceptions.py
failed even if providers were up to date with recursive import. But it was caught by exception handling, resulting infrom airflow.exceptions import PodMutationHookException
andfrom airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookException
pointing to different classes defeating the purpose of the fallback.This PR addresses both by moving PodGenerator exceptions to the separate module that only import its base class. This keeps imports backwards compatible, doesn't attempt to load k8s modules and fixes the divergence of exception classes in
airflow.providers.cncf.kubernetes.pod_generator
andairflow.exceptions
.It is worth noting, that all usages of those exceptions in core Airflow also has been cleaned up since then, so if you believe that we need to remove those exceptions from
airflow/exceptions.py
, please let me know.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.