Status of testing Providers that were prepared on March 04, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.19.0rc1

• Implement filter_permitted_menu_items in AWS auth manager (#37627): @vincbeck
• Implement batch_is_authorized_* APIs in AWS auth manager (#37430): @vincbeck
• Fix init checks for aws eks (#37674): @okirialbert
• init checks for aws gcs_to_s3 (#37662): @okirialbert
• Use named loggers instead of root logger (#37801): @Taragolis
• Avoid non-recommended usage of logging (#37792): @Taragolis
• Unify aws_conn_id type to always be str | None (#37768): @eejbyfeldt
• Limit pandas to <2.2 (#37748): @Taragolis
• Implement AIP-60 Dataset URI formats (#37005): @uranusjr
• Bump min versions of openapi validators (#37691): @potiuk
• Update action names in AWS auth manager (#37572): @vincbeck

Provider apache.beam: 5.6.2rc1

• Add Python 3.12 exclusions in providers/pyproject.toml (#37404): @potiuk

Provider apache.druid: 3.9.0rc1

• Adding optional SSL verification for druid operator (#37629): @danielbe11
• Modify ssl verification in druid operator (#37673): @danielbe11

Provider apache.hdfs: 4.3.3rc1

• Turn Pydantic into an optional dependency (#37320): @potiuk

Provider apache.hive: 7.0.1rc1

• Remove references from the code to Jira Issues (#37807): @Taragolis
• Unify aws_conn_id type to always be str | None (#37768): @eejbyfeldt
• Limit pandas to <2.2 (#37748): @Taragolis

Provider apache.livy: 3.7.3rc1

• Fix rendering LivyOperator.spark_params (#37361): @Taragolis

Provider apache.pinot: 4.3.1rc1

• Bump lower-limit for pinotdb (#37690): @potiuk

Provider celery: 3.6.1rc1

• Remove pid arg from celery option to fix duplicate pid issue, Move celery command to provider package (#36794): @pavansharma36
• Change AirflowTaskTimeout to inherit BaseException (#35653): @hterik
• Migrate executor docs to respective providers (#37728): @RNHTTR

Provider cncf.kubernetes: 8.0.1rc1

• Immediately fail the task in case of worker pod having a fatal container state (#37670): @dirrao
• Skip pod cleanup in case of pod creation failed (#37671): @dirrao
• Avoid non-recommended usage of logging (#37792): @Taragolis
• Migrate executor docs to respective providers (#37728): @RNHTTR

Provider common.sql: 1.11.1rc1

• Make executemany keyword arguments only in DbApiHook.insert_rows (#37840): @Taragolis
• Limit pandas to <2.2 (#37748): @Taragolis

Provider dbt.cloud: 3.7.0rc1

• add reuse_existing_run for allowing DbtCloudRunJobOperator to reuse existing run (#37474): @Lee-W

Provider docker: 3.9.2rc1

• Fix construct docker.TLSConfig for docker>=7 (#37481): @Taragolis

Provider exasol: 4.4.3rc1

• Limit pandas to <2.2 (#37748): @Taragolis

Provider google: 10.16.0rc1

• CloudRunExecuteJobOperator: Add project_id to hook.get_job calls (#37201): @freyrsae
• Add developer token as authentication method to GoogleAdsHook (#37417): @LeonGraveland
• Add GKEStartKueueInsideClusterOperator to support Kueue inside clusters (#37072): @VladaZakharova
• Support location for BigQueryInsertJobOperator in deferrable mode (#37282): @moiseenkov
• Add support for impersonation chain to GKEStartPodOperator in deferrable mode (#37486): @m1racoli
• Create GKEStartJobOperator and KubernetesJobOperator (#36847): @MaksYermak
• Fix invalid deprecation of DataFusionPipelineLinkHelper (#37755): @Taragolis
• fix templated field assignment google/cloud/operators/compute.py (#37659): @romsharon98
• Fix bigquery_to_mysql init checks (#37653): @okirialbert
• Fix Async GCSObjectsWithPrefixExistenceSensor xcom push (#37634): @pankajastro
• Fix GCSSynchronizeBucketsOperator timeout error (#37237): @kevgeo
• fix: Signature of "insert_rows" incompatible with supertype "DbApiHook" (#37391): @rawwar
• fix: use offset-naive datetime in _CredentialsToken (#37539): @m1racoli
• Use hook.wait_for_operation in DataprocInstantiateInlineWorkflowTemplateOperator (#37145): @edin-tapad
• Fix typo on DataflowStartFlexTemplateOperator documentation (#37595): @sarafonseca-123
• Make executemany keyword arguments only in DbApiHook.insert_rows (#37840): @Taragolis
• Unify aws_conn_id type to always be str | None (#37768): @eejbyfeldt
• Limit pandas to <2.2 (#37748): @Taragolis
• Remove broken deprecated fallback into the Google provider operators (#37740): @Taragolis
• Implement AIP-60 Dataset URI formats (#37005): @uranusjr
• resolve template fields init checks for bigquery (#37586): @okirialbert
• Update docs for the DataprocCreateBatchOperator (#37562): @molcay
• Replace usage of datetime.utcnow and datetime.utcfromtimestamp in providers (#37138): @Taragolis

Provider hashicorp: 3.6.4rc1

• Fix unexpected keyword argument 'secret_path' in secrets.kv.v2 (#37626): @tungbq

Provider http: 4.10.0rc1

• Change the URL building in HttpHookAsync to match the behavior of HttpHook (#37696): @alejorodriguez96
• Return back removed public HttpHook's method (#37738): @Taragolis

Provider microsoft.azure: 9.0.1rc1

• fix: Pass proxies config when using ClientSecretCredential in AzureDataLakeStorageV2Hook (#37103): @dabla

Provider microsoft.psrp: 2.6.0rc2

• Add PSRP connection type (#34766): @arjunanan6

Provider mysql: 5.5.4rc1

• Unify aws_conn_id type to always be str | None (#37768): @eejbyfeldt
• Implement AIP-60 Dataset URI formats (#37005): @uranusjr

Provider openlineage: 1.6.0rc1

• [OpenLineage] feat: Add metrics for event_size and extraction time (#37797): @kacpermuda
• [OpenLineage] feat: Add OpenLineage support for File and User Airflow's lineage entity (#37744): @kacpermuda
• [OpenLineage] Add support for JobTypeJobFacet properties. (#37255): @mattiabertorello
• [OpenLineage] chore: Update comments and logging in OpenLineage ExtractorManager (#37622): @kacpermuda
• [OpenLineage] Check for disabled operators in DefaultExtractor.extract_on_complete (#37392): @kacpermuda
• [OpenLineage] docs: Update whole OpenLineage Provider docs. (#37620): @kacpermuda

Provider opensearch: 1.1.2rc1

• Fix "Exception statement has no effect" (#37722): @Taragolis

Provider postgres: 5.10.2rc1

• Implement AIP-60 Dataset URI formats (#37005): @uranusjr

Provider presto: 5.4.2rc1

• Limit pandas to <2.2 (#37748): @Taragolis

Provider salesforce: 5.6.3rc1

• Limit pandas to <2.2 (#37748): @Taragolis

Provider smtp: 1.6.1rc1

• Fix backwards compatibility for SMTP provider (#37701): @potiuk
• Deprecate smtp configs in airflow settings / local_settings (#37711): @vchiapaikeo

Provider telegram: 4.4.0rc2

• Add telegram connection with documnetation (#37515): @romsharon98

Provider trino: 5.6.3rc1

• Limit pandas to <2.2 (#37748): @Taragolis
• Implement AIP-60 Dataset URI formats (#37005): @uranusjr

Provider weaviate: 1.3.3rc1

• Limit pandas to <2.2 (#37748): @Taragolis

All users involved in the PRs:
@vincbeck @VladaZakharova @pankajastro @dabla @m1racoli @RNHTTR @danielbe11 @Taragolis @pavansharma36 @kacpermuda @moiseenkov @dirrao @kevgeo @okirialbert @eejbyfeldt @mattiabertorello @MaksYermak @hter

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[moiseenkov]

hi,
#37282 works as expected

[kacpermuda]

All the PR's for the Provider openlineage: 1.6.0rc1 works as expected.

[Taragolis]

I've check my changes, most of them pretty simple.

• Replace usage of datetime.utcnow and datetime.utcfromtimestamp in providers #37138 in google Provider changes should be fixed by fix: use offset-naive datetime in _CredentialsToken #37539, cc: @m1racoli
• Fix rendering LivyOperator.spark_params #37361: I've checked that it rendered correctly, it would be nice if @LudvigLundberg have a time to look on changes as original reporter of the issue

[LudvigLundberg]

I've check my changes, most of them pretty simple.

• Replace usage of datetime.utcnow and datetime.utcfromtimestamp in providers #37138 in google Provider changes should be fixed by fix: use offset-naive datetime in _CredentialsToken #37539, cc: @m1racoli
• Fix rendering LivyOperator.spark_params #37361: I've checked that it rendered correctly, it would be nice if @LudvigLundberg have a time to look on changes as original reporter of the issue

Can confirm that the changes in #37361 have solved the issue I was experiencing! Big thanks for fixing it!

[vchiapaikeo]

#37711 looks good.

Huge thank you to everyone that helped resolve this mini fire last week (Jarek, Andrey, Elad 🙇 )

[arjunanan6]

#34766 works as expected. :)

[m1racoli]

I can confirm #37486. It works as expected.

Although I had to manually require apache-airflow-providers-cncf-kubernetes>=8.0.0 for my DAG to parse. Before that I had version 7.14.0 installed and getting

Broken DAG: [/usr/local/airflow/dags/test_gke.py] Traceback (most recent call last):
  File "/usr/local/airflow/dags/test_gke.py", line 3, in <module>
    from airflow.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperator, OnFinishAction
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py", line 35, in <module>
    from airflow.providers.cncf.kubernetes.operators.job import KubernetesJobOperator
ModuleNotFoundError: No module named 'airflow.providers.cncf.kubernetes.operators.job'

I assume this is because of #36847. Are we missing a requirement for apache-airflow-providers-cncf-kubernetes>=8.0.0 in the google provider's dependencies?

[moiseenkov]

I assume this is because of #36847. Are we missing a requirement for apache-airflow-providers-cncf-kubernetes>=8.0.0 in the google provider's dependencies?

@m1racoli , yes you're right. Thank you!
@eladkal , can we please exclude google from this release for now, merge the fix #37899 and release google afterwards with RC2?

[potiuk]

All my changes are there :)

[pavansharma36]

Verified #36794

[eladkal]

@moiseenkov I don't think this qualify for RC2. It's not a regression and there is a simple fix (bump k8s provider if not using latest). Also the RC1 contains many features and fixes that not worth delaying.
We can always improve the mechanism and have better error message for next release.

[moiseenkov]

@moiseenkov I don't think this qualify for RC2. It's not a regression and there is a simple fix (bump k8s provider if not using latest). Also the RC1 contains many features and fixes that not worth delaying.
We can always improve the mechanism and have better error message for next release.

I totally agree. We will adjust error messages in the google provider package for the next release and will follow this convention from now. Thank you!

[pankajastro]

#37634 looks good. thanks!

[eladkal]

Thank you everyone. Providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[dabla]

Maybe a bit late but I tested #37103 and I can confirm it works :)

[potiuk]

Never too late for good news :)