Skip to content

Commit

Permalink
Remove deprecated functionality from Google provider
Browse files Browse the repository at this point in the history
  • Loading branch information
moiseenkov committed Nov 15, 2024
1 parent 19303ca commit 7c1c4c6
Show file tree
Hide file tree
Showing 173 changed files with 341 additions and 8,160 deletions.
2 changes: 1 addition & 1 deletion airflow/serialization/serialized_objects.py
Original file line number Diff line number Diff line change
Expand Up @@ -1547,7 +1547,7 @@ def _serialize_operator_extra_links(cls, operator_extra_links: Iterable[BaseOper
Store the import path of the OperatorLink and the arguments passed to it.
For example:
``[{'airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleLink': {}}]``
``[{'airflow.providers.google.cloud.links.bigquery.BigQueryDatasetLink': {}}]``
:param operator_extra_links: Operator Link
:return: Serialized Operator Link
Expand Down
7 changes: 1 addition & 6 deletions docs/apache-airflow-providers-google/connections/gcp.rst
Original file line number Diff line number Diff line change
Expand Up @@ -207,11 +207,6 @@ In order for this example to work, the account ``impersonated_account`` must gra
access token, which will allow to act on its behalf using its permissions. ``impersonated_account``
does not even need to have a generated key.

.. warning::
:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreateJavaJobOperator` and
:class:`~airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator`
do not support direct impersonation as of now.

In case of operators that connect to multiple Google services, all hooks use the same value of
``impersonation_chain`` (if applicable). You can also impersonate accounts from projects
other than the project of the originating account. In that case, the project id of the impersonated
Expand Down Expand Up @@ -306,7 +301,7 @@ For example:
task_id="create-spreadsheet",
gcp_conn_id="google_cloud_default",
spreadsheet=SPREADSHEET,
delegate_to=f"projects/-/serviceAccounts/SA@{PROJECT_ID}.iam.gserviceaccount.com",
impersonation_chain=f"projects/-/serviceAccounts/SA@{PROJECT_ID}.iam.gserviceaccount.com",
)
Note that as domain-wide delegation is currently supported by most of the Google operators and hooks, its usage should be limited only to Google Workspace (gsuite) and marketing platform operators and hooks or by accessing these services through the GoogleDiscoveryAPI hook. It is deprecated in the following usages:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ returns the id of model in :ref:`XCom <concepts:xcom>` under ``model_id`` key.
This operator is deprecated when running for text, video and vision prediction and will be removed soon.
All the functionality of legacy AutoML Natural Language, Vision, Video Intelligence and new features are
available on the Vertex AI platform. Please use
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTextTrainingJobOperator`,
:class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.SupervisedFineTuningTrainOperator`,
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLImageTrainingJobOperator` or
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLVideoTrainingJobOperator`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ There are several ways to run a Dataflow pipeline depending on your environment,
dependencies must be installed on the worker. For Java, worker must have the JRE Runtime installed.
For Python, the Python interpreter. The runtime versions must be compatible with the pipeline versions.
This is the fastest way to start a pipeline, but because of its frequent problems with system dependencies,
it may cause problems. See: :ref:`howto/operator:DataflowCreateJavaJobOperator`,
:ref:`howto/operator:DataflowCreatePythonJobOperator` for more detailed information.
it may cause problems. See: :ref:`howto/operator:JavaSDKPipelines`,
:ref:`howto/operator:PythonSDKPipelines` for more detailed information.
Developer can also create a pipeline by passing its structure in a JSON format and then run it to create
a Job.
See: :ref:`howto/operator:DataflowCreatePipelineOperator` and :ref:`howto/operator:DataflowRunPipelineOperator`
Expand Down Expand Up @@ -107,7 +107,7 @@ the create job operators. The source file can be located on GCS or on the local
or
:class:`~airflow.providers.apache.beam.operators.beam.BeamRunPythonPipelineOperator`

.. _howto/operator:DataflowCreateJavaJobOperator:
.. _howto/operator:JavaSDKPipelines:

Java SDK pipelines
""""""""""""""""""
Expand Down Expand Up @@ -141,7 +141,7 @@ Here is an example of creating and running a pipeline in Java with jar stored on
:start-after: [START howto_operator_start_java_job_local_jar]
:end-before: [END howto_operator_start_java_job_local_jar]

.. _howto/operator:DataflowCreatePythonJobOperator:
.. _howto/operator:PythonSDKPipelines:

Python SDK pipelines
""""""""""""""""""""
Expand Down
14 changes: 0 additions & 14 deletions docs/apache-airflow-providers-google/operators/cloud/gcs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -174,20 +174,6 @@ Also you can use deferrable mode in this operator if you would like to free up t
:start-after: [START howto_sensor_object_exists_task_defered]
:end-before: [END howto_sensor_object_exists_task_defered]

.. _howto/sensor:GCSObjectExistenceAsyncSensor:

GCSObjectExistenceAsyncSensor
-----------------------------

:class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceAsyncSensor` is deprecated and will be removed in a future release. Please use :class:`~airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor` and use the deferrable mode in that operator.

.. exampleinclude:: /../../providers/tests/system/google/cloud/gcs/example_gcs_sensor.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_object_exists_task_async]
:end-before: [END howto_sensor_object_exists_task_async]


.. _howto/sensor:GCSObjectsWithPrefixExistenceSensor:

GCSObjectsWithPrefixExistenceSensor
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ To create a Google Vertex AI Auto ML training jobs you have five operators
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLForecastingTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLImageTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTabularTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTextTrainingJobOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.SupervisedFineTuningTrainOperator`
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLVideoTrainingJobOperator`
Each of them will wait for the operation to complete. The results of each operator will be a model
which was trained by user using these operators.
Expand Down Expand Up @@ -257,17 +257,6 @@ put dataset id to ``dataset_id`` parameter in operator.
:start-after: [START how_to_cloud_vertex_ai_create_auto_ml_tabular_training_job_operator]
:end-before: [END how_to_cloud_vertex_ai_create_auto_ml_tabular_training_job_operator]

How to run AutoML Text Training Job
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLTextTrainingJobOperator`

Operator is deprecated, the non-training (existing legacy models) AutoMLText API will be deprecated
on June 15, 2025.
There are 2 options for text classification, extraction, and sentiment analysis tasks replacement:
- Prompts with pre-trained Gemini model, using :class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.TextGenerationModelPredictOperator`.
- Fine tuning over Gemini model, For more tailored results, using :class:`~airflow.providers.google.cloud.operators.vertex_ai.generative_model.SupervisedFineTuningTrainOperator`.

Please visit the https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini-tuning for more details.

How to run AutoML Video Training Job
:class:`~airflow.providers.google.cloud.operators.vertex_ai.auto_ml.CreateAutoMLVideoTrainingJobOperator`

Expand Down

This file was deleted.

32 changes: 20 additions & 12 deletions docs/apache-airflow/howto/define-extra-link.rst
Original file line number Diff line number Diff line change
Expand Up @@ -138,24 +138,34 @@ Console, but if we wanted to change that link we could:
BIGQUERY_JOB_DETAILS_LINK_FMT = "http://console.cloud.google.com/bigquery?j={job_id}"
class BigQueryConsoleLink(BaseOperatorLink):
class BigQueryDatasetLink(BaseGoogleLink):
"""
Helper class for constructing BigQuery link.
Helper class for constructing BigQuery Dataset Link.
"""
name = "BigQuery Console"
operators = [BigQueryOperator]
name = "BigQuery Dataset"
key = "bigquery_dataset"
format_str = BIGQUERY_DATASET_LINK
def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey):
job_id = XCom.get_one(ti_key=ti_key, key="job_id")
return BIGQUERY_JOB_DETAILS_LINK_FMT.format(job_id=job_id) if job_id else ""
@staticmethod
def persist(
context: Context,
task_instance: BaseOperator,
dataset_id: str,
project_id: str,
):
task_instance.xcom_push(
context,
key=BigQueryDatasetLink.key,
value={"dataset_id": dataset_id, "project_id": project_id},
)
# Defining the plugin class
class AirflowExtraLinkPlugin(AirflowPlugin):
name = "extra_link_plugin"
operator_extra_links = [
BigQueryConsoleLink(),
BigQueryDatasetLink(),
]
Expand All @@ -171,9 +181,7 @@ by ``apache-airflow-providers-google`` provider currently:
.. code-block:: yaml
extra-links:
- airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleLink
- airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleIndexableLink
- airflow.providers.google.cloud.operators.mlengine.AIPlatformConsoleLink
- airflow.providers.google.cloud.links.bigquery.BigQueryDatasetLink
- airflow.providers.google.cloud.links.bigquery.BigQueryTableLink
You can include as many operators with extra links as you want.
Loading

0 comments on commit 7c1c4c6

Please sign in to comment.