-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add GCP Financial Services API #43446
base: main
Are you sure you want to change the base?
Changes from 18 commits
286e38c
8673f7f
e2a0196
44cd7a6
3477b25
3f5abef
ed45f03
2994681
3ea9891
8eb9a8a
18215a6
fb6adda
bdf7140
5027afe
6b9fe2e
a874878
9b62db0
9905ecf
223eaf2
188de90
e8f5339
02ce717
89fa2e7
48ee457
628b1a5
1395315
892db75
1058c09
fb67fd9
e43f6e5
cc783fe
984fa74
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
.. Licensed to the Apache Software Foundation (ASF) under one | ||
or more contributor license agreements. See the NOTICE file | ||
distributed with this work for additional information | ||
regarding copyright ownership. The ASF licenses this file | ||
to you under the Apache License, Version 2.0 (the | ||
"License"); you may not use this file except in compliance | ||
with the License. You may obtain a copy of the License at | ||
|
||
.. http://www.apache.org/licenses/LICENSE-2.0 | ||
|
||
.. Unless required by applicable law or agreed to in writing, | ||
software distributed under the License is distributed on an | ||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
KIND, either express or implied. See the License for the | ||
specific language governing permissions and limitations | ||
under the License. | ||
|
||
|
||
|
||
Google Cloud Financial Services Operators | ||
========================================= | ||
|
||
Prerequisite Tasks | ||
^^^^^^^^^^^^^^^^^^ | ||
|
||
.. include:: /operators/_partials/prerequisite_tasks.rst | ||
|
||
Manage instances | ||
^^^^^^^^^^^^^^^^ | ||
|
||
.. _howto/operator:FinancialServicesCreateInstanceOperator: | ||
|
||
Create an AML AI instance | ||
------------------------- | ||
|
||
Use the :class:`~airflow.providers.google.cloud.operators.financial_services.FinancialServicesCreateInstanceOperator` | ||
operator to create an AML AI instance in the Google Cloud Financial Services API. | ||
|
||
Using the operator | ||
"""""""""""""""""" | ||
|
||
.. exampleinclude:: /../../providers/tests/system/google/cloud/financial_services/example_financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_operator_financial_services_create_instance] | ||
:end-before: [END howto_operator_financial_services_create_instance] | ||
|
||
Templating | ||
"""""""""" | ||
|
||
.. literalinclude:: /../../providers/src/airflow/providers/google/cloud/operators/financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_operator_financial_services_create_instance_template_fields] | ||
:end-before: [END howto_operator_financial_services_create_instance_template_fields] | ||
|
||
More information | ||
"""""""""""""""" | ||
|
||
See Google Cloud Financial Services API documentation to `create an instance | ||
<https://cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest/v1/projects.locations.instances/create>`_. | ||
|
||
.. _howto/operator:FinancialServicesCreateInstanceOperator: | ||
|
||
|
||
.. _howto/operator:FinancialServicesGetInstanceOperator: | ||
|
||
Get an AML AI instance | ||
------------------------- | ||
|
||
Use the :class:`~airflow.providers.google.cloud.operators.financial_services.FinancialServicesGetInstanceOperator` | ||
operator to get an AML AI instance in the Google Cloud Financial Services API. | ||
|
||
Using the operator | ||
"""""""""""""""""" | ||
|
||
.. exampleinclude:: /../../providers/tests/system/google/cloud/financial_services/example_financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_operator_financial_services_get_instance] | ||
:end-before: [END howto_operator_financial_services_get_instance] | ||
|
||
Templating | ||
"""""""""" | ||
|
||
.. literalinclude:: /../../providers/src/airflow/providers/google/cloud/operators/financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_operator_financial_services_get_instance_template_fields] | ||
:end-before: [END howto_operator_financial_services_get_instance_template_fields] | ||
|
||
More information | ||
"""""""""""""""" | ||
|
||
See Google Cloud Financial Services API documentation to `get an instance | ||
<https://cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest/v1/projects.locations.instances/get>`_. | ||
|
||
.. _howto/operator:FinancialServicesGetInstanceOperator: | ||
|
||
|
||
.. _howto/operator:FinancialServicesDeleteInstanceOperator: | ||
|
||
Delete an AML AI instance | ||
------------------------- | ||
|
||
Use the :class:`~airflow.providers.google.cloud.operators.financial_services.FinancialServicesDeleteInstanceOperator` | ||
operator to delete an AML AI instance in the Google Cloud Financial Services API. | ||
|
||
Using the operator | ||
"""""""""""""""""" | ||
|
||
.. exampleinclude:: /../../providers/tests/system/google/cloud/financial_services/example_financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_operator_financial_services_delete_instance] | ||
:end-before: [END howto_operator_financial_services_delete_instance] | ||
|
||
Templating | ||
"""""""""" | ||
|
||
.. literalinclude:: /../../providers/src/airflow/providers/google/cloud/operators/financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_operator_financial_services_delete_instance_template_fields] | ||
:end-before: [END howto_operator_financial_services_delete_instance_template_fields] | ||
|
||
More information | ||
"""""""""""""""" | ||
|
||
See Google Cloud Financial Services API documentation to `delete an instance | ||
<https://cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest/v1/projects.locations.instances/delete>`_. | ||
|
||
.. _howto/operator:FinancialServicesDeleteInstanceOperator: | ||
|
||
|
||
Sensors | ||
^^^^^^^ | ||
|
||
.. _howto/sensor:FinancialServicesOperationSensor: | ||
|
||
Check operation status | ||
---------------------- | ||
|
||
Use the :class:`~airflow.providers.google.cloud.sensors.financial_services.FinancialServicesOperationSensor` | ||
sensor to check the status of an operation in the Google Cloud Financial Services API. | ||
|
||
Using the sensor | ||
"""""""""""""""""" | ||
|
||
.. exampleinclude:: /../../providers/tests/system/google/cloud/financial_services/example_financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_sensor_financial_services_operation] | ||
:end-before: [END howto_sensor_financial_services_operation] | ||
|
||
Templating | ||
"""""""""" | ||
|
||
.. literalinclude:: /../../providers/src/airflow/providers/google/cloud/operators/financial_services.py | ||
:language: python | ||
:dedent: 4 | ||
:start-after: [START howto_sensor_financial_services_operation_template_fields] | ||
:end-before: [END howto_sensor_financial_services_operation_template_fields] | ||
|
||
More information | ||
"""""""""""""""" | ||
|
||
See Google Cloud Financial Services API documentation to `get an operation | ||
<https://cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest/v1/projects.locations.operations/get>`_. | ||
|
||
.. _howto/sensor:FinancialServicesOperationSensor: |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,128 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
from __future__ import annotations | ||
|
||
from googleapiclient.discovery import Resource, build_from_document | ||
|
||
from airflow.providers.google.common.hooks.base_google import GoogleBaseHook | ||
|
||
|
||
class FinancialServicesHook(GoogleBaseHook): | ||
""" | ||
Hook for interacting with the Google Financial Services API. | ||
|
||
:param discovery_doc: Discovery document for building the Financial Services API | ||
as described `here <https://cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest#discovery-document>`__ | ||
:param gcp_conn_id: Identifier of connection to Google Cloud Platform. | ||
Defaults to "google_cloud_default". | ||
""" | ||
|
||
connection: Resource | None = None | ||
|
||
def __init__( | ||
self, | ||
discovery_doc: dict, | ||
gcp_conn_id: str = "google_cloud_default", | ||
**kwargs, | ||
) -> None: | ||
if kwargs.get("delegate_to") is not None: | ||
raise RuntimeError( | ||
"The `delegate_to` parameter has been deprecated before and finally" | ||
" removed in this version" | ||
" of Google Provider. You MUST convert it to `impersonate_chain`" | ||
) | ||
super().__init__( | ||
gcp_conn_id=gcp_conn_id, | ||
impersonation_chain=None, | ||
) | ||
self.discovery_doc = discovery_doc | ||
|
||
def get_conn(self) -> Resource: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we try to download a discovery document for an authorized user? Firstly authorize the user, then download a discovery doc and after that build a connection. Similar to this code for firebase There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The credentials retrieved using I have made updates to enable users to supply either a developer API key OR a discovery document as a local .json file by setting Airflow variables. An exception will be raised if neither of these variables are set. |
||
""" | ||
Establish a connection to the Google Financial Services API. | ||
|
||
:return: A Google Cloud Financial Services API service resource. | ||
""" | ||
if not self.connection: | ||
self.connection = build_from_document(self.discovery_doc) | ||
|
||
return self.connection | ||
|
||
def get_instance(self, instance_resource_uri: str) -> dict: | ||
""" | ||
Get a Financial Services Anti-Money Laundering AI instance. | ||
|
||
:param instance_resource_uri: URI of the instance to get (format: | ||
'projects/<Project ID>/locations/<Location>/instances/<Instance ID>) | ||
|
||
:returns: A dictionary containing the instance metadata | ||
""" | ||
conn = self.get_conn() | ||
response = conn.projects().locations().instances().get(name=instance_resource_uri).execute() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For all our methods in hooks we divide There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for your comment. I have made the updates to hook and operator code accordingly. |
||
return response | ||
|
||
def create_instance(self, instance_id: str, kms_key_uri: str, location_resource_uri: str) -> dict: | ||
""" | ||
Create a Financial Services Anti-Money Laundering AI instance. | ||
|
||
:param instance_id: Identifier for the instance to create | ||
:param kms_key_uri: URI of the KMS key to that will be used for instance encryption | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The same as mentioned above for |
||
(format: 'projects/<Project ID>/locations/<Location>/keyRings/<Key Ring>/ | ||
cryptoKeys/<Key>') | ||
|
||
:returns: A dictionary containing metadata for the create instance operation | ||
""" | ||
conn = self.get_conn() | ||
response = ( | ||
conn.projects() | ||
.locations() | ||
.instances() | ||
.create( | ||
parent=location_resource_uri, | ||
instanceId=instance_id, | ||
body={"kmsKey": kms_key_uri}, | ||
) | ||
.execute() | ||
) | ||
return response | ||
|
||
def delete_instance(self, instance_resource_uri: str) -> dict: | ||
""" | ||
Delete a Financial Services Anti-Money Laundering AI instance. | ||
|
||
:param instance_resource_uri: URI of the instance to delete (format: | ||
'projects/<Project ID>/locations/<Location>/instances/<Instance ID>) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The same |
||
|
||
:returns: A dictionary containing metadata for the delete instance | ||
operation | ||
""" | ||
conn = self.get_conn() | ||
response = conn.projects().locations().instances().delete(name=instance_resource_uri).execute() | ||
return response | ||
|
||
def get_operation(self, operation_resource_uri: str) -> dict: | ||
""" | ||
Get a Financial Services Anti-Money Laundering AI operation. | ||
|
||
:param operation_resource_uri: URI of the operation to get (format: | ||
'projects/<Project ID>/locations/<Location>/operations/<Operation ID>) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The same |
||
|
||
:return: A dictionary containing metadata for the operation | ||
""" | ||
conn = self.get_conn() | ||
response = conn.projects().locations().operations().get(name=operation_resource_uri).execute() | ||
return response |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code is not needed, because you created a new Hook. And this if-clause was added for deprecate
delegate_to
parameter in old hooks.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your comment. I've removed this part from the code.