Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Oracle to airflow->cosmos profile map #1190

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Conversation

slords
Copy link

@slords slords commented Sep 4, 2024

Description

Add the ability to map Oracle connections from airflow to comsos

Related Issue(s)

closes #1189

Breaking Change?

None that I'm aware of.

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Sep 4, 2024
Copy link

netlify bot commented Sep 4, 2024

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit f174678
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66f599e60c3f0800083b312b

@dosubot dosubot bot added the area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc label Sep 4, 2024
Copy link

codecov bot commented Sep 19, 2024

Codecov Report

Attention: Patch coverage is 86.44068% with 16 lines in your changes missing coverage. Please review.

Project coverage is 95.43%. Comparing base (3414513) to head (f174678).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
cosmos/profiles/oracle/user_pass.py 62.50% 15 Missing ⚠️
cosmos/operators/gcp_cloud_run_job.py 98.36% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1190      +/-   ##
==========================================
- Coverage   95.72%   95.43%   -0.30%     
==========================================
  Files          64       67       +3     
  Lines        3672     3788     +116     
==========================================
+ Hits         3515     3615     +100     
- Misses        157      173      +16     
Flag Coverage Δ
95.43% <86.44%> (-0.30%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

believe it's worth adding the missing tests to avoid regression. Otherwise, it looks great!

@slords
Copy link
Author

slords commented Sep 23, 2024

Working on that. Should have something in the next few days.

@tatiana
Copy link
Collaborator

tatiana commented Sep 24, 2024

Hi @slords, thanks a lot for working on this feature!
It feels like we're very close to getting to the finishing line.
If you can cover tests for the missing branches this week, we'll be able to release this with Cosmos 1.7 on the 30th of September..! Otherwise, it will be released in the following Cosmos minor release.

cmarteepants and others added 5 commits September 26, 2024 11:28
…#1221)

This adds a privacy notice and website analytics to the Cosmos readme
and auto-generated docs.

Note that while you cannot explicitly opt out of website analytics for
the publicly hosted readme (and docs), Scarf respects browser DND. If
that is set via the browser, telemetry for that user will not be sent to
Scarf.

Scarf privacy policy: https://about.scarf.sh/privacy-policy
Astronomer privacy policy: https://www.astronomer.io/privacy/
Added new `GCP_CLOUD_RUN_JOB` execution mode that triggers Google Cloud
Platform's Cloud Run Job instance with dbt model in it.

It extends Airflow's `CloudRunExecuteJobOperator` and overrides Cloud
Run Job's container with dbt command generated by cosmos.

Note: `CloudRunExecuteJobOperator` has `container_overrides` parameter
implemented in `apache-airflow-providers-google==10.13.0` which is
supported by `airflow >=2.6.0`.

Resolves astronomer#1149 

Co-authored-by: Agata Zalewska <[email protected]>
Co-authored-by: Tatiana Al-Chueyr <[email protected]>
…1198)

In projects containing models with names like the following, dataset
creation fails, and an error occurs during execution.
```txt
└── dbt
    └── my_project
        └── models
            ├── 日本語名モデル.sql
            └── 日本語名モデル.yml

```

```
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/datasets/__init__.py", line 78, in _sanitize_uri
    raise ValueError("Dataset URI must only consist of ASCII characters")
ValueError: Dataset URI must only consist of ASCII characters
```

To support model names with multibyte characters, it might be good to
URL encode the names.

closes: astronomer#1197

Co-authored-by: Tatiana Al-Chueyr <[email protected]>
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Sep 26, 2024
@tatiana tatiana modified the milestones: Cosmos 1.7.0, Cosmos 1.8.0 Oct 7, 2024
@tatiana
Copy link
Collaborator

tatiana commented Oct 29, 2024

Hey @slords, we're really close to completing this feature - it would be great to release it as part of Cosmos 1.8. Please, do let us know if you'd like any additional support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Add oracle to profile mapping
7 participants