Add support for loading manifest from cloud store using Airflow Object Store

[pankajkoti]

Summary

This PR introduces the capability to load manifest.json files from various cloud storage services using Airflow's Object Store integration. The supported cloud storages include AWS S3, Google Cloud Storage (GCS), and Azure Blob Storage. The feature allows seamless integration with remote paths, providing enhanced flexibility and scalability for managing DBT projects.

Key Changes

1. Parameters in DbtDag and DbtTaskGroup:
   manifest_path: Accepts both local paths and remote URLs (e.g., S3, GCS, Azure Blob Storage).
   manifest_conn_id: (Optional) An Airflow connection ID for accessing the remote path.
2. Automatic Detection of Storage Type:
   The system automatically identifies the storage service based on the scheme of the URL provided (e.g., s3://, gs://, abfs://) by integrating with Airflow Object Store
3. If a manifest_conn_id is provided, it is used to fetch the necessary credentials.
4. If no manifest_conn_id is provided, the default connection ID for the identified scheme is used.

Validation and Error Handling

1. Validates the existence of the manifest.json file when a path is specified.
2. Raises appropriate errors if a remote manifest_path is given but the required min Airflow version 2.8(Object Store feature) support is not available.

Backward Compatibility

Ensures compatibility with existing workflows that use local paths for the manifest.json.

How to Use

1. Local Path:

DbtDag(
    project_config=ProjectConfig(
        manifest_path="/path/to/local/manifest.json",
    ),
    ...
)

2. Remote Path (e.g., S3):

DbtDag(
    project_config=ProjectConfig(
        manifest_path="s3://bucket/path/to/manifest.json",
        manifest_conn_id="aws_s3_conn",
    ),
    ...
)

3. Remote Path without Explicit Connection ID:

DbtDag(
    project_config=ProjectConfig(
        manifest_path="gs://bucket/path/to/manifest.json",
        # No manifest_conn_id provided, will use default Airflow GCS connection `google_cloud_default`
    ),
    ...
)

Additional Notes

1. Ensure that the required Airflow version (2.8 or later) is used to take advantage of the Object Store features.
2. Review the updated documentation for detailed usage instructions and examples.

Testing

1. Added unit tests to cover various scenarios including local paths, remote paths with and without manifest_conn_id.
2. Verified integration with different cloud storage services (AWS S3, GCS, Azure Blob Storage).
3. Ensured backward compatibility with existing local path workflows.

Related Issue(s)

closes: #448

Breaking Change?

No.

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	ecee4db
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66a0a330e04d37000869023c
😎 Deploy Preview	https://deploy-preview-1109--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.50%. Comparing base (236488b) to head (77022c4).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1109      +/-   ##
==========================================
+ Coverage   96.48%   96.50%   +0.02%     
==========================================
  Files          64       64              
  Lines        3301     3321      +20     
==========================================
+ Hits         3185     3205      +20     
  Misses        116      116              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

It looks great, @pankajkoti , great work!

I know this PR is still in draft, but some early feedback:

• docs: we need to update the docs (both on the ProjectConfig and also in the manifest parsing method, mentioning the minimum version of Cosmos/Airflow to support remote manifest locations
• dependencies: probably worth have extras specific per cloud provider, similar to Airflow, so we do not to introduce new extras if we add additional improvements per cloud provider

[tatiana]

This is a very waited feature, @pankajkoti , thanks a lot for addressing all the feedback! I left three minor comments, and we're good to merge.

[dwreeves]

Hi, just noticed this one testing some other code and this caused my code to fail.

I don't think it is appropriate for Cosmos to create explicit dependencies on AWS, GCP, and Azure providers, which is what this code does by eagerly loading the providers into the level. This will be a disruption to a lot of users, and also adds additional bloat to their environments that they genuinely don't need.

[dwreeves]

^ I've opened PR #1132 to address the issue raised in the above comment.