feat: allow configuring of object storage files deletion for warehouse destinations

[shekhar-rudder]

What are the changes introduced in this PR?

A new boolean field is being added in this PR for all warehouse destinations which will be rendered only if user has selected their own object storage.

What is the related Linear task?

Resolves INT-XXX

Please explain the objectives of your changes below

Put down any required details on the broader aspect of your changes. If there are any dependent changes, mandatorily mention them here

Any changes to existing capabilities/behaviour, mention the reason & what are the changes ?

N/A

Any new dependencies introduced with this change?

N/A

Any new checks got introduced or modified in test suites. Please explain the changes.

N/A

---

Developer checklist

• My code follows the style guidelines of this project

• No breaking changes are being introduced.

• All related docs linked with the PR?

• All changes manually tested?

• Any documentation changes needed with this change?

• I have executed schemaGenerator tests and updated schema if needed

• Are sensitive fields marked as secret in definition config?

• My test cases and placeholders use only masked/sample values for sensitive fields

• Is the PR limited to 10 file changes & one task?

Reviewer checklist

• Is the type of change in the PR title appropriate as per the changes?

• Verified that there are no credentials or confidential data exposed with the changes.

Summary by CodeRabbit

• New Features
  • Introduced a new configuration option to enable cleanup of object storage files after successful sync across multiple destinations.
  • Added a toggle (checkbox) in the storage settings for each destination that lets users opt in for file cleanup (default set to off).
  • Updated the validation schema for various destinations to support the new cleanup setting, ensuring seamless configuration management.

[coderabbitai]

Walkthrough

The changes introduce a new configuration option for cleaning up object storage files across multiple destinations. A new key, "cleanupObjectStorageFiles", is added in the default configuration files (db-config.json), the JSON schema (schema.json) with its boolean definition and default value, and the UI configuration (ui-config.json) as a checkbox field conditioned on "useRudderStorage" being false. These modifications are consistent across various destination configurations, enhancing their management capabilities without altering existing structures.

Changes

File(s)	Change Summary
src/.../rs/db-config.json, src/.../rs/schema.json, src/.../rs/ui-config.json	Added the "cleanupObjectStorageFiles" option: a new key in the default configuration, a boolean property (default false) in the schema, and a conditional checkbox in the UI configuration.
src/.../azure_datalake/db-config.json, src/.../azure_datalake/schema.json, src/.../azure_datalake/ui-config.json	Introduced the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for object storage cleanup management.
src/.../azure_synapse/db-config.json, src/.../azure_synapse/schema.json, src/.../azure_synapse/ui-config.json	Added the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration, all conditioned on "useRudderStorage" being false.
src/.../bq/db-config.json, src/.../bq/schema.json, src/.../bq/ui-config.json	Included the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for managing object storage file cleanup.
src/.../clickhouse/db-config.json, src/.../clickhouse/schema.json, src/.../clickhouse/ui-config.json	Added the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for object storage file cleanup management.
src/.../deltalake/db-config.json, src/.../deltalake/schema.json, src/.../deltalake/ui-config.json	Introduced the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for enabling cleanup of object storage files.
src/.../gcs_datalake/db-config.json, src/.../gcs_datalake/schema.json, src/.../gcs_datalake/ui-config.json	Added the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for managing object storage file cleanup.
src/.../mssql/db-config.json, src/.../mssql/schema.json, src/.../mssql/ui-config.json	Included the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for cleanup of object storage files after successful sync.
src/.../postgres/db-config.json, src/.../postgres/schema.json, src/.../postgres/ui-config.json	Added the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for managing object storage file cleanup.
src/.../s3_datalake/db-config.json, src/.../s3_datalake/schema.json, src/.../s3_datalake/ui-config.json	Introduced the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for managing the cleanup of object storage files.
src/.../snowflake/db-config.json, src/.../snowflake/schema.json, src/.../snowflake/ui-config.json	Added the "cleanupObjectStorageFiles" key in the default config, a boolean property in the schema, and a checkbox in the UI configuration for enabling cleanup of object storage files after successful sync.

Suggested reviewers

• lvrach
• am6010
• krishna2020
• ssbeefeater
• debanjan97
• cisse21
• 1abhishekpandey
• ItsSudip
• sandeepdsvs
• RanjeetMishra
• koladilip

Poem

I'm a rabbit with a keen coder's gaze,
Hopping through JSON in delightful arrays,
New cleanup keys make the configs sing,
Schema and UI now sparkle like spring,
With every commit my ears stand tall—
Code leaps and bounds, a hoppy ball! 🐰

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3c035fc and f71761e.

📒 Files selected for processing (30)

• src/configurations/destinations/azure_datalake/db-config.json (1 hunks)
• src/configurations/destinations/azure_datalake/schema.json (1 hunks)
• src/configurations/destinations/azure_datalake/ui-config.json (1 hunks)
• src/configurations/destinations/azure_synapse/db-config.json (1 hunks)
• src/configurations/destinations/azure_synapse/schema.json (1 hunks)
• src/configurations/destinations/azure_synapse/ui-config.json (1 hunks)
• src/configurations/destinations/bq/db-config.json (1 hunks)
• src/configurations/destinations/bq/schema.json (1 hunks)
• src/configurations/destinations/bq/ui-config.json (1 hunks)
• src/configurations/destinations/clickhouse/db-config.json (1 hunks)
• src/configurations/destinations/clickhouse/schema.json (1 hunks)
• src/configurations/destinations/clickhouse/ui-config.json (1 hunks)
• src/configurations/destinations/deltalake/db-config.json (1 hunks)
• src/configurations/destinations/deltalake/schema.json (1 hunks)
• src/configurations/destinations/deltalake/ui-config.json (1 hunks)
• src/configurations/destinations/gcs_datalake/db-config.json (1 hunks)
• src/configurations/destinations/gcs_datalake/schema.json (1 hunks)
• src/configurations/destinations/gcs_datalake/ui-config.json (1 hunks)
• src/configurations/destinations/mssql/db-config.json (1 hunks)
• src/configurations/destinations/mssql/schema.json (1 hunks)
• src/configurations/destinations/mssql/ui-config.json (1 hunks)
• src/configurations/destinations/postgres/db-config.json (1 hunks)
• src/configurations/destinations/postgres/schema.json (1 hunks)
• src/configurations/destinations/postgres/ui-config.json (1 hunks)
• src/configurations/destinations/s3_datalake/db-config.json (1 hunks)
• src/configurations/destinations/s3_datalake/schema.json (1 hunks)
• src/configurations/destinations/s3_datalake/ui-config.json (1 hunks)
• src/configurations/destinations/snowflake/db-config.json (1 hunks)
• src/configurations/destinations/snowflake/schema.json (1 hunks)
• src/configurations/destinations/snowflake/ui-config.json (1 hunks)

🔇 Additional comments (30)

src/configurations/destinations/gcs_datalake/db-config.json (1)

49-49: New Key Addition in Default Configuration.
The "cleanupObjectStorageFiles" key has been added to the defaultConfig array. This update aligns with the PR objective to allow configurable deletion of object storage files. Please ensure that the corresponding schema and UI configurations reflect the correct type (boolean) and default value.

src/configurations/destinations/azure_datalake/db-config.json (1)

37-37: New Config Key in Azure Datalake.
The addition of "cleanupObjectStorageFiles" to the defaultConfig array is clear and consistent with the other destination configurations. Verify that its integration with the JSON schema and UI (which conditions display on the user’s storage selection) is correctly implemented.

src/configurations/destinations/bq/db-config.json (1)

40-41: Inclusion of Cleanup Option in BigQuery Config.
The new key "cleanupObjectStorageFiles" has been inserted into the defaultConfig array (between "partitionColumn" and "partitionType"). This change is consistent with the overall update across destinations. Please confirm that related schema and UI changes reflect the intended default value and behavior.

src/configurations/destinations/s3_datalake/db-config.json (1)

55-55: S3 Datalake: New Cleanup Key Added.
The "cleanupObjectStorageFiles" key is now part of the default configuration. Ensure that its corresponding schema validation (as a boolean with a default of false) and UI checkbox presentation (conditional on the storage selection) are implemented correctly.

src/configurations/destinations/mssql/db-config.json (1)

68-68: Microsoft SQL Server: Addition of Cleanup Key.
The new key "cleanupObjectStorageFiles" has been added in the defaultConfig array. This update extends the configurable options consistently across the warehouse destinations. Double-check that the related UI and schema files are updated to conditionally display and validate this field as specified.

src/configurations/destinations/azure_synapse/db-config.json (1)

55-56: New Configuration Key Added ("cleanupObjectStorageFiles")

The new key "cleanupObjectStorageFiles" has been added in the defaultConfig array. This addition aligns with the PR objective to allow users to optionally configure cleanup of object storage files when they manage their own object storage. Please verify that its usage is consistent with the corresponding schema and UI configurations.

src/configurations/destinations/snowflake/db-config.json (1)

70-72: Consistent Addition of "cleanupObjectStorageFiles"

The Snowflake configuration now includes "cleanupObjectStorageFiles" in its defaultConfig array. This is consistently added after "underscoreDivideNumbers" and before "allowUsersContextTraits". Ensure that the default value (false) is maintained in the schema file so that the behavior remains predictable.

src/configurations/destinations/deltalake/db-config.json (1)

55-57: Incorporation of the Cleanup Option

"cleanupObjectStorageFiles" is now part of the Delta Lake defaultConfig. This addition, consistent with other warehouse configurations, enables the new cleanup functionality. Please double-check that the related schema defines this option as a boolean with a default value of false.

src/configurations/destinations/clickhouse/db-config.json (1)

58-60: Addition of "cleanupObjectStorageFiles" for ClickHouse

The ClickHouse configuration file now includes the new "cleanupObjectStorageFiles" key in its defaultConfig list. The ordering is consistent with similar destinations, ensuring uniformity across configurations. Verify that any dependent UI or schema changes are in sync.

src/configurations/destinations/postgres/db-config.json (1)

76-78: Postgres Configuration Updated with "cleanupObjectStorageFiles"

The Postgres configuration has been updated to include "cleanupObjectStorageFiles" in the defaultConfig array. This change expands the configurable options for object storage management. Confirm that the default value is appropriately set in the schema and that this option will be conditionally displayed in the UI when "useRudderStorage" is false.

src/configurations/destinations/azure_datalake/ui-config.json (1)

87-92: New Cleanup Checkbox Field Added
A new checkbox field ("cleanupObjectStorageFiles") has been added with a default value of false. This addition fulfills the requirement to allow users to enable cleanup of object storage files after a successful sync.
Consider verifying whether this field should be conditionally displayed—e.g. only when the user opts for their own object storage—as mentioned in the PR objectives.

src/configurations/destinations/gcs_datalake/ui-config.json (1)

49-54: New Cleanup Checkbox Field in GCS Configuration
The new "cleanupObjectStorageFiles" checkbox field is introduced in the "Connection Credentials" section with a default value set to false.
This change is consistent with similar modifications in other destination configurations.

src/configurations/destinations/s3_datalake/ui-config.json (1)

114-123: Conditional Display of Cleanup Option in S3 Configuration
The new cleanup field is implemented with a preRequisiteField ensuring it is only displayed when "useRudderStorage" is false. This conditional logic aligns with the PR objective that this feature should only be available when users have selected their own object storage.

src/configurations/destinations/bq/ui-config.json (1)

70-75: New Cleanup Checkbox Field in BigQuery Configuration
A new checkbox ("cleanupObjectStorageFiles") has been added with the default state set to false.
Please verify whether a conditional display is intended here—similar to S3 and Delta Lake—so that the field is only shown when custom object storage is used, as suggested by the PR objectives.

src/configurations/destinations/deltalake/ui-config.json (1)

529-539: Conditional Cleanup Checkbox Field for Delta Lake
The new checkbox field for cleaning up object storage files is added with a preRequisiteField making it available only when "useRudderStorage" is false. This implementation meets the intended requirement.

src/configurations/destinations/azure_synapse/ui-config.json (1)

554-563: New Cleanup Field Addition in Azure Synapse UI Config

The new checkbox field "Enable for cleanup of object storage files (deletion) after successful sync" is added with a default value of false. The use of "preRequisiteField" (requiring "useRudderStorage" to be false) correctly ensures that this option is only displayed for users configuring their own object storage. This change aligns nicely with the PR objectives.

src/configurations/destinations/clickhouse/ui-config.json (1)

562-570: Consistent Cleanup Field for ClickHouse Destination

The newly introduced "cleanupObjectStorageFiles" field is implemented as a checkbox with a default of false. The "preRequisiteField" condition (checking that "useRudderStorage" is false) guarantees that this option is only visible when appropriate. This implementation is consistent with similar changes across the project.

src/configurations/destinations/mssql/ui-config.json (1)

554-562: Cleanup Field Addition in MSSQL UI Config

The checkbox "Enable for cleanup of object storage files (deletion) after successful sync" is properly integrated into the MSSQL UI configuration. With a default value of false and the "preRequisiteField" enforcing that it is only shown when "useRudderStorage" is false, the change meets the intended design goal.

src/configurations/destinations/snowflake/ui-config.json (1)

577-586: New Cleanup Option for Snowflake

The addition of the "cleanupObjectStorageFiles" checkbox in the Snowflake UI configuration is clear and consistent. It is set to false by default and features a prerequisite condition on "useRudderStorage" being false, ensuring that the cleanup option is only available when the user opts out of RudderStack-managed storage.

src/configurations/destinations/postgres/ui-config.json (1)

646-654: Postgres UI Cleanup Field Update

The new "cleanupObjectStorageFiles" field has been added as a checkbox with a default value of false. Its "preRequisiteField" correctly specifies that this option should only appear when "useRudderStorage" is false. This implementation is consistent with the modifications made in other destination configurations.

src/configurations/destinations/gcs_datalake/schema.json (1)

27-30: New Property Addition – cleanupObjectStorageFiles
The new property is correctly added as a boolean with a default value of false, which is consistent with the PR objectives. Please ensure that any associated UI or configuration logic that depends on this property is updated elsewhere.

src/configurations/destinations/bq/schema.json (1)

69-72: Consistent Schema Enhancement – cleanupObjectStorageFiles
This addition correctly extends the schema with a new boolean property (default false) for controlling object storage file cleanup. It aligns with similar changes in other destination configurations.

src/configurations/destinations/azure_datalake/schema.json (1)

43-46: Proper Integration of cleanupObjectStorageFiles
The new property is seamlessly integrated into the schema as a boolean with its default set to false. This is in line with the overall enhancement for storage file management.

src/configurations/destinations/s3_datalake/schema.json (1)

47-50: Addition of cleanupObjectStorageFiles for S3 Datalake
The property is added correctly, maintaining the same naming convention and default behavior as in the other destinations. Ensure that any corresponding updates in the UI configuration (checkbox field) remain in sync with this change.

src/configurations/destinations/azure_synapse/schema.json (1)

930-935: Conditional Schema Update – cleanupObjectStorageFiles in Azure Synapse
Within the conditional block (when useRudderStorage is false), the new property is added properly with the expected type and default value. Please verify that including this property does not interfere with the "required" condition on "bucketProvider" and that downstream configuration logic correctly handles it.

src/configurations/destinations/deltalake/schema.json (1)

925-930: New Cleanup Field Added Correctly.
The "cleanupObjectStorageFiles" property has been integrated in the "then" clause under the condition that "useRudderStorage" is false. The type and default value (false) are correctly defined, and the surrounding conditional logic (preserving "bucketProvider" as required) is consistent with our design.

src/configurations/destinations/clickhouse/schema.json (1)

967-974: Cleanup Property Integration in ClickHouse Schema.
The new "cleanupObjectStorageFiles" property is added conditionally when "useRudderStorage" is false with a default value of false. This matches the pattern used in other destination schemas and preserves the existing requirement for "bucketProvider".

src/configurations/destinations/snowflake/schema.json (1)

34-41: Snowflake Schema: New Cleanup Option Added.
The "cleanupObjectStorageFiles" property is now provided when "useRudderStorage" is false. The property is defined as a boolean with a default of false, which is in line with similar changes across other destinations. Verify that the "required" clause (listing "bucketProvider") is appropriate for Snowflake configurations.

src/configurations/destinations/postgres/schema.json (1)

969-974: Postgres Schema: Cleanup Field Addition.
The new "cleanupObjectStorageFiles" field has been added under the condition that "useRudderStorage" is false. Its type and default value (false) are correctly declared, and the conditional logic (with "bucketProvider" remaining required) aligns with the approach taken in the other destination schemas.

src/configurations/destinations/mssql/schema.json (1)

923-927: New boolean field “cleanupObjectStorageFiles” is correctly defined.
The property is added conditionally under the “then” block when “useRudderStorage” is false and is defined as a boolean with a default value of false. This aligns with the PR objective to display this field only when the user is configuring their own object storage. Please ensure that the UI and any downstream configuration consumers correctly interpret and display/hide this option as intended.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (dd6bb8b) to head (f71761e).
Report is 4 commits behind head on develop.

Additional details and impacted files

@@            Coverage Diff            @@
##           develop     #1896   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            2         2           
  Lines           53        53           
  Branches         7         7           
=========================================
  Hits            53        53           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.