Sync Failure: Missing Rows When Syncing Large MySQL Table to Snowflake

[Khoulii]

Helm Chart Version

1.3.1

What step the error happened?

During the Sync

Relevant information

We set up a connection from a MySQL database to Snowflake using Airbyte. When testing the sync on a small table, all data transferred successfully. However, when attempting to sync a 38.27 GB table, we started seeing error logs indicating missing rows or rows that were not synced successfully.

We also attempted to sync the data using Change Data Capture (CDC) with binary logs, but encountered the same issue.

Environment Details:

• Airbyte Version: 1.4.1
• Installation Method: Installed using abctl
• Infrastructure:
  • RAM: 64GB
  • CPUs: 8
  • Disk Storage: 2TB
• Source: MySQL
• Destination: Snowflake

Steps to Reproduce:

1. Create a connection from MySQL to Snowflake.
2. Sync a small table (successful).
3. Sync a large table (38.27 GB).
4. Observe error logs indicating missing rows.
5. Attempt to use CDC (binary logs) to transfer data (same issue occurs).

Expected Behavior:
All rows from MySQL should be successfully transferred to Snowflake without data loss.

Actual Behavior:
Missing rows after sync.
Error logs indicating unsuccessful row sync.

Additional Context:
No network disruptions were observed during the sync.
We would appreciate any guidance on resolving this issue or debugging steps to identify the root cause.

Relevant log output

[marcosmarxm]

What are the connectors version? Are you customizing Helm to have more job resources?

[Khoulii]

@marcosmarxm we installed the latest version of Airbyte on production environment following the Quick Start document found here. Also, we didn't add any customization to Helm as we directly installed Airbyte using "abctl local install".
For the connectors version, we used MySQL airbyte connector version 3.11.1, and Snowflake airbyte connector 3.15.4.