Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Prevent truncated Parquet files in S3 after failed CreateMultipartUpload (2987) #2993

Merged
merged 2 commits into from
Oct 10, 2024

Commits on Oct 8, 2024

  1. fix: Prevent truncated Parquet files in S3 after failed CreateMultipa…

    …rtUpload
    
    During a call to s3.to_parquet(), if the size of the data exceeds 5MB a multi-part upload
    operation will be initiated.
    If the S3 call to CreateMultipartUpload fails (such as with a 503 SlowDown error) then
    the incomplete Parquet file data was being written to S3 using 'put_object' during close().
    This resulted in broken Parquet files in S3, causing errors when queried by services like Athena.
    
    Now, the data buffer is cleared at the end of the call to flush() -- even when an exception occurs.
    rdwebster authored Oct 8, 2024
    Configuration menu
    Copy the full SHA
    f69e058 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Configuration menu
    Copy the full SHA
    51def0d View commit details
    Browse the repository at this point in the history