Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Checksum not synced #168

Open
antareson opened this issue Feb 12, 2025 · 1 comment
Open

[BUG] Checksum not synced #168

antareson opened this issue Feb 12, 2025 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@antareson
Copy link

Describe the bug
The checksum meta data is not synced from the source, resulting in "Skipping checksum validation message by boto3.s3.download_file()"

To Reproduce

  1. Get s3 file synced (with Include Object Metadata = Yes)
  2. On S3 web console, verify the source has CRC32 checksum but target Checksum = "-"
  3. Use boto3 python pacakge to download the file and get warning

Expected behavior
Checksum data is synced with the S3 object

Please complete the following information about the solution:
(SO8002) - Data Transfer Hub - S3 Plugin - Template version v2.6.6
(SO8001) - Data Transfer Hub with aws-solutions-constructs: This template deploys an one-stop toolset for transferring data from different sources into AWS. Template version v2.6.6

  • [us-east-1] Region: [e.g. us-east-1]
  • [No] Was the solution modified from the version published on this repository?
  • [Yes] Have you checked your service quotas for the sevices this solution uses?
  • [No] Were there any errors in the CloudWatch Logs?

Screenshots
Job def
Image
Target Checksum
Image

@antareson antareson added the bug Something isn't working label Feb 12, 2025
@bassemwanis
Copy link
Contributor

bassemwanis commented Feb 13, 2025

Thank you @antareson for bringing this to our attention. We will investigate the issue. In the meantime, here is the logic that handles the S3 object upload https://github.com/aws-solutions/data-transfer-hub-cli/blob/d6fb4e09f8a0cac12d4dc85d8e1eeb2a4953ffb8/dth/client.go#L514. MD5 hashing validation is used, and Include Object Metadata? setting does not transfer the checksum value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants