Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Glue Data catalog csv files with header parameter cause error GENERIC_INTERNAL_ERROR: Failed to construct table metadata for table xxxx #2967

Closed
ctalucci opened this issue Sep 19, 2024 · 1 comment · Fixed by #2977
Assignees
Labels
bug Something isn't working

Comments

@ctalucci
Copy link

When using wr.s3.to_csv with glue catalog and the header parameter the corresponding table schema parameter skip.header.line.count is getting set to True which causes an GENERIC_INTERNAL_ERROR: Failed to construct table metadata for table xxxx on querying with Athena. Manually updating the value to 1 fixes the error

https://github.com/aws/aws-sdk-pandas/blob/8cef7504c4b4aadd44306cde49001fcd062c5bb7/awswrangler/s3/_write_text.py#L585C1-L585C68

"skip_header_line_count": True if header else None,
should be:
"skip_header_line_count": 1 if header else None,

@jaidisido jaidisido self-assigned this Sep 30, 2024
@jaidisido jaidisido added bug Something isn't working and removed needs-triage labels Sep 30, 2024
@jaidisido
Copy link
Contributor

Thanks, as far as I can tell from the AWS Glue docs, skip.header.line.count should indeed be an integer

@jaidisido jaidisido linked a pull request Sep 30, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants