Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using external JSON materalization: bumping into default maximum_object_size limit. #409

Open
firewall413 opened this issue Jun 26, 2024 · 1 comment

Comments

@firewall413
Copy link

create or replace view {{ intermediate_relation }} as (

When materializing a table to a JSON file bigger than 30MB, we bump into the following:

Invalid Input Error: "maximum_object_size" of 16777216 bytes exceeded while reading file "s3://xxxxxx.json" (>33554428 bytes). Try increasing "maximum_object_size".

This is likely due to the select * from '{{ read_location }}' trying to build a view with the default read_json_auto() and default options params.

Would it be possible to pass the read_json/read_parquet/read_csv functions and their options params?

@jwills
Copy link
Collaborator

jwills commented Jun 26, 2024

Yes, I think; there would need to be a PR that modified this function to let you override more of the defaults using the rendered_options dictionary (like we do for external materializations that use partitioning): https://github.com/duckdb/dbt-duckdb/blob/master/dbt/adapters/duckdb/impl.py#L166

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants