Added basic AWS Glue struct datatype support and JSON/CSV/PARQUET read_options #464
Conversation
|
Amazing, thank you @firewall413! Looking forward to reading this one |
| @@ -23,11 +23,11 @@ | |||
| """ | |||
|
|
|||
| config_materialized_parquet_location = """ | |||
| {{ config(materialized="external", location="{{ adapter.external_root() }}/test.parquet") }} | |||
| {{ config(materialized="external", location="{{ adapter.external_root() }}/test.parquet", format="parquet") }} | |||
There was a problem hiding this comment.
Ah so this is a behavior change? i.e., you now have to specify the format arg for external materializations in order to get the read options right for the corresponding view?
There was a problem hiding this comment.
Given that we need this info to config the views correctly, can we just infer it from the extension on the file if the format isn't specified? (And if there is no extension at all and no format specified, I think the default is parquet)?
Context here is that I don't want to break existing users when they upgrade unless it is completely unavoidable and even then we want to provide as much warning that it is coming as possible.
There was a problem hiding this comment.
ok yep, I can give that (infer the type from the extension) a try!
Added basic AWS Glue struct datatype support and JSON/CSV/PARQUET read_options