-Support for Sink Codecs #3081
-Support for Sink Codecs #3081
Conversation
Signed-off-by: omkarmmore95 <[email protected]>
omkarmmore95
requested review from
chenqi0805,
engechas,
graytaylor0,
dinujoh,
kkondaka,
cmanning09,
asifsmohammed,
dlvenable and
oeyh
as code owners
| return "{\"type\":\"array\", \"items\":\"string\"}"; | ||
| } | ||
|
|
||
| public static void iterateRecursively(final StringBuilder mainSchemaBuilder, final String recordSchema, |
There was a problem hiding this comment.
This code appears to be taking a SQL-like schema and then converting it into an Avro schema. Right?
Why not just accept the Avro schema in the pipeline configuration?
This appears to be introducing an undefined language and there may be many different edge cases that we are not accounting for.
If this is using a well defined language, can we perform a model mapping to make sure it is valid? Say for example, it is a Postgresql DDL, can we parse the DDL using a Postgresql library into a Java model? Then we can perform a more accurate mapping.
There was a problem hiding this comment.
Yes @dlvenable , This is Raj's suggestion to support glue like schema and converting it onto avro schema, We are already accepting Avro schema in pipeline YAML itself,
As of now there is no library to do mapping between glue schema like structure to Avro/Parquet schema as there are nested and logical types in Avro/Parquet also.
There was a problem hiding this comment.
What is the syntax of this table structure?
There was a problem hiding this comment.
@dlvenable , Shared over mail, as I cant paste here
|
I'm closing this as obsolete. |
Signed-off-by: omkarmmore95 [email protected]
Description
support for tabular schema structure
Issues Resolved
resolved #2403
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.