Add model for table column metadata for Global state #5136
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Dinu John <[email protected]>
dinujoh
requested review from
sb2k16,
chenqi0805,
engechas,
graytaylor0,
kkondaka,
KarstenSchnitter,
dlvenable and
oeyh
as code owners
oeyh
reviewed
Oct 31, 2024
| private static final String DB_METADATA_KEY = "dbMetadata"; | ||
| private static final String TABLE_COLUMN_METADATA_KEY = "tableColumnDataTypeMap"; | ||
| private final DbMetadata dbMetadata; | ||
| private final Map<String, Map<String, String>> tableColumnDataTypeMap; |
There was a problem hiding this comment.
Can you clarify the key and value in this map? Maybe add a javadoc?
There was a problem hiding this comment.
It's Map of table name to table column data type map. e.g. { "table1" : { "column1" : "int", "column2" : "varchar" }, "table2" : { "column1" : "int" } }
Will add java doc
|
|
||
| import java.util.Map; | ||
|
|
||
| public class DbTableMetadata { |
There was a problem hiding this comment.
It's called DbTableMetadata, yet it doesn't contain metadata about the table, like table name and database (schema) name. Wondering how this model will be used.
There was a problem hiding this comment.
Ok. I got most of it from description. When do we need to pull DbMetadata from the coordination store?
There was a problem hiding this comment.
I will have following PRs. This will be stored in global state. It will be used in BinLogListener and ExportWorker.
chenqi0805
approved these changes
Oct 31, 2024
oeyh
approved these changes
Oct 31, 2024
san81
added a commit
to san81/data-prepper
that referenced
this pull request
Nov 2, 2024
# This is the 1st commit message: additional test coverage Signed-off-by: Santhosh Gandhe <[email protected]> # This is the commit message #2: cleaned up JiraOauthConfig file Signed-off-by: Santhosh Gandhe <[email protected]> # This is the commit message #3: addressing review comments and simplifying the exception handling Signed-off-by: Santhosh Gandhe <[email protected]> # This is the commit message #4: Add external origination time for events created from S3 Object (opensearch-project#5104) Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]> # This is the commit message #5: moved the wait block out of the catch block Signed-off-by: Santhosh Gandhe <[email protected]> # This is the commit message #6: Renewal logic adjusted Signed-off-by: Santhosh Gandhe <[email protected]> # This is the commit message #7: partial Signed-off-by: Maxwell Brown <[email protected]> fix merge issues Signed-off-by: Maxwell Brown <[email protected]> update Signed-off-by: Maxwell Brown <[email protected]> Add getColumnDataTypes method to SchemaManager to get datatype for table columns (opensearch-project#5135) Add getColumnDataTypes method to SchemaManager Signed-off-by: Dinu John <[email protected]> Add model for table column metadata for Global state (opensearch-project#5136) Signed-off-by: Dinu John <[email protected]> Rename the KDS source plugin name to "kinesis-data-streams" (opensearch-project#5138) Signed-off-by: Souvik Bose <[email protected]> Co-authored-by: Souvik Bose <[email protected]> Addressed review comments (opensearch-project#5108) Signed-off-by: Krishna Kondaka <[email protected]> fixes related to source config properties change Signed-off-by: Santhosh Gandhe <[email protected]> removed future handling for loop based operations Signed-off-by: Santhosh Gandhe <[email protected]> additional test cases Signed-off-by: Santhosh Gandhe <[email protected]> addressing review comments Signed-off-by: Santhosh Gandhe <[email protected]> Jira Service Test coverage Signed-off-by: Maxwell Brown <[email protected]> jirasourceconfigTest comments Signed-off-by: Maxwell Brown <[email protected]> introduced RestClient and moved rest template interactions to there. Similar chage on the test cases too Signed-off-by: Santhosh Gandhe <[email protected]> backingoff for any kind of exception. Signed-off-by: Santhosh Gandhe <[email protected]> restructured constants file Signed-off-by: Santhosh Gandhe <[email protected]> JiraSourceTests Signed-off-by: Maxwell Brown <[email protected]> JiraItemInfo coverage Signed-off-by: Maxwell Brown <[email protected]> jira service branch coverage Signed-off-by: Maxwell Brown <[email protected]> branch coverage jira service Signed-off-by: Maxwell Brown <[email protected]> move add Items to queue logic into JiraItemInfo Signed-off-by: Maxwell Brown <[email protected]> fixing regex and adding date time formatter Signed-off-by: Maxwell Brown <[email protected]> Revert "Jira source" re add changes and fix issues Signed-off-by: Maxwell Brown <[email protected]> unneeded comment Signed-off-by: Maxwell Brown <[email protected]> using issue bean methods to simplify the logic Signed-off-by: Santhosh Gandhe <[email protected]>
san81
pushed a commit
to san81/data-prepper
that referenced
this pull request
Nov 3, 2024
…ect#5136) Signed-off-by: Dinu John <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add model for table column metadata for Global state. The model will be stored in global state and will be available for export and bin log worker to fetch table column metadata such as data type.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.