Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large columns #643

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open

Large columns #643

wants to merge 27 commits into from

Conversation

jsims-slower
Copy link

Retrieve large LOB columns from DB based on Kafka Messages.

@jsims-slower
Fix OrcUtil bug where convertStruct() returns an Array instead of an …
30ac707 
…OrcStruct. Added support to create DATE, TIMESTAMP and INT values from KafkaConnect Date, Timestamp and Time respectively.
@jsims-slower
Fix OrcUtil to correctly handle Kafka Connect Decimals.
d091270
@jsims-slower
Fix bug in OrcUtil.convertStruct where the outer TypeInfo is passed t…
b7e5ce4 
…hrough, not the field TypeInfo. Update comments. Fix related Test. Fix codestyle warnings.
@jsims-slower
Initial JDBC large-column support. Query back into the DB for columns…
b85fe16 
… too large for Kafka.
@jsims-slower
Correctly identify and load LOB columns from the DB, and write them t…
947b463 
…o HDFS.
@jsims-slower
Add support for Connection reconnect-and-retry. Refactor to fit Confl…
347f233 
…uent code format requirements. Simplify SQL Metadata caching.
@jsims-slower
Cache LOB Hashes in memory (LRU). Restructure code to be less complex…
ef10ed4 
… and also less flexible.
@jsims-slower
Make retries configurable. Move Retry logic into the retrySpec class.…
62b89bb 
… Move Record transformation into a separate class.
@jsims-slower
Make retries more intuitive. Make most SQLExceptions be retriable.
6958cf3
@jsims-slower
Refactor some code to be less coupled. Clean up some logging and add …
351fc49 
…some useful TODOs and other comments.
@jsims-slower
Improve logging, and related cleanup.
fd5f059
@jsims-slower
Minor refactor so JdbcTableInfo is less coupled to Record structure. …
f5d852f 
…Update several comments and TODOs
@jsims-slower
Use Hikari for DB Connection pooling and management. Remove custom DB…
bcb848a 
… management code.
@jsims-slower jsims-slower requested a review from a team as a code owner December 13, 2022 19:37
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@jsims-slower
Fix code-quality warnings.
2a55a16
@jsims-slower
Rename to JdbcColumnInfo, and add equals/hashCode methods.
3816732
@jsims-slower
Generify reading data from the original SourceRecord so testing will …
7a1f8ca 
…be easier.
@jsims-slower
Refactor how filtered columns are configured. Make hash-cache size co…
3c59661 
…nfigurable.
@jsims-slower
Make table+column configuration be in line with the source connector …
9011272 
…config structure.
@jsims-slower
Add support for float and double types as primary keys. Rename HashCa…
b6f2b56 
…che.
@jsims-slower
Remove repetitive method call in loop.
c1e8a19
@jsims-slower
Make Hash algorithm dynamic. Add LONG+VARCHAR as potential large colu…
0b0ff35 
…mns.
@jsims-slower
Merge remote-tracking branch 'upstream/master' into logical-types
3e15048
@jsims-slower
Merge branch 'logical-types' into large-columns
41e9996
@jsims-slower
Correctly handle null value from transformed record. Add VARCHAR type…
1e3eece 
…s to KafkaConnect Schema generator.
@jsims-slower
Make HashCache configurable.
dbb1864
@jsims-slower
Make HashCache configurable.
972200b
@jsims-slower
Make HashCache configurable.
bf12a16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jsims-slower @CLAassistant