Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large columns #643

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open

Large columns #643

wants to merge 27 commits into from

Commits on Nov 10, 2022

  1. Fix OrcUtil bug where convertStruct() returns an Array instead of an … 

    …OrcStruct. Added support to create DATE, TIMESTAMP and INT values from KafkaConnect Date, Timestamp and Time respectively.
    @jsims-slower
    jsims-slower committed Nov 10, 2022
    Configuration menu
    Copy the full SHA
    30ac707 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2022

  1. Fix OrcUtil to correctly handle Kafka Connect Decimals.

    @jsims-slower
    jsims-slower committed Nov 11, 2022
    Configuration menu
    Copy the full SHA
    d091270 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2022

  1. Fix bug in OrcUtil.convertStruct where the outer TypeInfo is passed t… 

    …hrough, not the field TypeInfo. Update comments. Fix related Test. Fix codestyle warnings.
    @jsims-slower
    jsims-slower committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    b7e5ce4 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2022

  1. Initial JDBC large-column support. Query back into the DB for columns… 

    … too large for Kafka.
    @jsims-slower
    jsims-slower committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    b85fe16 View commit details
    Browse the repository at this point in the history

  2. Correctly identify and load LOB columns from the DB, and write them t… 

    …o HDFS.
    @jsims-slower
    jsims-slower committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    947b463 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2022

  1. Add support for Connection reconnect-and-retry. Refactor to fit Confl… 

    …uent code format requirements. Simplify SQL Metadata caching.
    @jsims-slower
    jsims-slower committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    347f233 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2022

  1. Cache LOB Hashes in memory (LRU). Restructure code to be less complex… 

    … and also less flexible.
    @jsims-slower
    jsims-slower committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    ef10ed4 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2022

  1. Make retries configurable. Move Retry logic into the retrySpec class.… 

    … Move Record transformation into a separate class.
    @jsims-slower
    jsims-slower committed Dec 8, 2022
    Configuration menu
    Copy the full SHA
    62b89bb View commit details
    Browse the repository at this point in the history

  2. Make retries more intuitive. Make most SQLExceptions be retriable.

    @jsims-slower
    jsims-slower committed Dec 8, 2022
    Configuration menu
    Copy the full SHA
    6958cf3 View commit details
    Browse the repository at this point in the history

  3. Refactor some code to be less coupled. Clean up some logging and add … 

    …some useful TODOs and other comments.
    @jsims-slower
    jsims-slower committed Dec 8, 2022
    Configuration menu
    Copy the full SHA
    351fc49 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2022

  1. Improve logging, and related cleanup.

    @jsims-slower
    jsims-slower committed Dec 9, 2022
    Configuration menu
    Copy the full SHA
    fd5f059 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2022

  1. Minor refactor so JdbcTableInfo is less coupled to Record structure. … 

    …Update several comments and TODOs
    @jsims-slower
    jsims-slower committed Dec 12, 2022
    Configuration menu
    Copy the full SHA
    f5d852f View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2022

  1. Use Hikari for DB Connection pooling and management. Remove custom DB… 

    … management code.
    @jsims-slower
    jsims-slower committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    bcb848a View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2022

  1. Fix code-quality warnings.

    @jsims-slower
    jsims-slower committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    2a55a16 View commit details
    Browse the repository at this point in the history

  2. Rename to JdbcColumnInfo, and add equals/hashCode methods.

    @jsims-slower
    jsims-slower committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    3816732 View commit details
    Browse the repository at this point in the history

  3. Generify reading data from the original SourceRecord so testing will … 

    …be easier.
    @jsims-slower
    jsims-slower committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    7a1f8ca View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2022

  1. Refactor how filtered columns are configured. Make hash-cache size co… 

    …nfigurable.
    @jsims-slower
    jsims-slower committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    3c59661 View commit details
    Browse the repository at this point in the history

  2. Make table+column configuration be in line with the source connector … 

    …config structure.
    @jsims-slower
    jsims-slower committed Dec 16, 2022
    Configuration menu
    Copy the full SHA
    9011272 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2022

  1. Add support for float and double types as primary keys. Rename HashCa… 

    …che.
    @jsims-slower
    jsims-slower committed Dec 19, 2022
    Configuration menu
    Copy the full SHA
    b6f2b56 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2022

  1. Remove repetitive method call in loop.

    @jsims-slower
    jsims-slower committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    c1e8a19 View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2022

  1. Make Hash algorithm dynamic. Add LONG+VARCHAR as potential large colu… 

    …mns.
    @jsims-slower
    jsims-slower committed Dec 23, 2022
    Configuration menu
    Copy the full SHA
    0b0ff35 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2022

  1. Merge remote-tracking branch 'upstream/master' into logical-types

    @jsims-slower
    jsims-slower committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    3e15048 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'logical-types' into large-columns

    @jsims-slower
    jsims-slower committed Dec 29, 2022
    Configuration menu
    Copy the full SHA
    41e9996 View commit details
    Browse the repository at this point in the history

Commits on Dec 30, 2022

  1. Correctly handle null value from transformed record. Add VARCHAR type… 

    …s to KafkaConnect Schema generator.
    @jsims-slower
    jsims-slower committed Dec 30, 2022
    Configuration menu
    Copy the full SHA
    1e3eece View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2023

  1. Make HashCache configurable.

    @jsims-slower
    jsims-slower committed Jan 3, 2023
    Configuration menu
    Copy the full SHA
    dbb1864 View commit details
    Browse the repository at this point in the history

  2. Make HashCache configurable.

    @jsims-slower
    jsims-slower committed Jan 3, 2023
    Configuration menu
    Copy the full SHA
    972200b View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2023

  1. Make HashCache configurable.

    @jsims-slower
    jsims-slower committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    bf12a16 View commit details
    Browse the repository at this point in the history