Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

source-kafka: support schema registry and Avro decoding #2089

Merged
merged 16 commits into from
Nov 15, 2024
Merged

source-kafka: support schema registry and Avro decoding #2089

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
5a3e3ef
source-kafka(refactor): format existing code
williamhbaker Oct 18, 2024
dace9e6
source-kafka: refactor code & support backfills
williamhbaker Oct 25, 2024
e95da7f
source-kafka: rework tests to use `flowctl preview`
williamhbaker Oct 25, 2024
b803231
source-kafka: simplify integration tests
williamhbaker Oct 25, 2024
1c194ae
source-kafka: discover collection keys for Avro schemas using schema …
williamhbaker Oct 29, 2024
c3b1323
source-kafka: capture Avro encoded messages using schema registry
williamhbaker Oct 30, 2024
64d4fde
source-kafka: discovery for JSON schema keys
williamhbaker Nov 4, 2024
234bf2c
source-kafka: add deletions to snapshot tests
williamhbaker Nov 4, 2024
12e1040
source-kafka: capture headers and timestamps
williamhbaker Nov 4, 2024
35b48d8
source-kafka: error messages and test cleanup
williamhbaker Nov 6, 2024
3569398
source-kafka: merge deletion events
williamhbaker Nov 6, 2024
0ac7a0e
source-kafka: various fixes
williamhbaker Nov 6, 2024
253ee5e
source-kafka: do not use async stdout
williamhbaker Nov 6, 2024
9d89f3f
source-kafka: fix AWS MSK connectivity
williamhbaker Nov 7, 2024
67f4db1
source-kafka: try to parse schematized messages as JSON if schema reg…
williamhbaker Nov 7, 2024
32b0d3c
source-kafka: minor adjustments to error messages and schema registry…
williamhbaker Nov 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ jobs:
"materialize-dynamodb",
"materialize-elasticsearch",
"source-dynamodb",
"source-kafka",
"source-kinesis",
"source-mysql",
"source-postgres",
Expand Down Expand Up @@ -232,11 +233,6 @@ jobs:
cd ..;
pytest ${{matrix.connector}}/tests;

- name: Start Dockerized test infrastructure
if: matrix.connector == 'source-kafka'
run: |
docker compose --file infra/docker-compose.yaml up --wait

- name: Source connector ${{ matrix.connector }} integration tests
if: |
contains(fromJson('[
Expand Down
65 changes: 0 additions & 65 deletions infra/docker-compose.yaml
View file
Open in desktop

This file was deleted.

21 changes: 0 additions & 21 deletions infra/kafka_jaas.conf
View file
Open in desktop

This file was deleted.

Loading
Loading