Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flink 1.18.1 upgrade #48

Merged
merged 47 commits into from
Jan 23, 2025
Merged

Flink 1.18.1 upgrade #48

merged 47 commits into from
Jan 23, 2025

Conversation

jeffxiang
Copy link
Contributor

@jeffxiang jeffxiang commented Dec 12, 2024 •
edited
Loading

This PR upgrades psc-flink module to support Flink version 1.18.1. Major changes since the last upgrade to 1.15.1 include:

  • Implementation of DynamicPscSource. This is an experimental feature in 1.18.1; unit and integration tests pass but functionally untested in a prod-like environment.
  • Major refactoring of Source / Sink API's to accommodate changes between Flink 1.15.1 (last version supported by PSC-Flink) and 1.18.1
  • Support for native Kafka-Flink (KafkaSource/KafkaSink) checkpoint migration to Psc-Flink (PscSource/PscSink)
  • Some minor additions / changes to PSC-Core module to support the upgrade, and some minor code cleanups

Extensive unit, integration, and functional testing were conducted to ensure:

  • Backward-compatibility in checkpoint recovery with 1.15.1
  • Stability and reliability of both the old (DataStream) and new (Source/Sink) API's
  • Expected behavior with various startup modes
  • Automatic partition discovery

@jeffxiang
WIP 1.18 upgrade initial commit - added dynamic PSC source and finish…
ecf7d2c 
…ed first round of changes
@jeffxiang
Finish sink code changes
034ab2c
@jeffxiang
Finish source changes; add pause + resume API's to PscConsumer in cor…
39f067f 
…e to support new functionality in flink
@jeffxiang
Finish streaming.connectors.psc (mainly table) code changes
f5c45ae
@jeffxiang
Finish first round code changes for sink tests
9e01a18
@jeffxiang
Finish code changes for source tests
54a6c4f
@jeffxiang
WIP finish test code changes for streaming.connectors.psc top level
78c7b73
@jeffxiang
WIP finish test code changes for PscDynamicTableFactoryTest
d588c71
@jeffxiang
Finish table tests refactor
fdecfdc
@jeffxiang
Finish test refactor
4004bef
@jeffxiang
Compiles
16d3155
@jeffxiang
All dynamic.source tests pass except DynamicPscSourceITTest specific …
153d685 
…tests
@jeffxiang
WIP fixing basic cluster read dynamic source
b50fcbf
@jeffxiang
Remove extra debugging logic
bf4b858
@jeffxiang
Fixed DynamicPscSourceITTest
d325fb4
@jeffxiang
Finish fixing dynamic.source tests
5c95746
@jeffxiang
WIP all sink tests pass except one test checkProducerLeak()
d8d23e8
@jeffxiang
Finish source tests
8b6cea8
@jeffxiang
WIP finish fixing streaming.connectors.psc internals, shuffle, table …
a789999 
…tests
@jeffxiang
WIP fixing PscSerializerUpgradeTest
9f95a6b
@jeffxiang
Generated test files for migration/snapshot
bf362ae
@jeffxiang
Everything should pass except testBrokerFailure
18ced74
@jeffxiang
Fix testIteratorFor and testTransactionalProducerWithKafkaBackendCann…
7406f35 
…otBeNonTransactional
@jeffxiang
Force close pscProducer in testTransactionalProducerWithKafkaBackendC…
e30306f 
…annotBeNonTransactional after 1 second
@jeffxiang
Update log4j settings to prevent flood of build logs
3bd478d
@jeffxiang
Add repository io.confluent for kafka-schema-registry-client; change …
c7aa188 
…log level for tests to ERROR
@jeffxiang
Add thread.sleep(5s) in FlinkPscProducerITCase.testFlinkPscProducerFa…
70d83e2 
…ilBeforeNotify to allow for broker startup before deleting topic
@jeffxiang
Wrap metrics() in sychronized block to try and prevent CME
30a5b22
@jeffxiang
Split build to parallelize
7195742
@jeffxiang
Rename build workflow
4c5f16b
@jeffxiang
Attempt to allow for Flink-Kafka checkpoint recovery
ddc8172
@jeffxiang
Remove check to set clusterUri only once
defa065
@jeffxiang
Create PscMetadataClient config converter and related logic
c78756d
@jeffxiang
Update PscSubscriberTest to fix after secure protocol is allowable in…
3b00c3d 
… PscMetadataClient
@jeffxiang
Add logs to debug double consumption during checkpoint recovery from …
7d20d6c 
…native kafka
@jeffxiang
Add more debug logs
654cdf8
@jeffxiang
Implement versioning for PSC state serializers so that we can recover…
bcc57d5 
… from Kafka checkpoints
@jeffxiang
Add back PscSimpleTypeSerializerSnapshot to try and fix checkpoint re…
39d5dbb 
…covery issue; update PscTopicUriPartitionSplitSerializerTest
@jeffxiang
Revert "Add back PscSimpleTypeSerializerSnapshot to try and fix check…
84281db 
…point recovery issue; update PscTopicUriPartitionSplitSerializerTest"

This reverts commit 39d5dbb.
@jeffxiang
Update PscTopicUriPartitionSplitSerializerTest
06a836c
@jeffxiang
Add debug logs to investigate partition discovery
2386806
@jeffxiang
Remove unnecessary logs
f2c41e4
@jeffxiang
Dedupe between PscConfigurationInternal and PscConfiguration
c91994f
@jeffxiang
Fix missing import
7ae3cd0
@jeffxiang
Rearrange logs to prevent noise
e143f9c
@jeffxiang jeffxiang marked this pull request as ready for review January 22, 2025 21:31
@jeffxiang jeffxiang requested a review from a team as a code owner January 22, 2025 21:31
@jeffxiang jeffxiang merged commit c6566ce into 4.0 Jan 23, 2025
2 checks passed
@jeffxiang jeffxiang deleted the upgrade branch January 23, 2025 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vahidhashemian vahidhashemian vahidhashemian approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jeffxiang @vahidhashemian