Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-34853] Draft: Submit CDC Job To Flink K8S Native Application Mode #3093

Merged
merged 5 commits into from
Aug 8, 2024
Merged

[FLINK-34853] Draft: Submit CDC Job To Flink K8S Native Application Mode #3093

merged 5 commits into from
Aug 8, 2024

Conversation

czy006
Copy link
Contributor

@czy006 czy006 commented Mar 4, 2024

Support Submit CDC Job To Flink K8S Native Application Mode #2934
Design Doc: https://docs.google.com/document/d/1HlvRS5aVNqPtOx14mynEL3RFaZORuTFkWZAjvl4k998/edit?usp=sharing

@czy006 czy006 force-pushed the cli-kubernetes-submit branch 2 times, most recently from 5d0f014 to 9d7495f Compare March 8, 2024 09:46
@czy006 czy006 changed the title Draft: [2934]Submit CDC Job To Flink K8S Native Application Mode [cdc][cli][2934]Submit CDC Job To Flink K8S Native Application Mode Mar 8, 2024
herunkang2018
herunkang2018 reviewed Mar 12, 2024
View reviewed changes
Copy link

@herunkang2018 herunkang2018 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the great contribution, I left some comments.

flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontendOptions.java Show resolved Hide resolved
.../main/java/org/apache/flink/cdc/composer/flink/deployment/K8SApplicationComposeExecutor.java Outdated Show resolved Hide resolved
@czy006 czy006 changed the title [cdc][cli][2934]Submit CDC Job To Flink K8S Native Application Mode [FLINK-34853] Submit CDC Job To Flink K8S Native Application Mode Mar 21, 2024
laglangyue
laglangyue reviewed Mar 27, 2024
View reviewed changes
Dockerfile Outdated Show resolved Hide resolved
@czy006 czy006 changed the title [FLINK-34853] Submit CDC Job To Flink K8S Native Application Mode [FLINK-34853] Draft: Submit CDC Job To Flink K8S Native Application Mode Mar 28, 2024
@PatrickRen
Copy link
Contributor

@czy006 Could you rebase the PR to the latest master? Thanks

@beryllw beryllw mentioned this pull request May 28, 2024
Draft
@yuxiqian
Copy link
Contributor

yuxiqian commented Jul 15, 2024 •
edited
Loading

Hi @czy006, is there any updates on this? Seems this PR is still in draft stage and not ready for review and merge. Feel free to comment if there's anything blocking this.

yuxiqian
yuxiqian reviewed Jul 29, 2024
View reviewed changes
Copy link
Contributor

@yuxiqian yuxiqian left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for @czy006's great work! Still don't feel certain about some parts of this PR, could you please kindly clarify it?

Dockerfile Outdated Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
...r/src/main/java/org/apache/flink/cdc/composer/flink/deployment/ComposeDeploymentFactory.java Outdated Show resolved Hide resolved
flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/CliFrontendTest.java Show resolved Hide resolved
...in/java/org/apache/flink/cdc/composer/flink/deployment/K8SApplicationDeploymentExecutor.java Outdated Show resolved Hide resolved
Dockerfile
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are Dockerfile & mysql-doris generic deployment configurations or just an example? Maybe somewhere like /example or /conf is better than putting them at root?

...test/java/org/apache/flink/cdc/composer/flink/deployment/K8SApplicationDeploymentITCase.java Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any test cases for K8s deployment?

flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontendOptions.java Show resolved Hide resolved
mysql-doris.yaml Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same concern like Dockerfile, maybe it's not ideal to put it in root folder.

@czy006
Copy link
Contributor Author

czy006 commented Aug 7, 2024

Thanks for @PatrickRen help improving CI and Unit Tests. I will help Review this PR

@yuxiqian
Copy link
Contributor

yuxiqian commented Aug 7, 2024

Seems this PR changes how CDC CLI behaves. Previously, by running ./bin/flink-cdc.sh <YAML> without extra arguments, it submits the Flink job, prints Job ID / description and quits.

However, now CLI will not quit and prints all events out synchronously, as if it's in the "mini-cluster" mode. It won't quit until job finishes, so e2e / Migration tests will never finish.

yuxiqian
yuxiqian reviewed Aug 7, 2024
View reviewed changes
Dockerfile Show resolved Hide resolved
PatrickRen
PatrickRen approved these changes Aug 8, 2024
View reviewed changes
Copy link
Contributor

@PatrickRen PatrickRen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@czy006 Thanks for the PR! LGTM

@PatrickRen PatrickRen merged commit d6b687b into apache:master Aug 8, 2024
20 checks passed
qiaozongmi pushed a commit to qiaozongmi/flink-cdc that referenced this pull request Sep 23, 2024
@czy006 @qiaozongmi
[FLINK-34853] Submit CDC Job To Flink K8S Native Application Mode (ap…
54228b8 
…ache#3093)
@levrik
Copy link

levrik commented Oct 11, 2024

Is there any information on how to use this? It seems to be part of the 3.2.0 release but I couldn't find any documentation for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@herunkang2018 herunkang2018 herunkang2018 left review comments

@yuxiqian yuxiqian yuxiqian left review comments

@laglangyue laglangyue laglangyue left review comments

@lvyanquan lvyanquan lvyanquan left review comments

@PatrickRen PatrickRen PatrickRen approved these changes

Assignees
No one assigned
Labels
approved cli composer reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@czy006 @PatrickRen @yuxiqian @levrik @herunkang2018 @laglangyue @lvyanquan