Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created a mysql-cdc source, but did not configure schema history to kafka topic, causing RisingWave to hang up #18734

Open
cumtcwf opened this issue Sep 26, 2024 · 0 comments
Assignees
Labels
type/bug Something isn't working
Milestone

Comments

@cumtcwf
Copy link

cumtcwf commented Sep 26, 2024

Describe the bug

  1. Created a mysql-cdc source, but did not configure schema history to kafka topic, causing RisingWave to hang up
  2. Created a risingwave table (t_test_risingwave) and mapped it to a table in the mysql-cdc source
  3. Cerated a mysql jdbc sink t_sink_risingwave from this table (t_test_risingwave)
  4. Modify the metadata of the MySQL table associated with the sink (change the type of a field from varchar to timestamp)
  5. Inserting new data into tables in MySQL will result in an error, causing RisingWave to crash

Error message/log

2024-09-26T06:55:32.642280809Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2239 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642287551Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2235 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642310586Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2234 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642300676Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2233 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642324382Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2241 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642317783Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2133 error=Executor error: exchange channel from local upstream actor 2162 closed unexpectedly
2024-09-26T06:55:32.642314431Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2242 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642355798Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2237 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642357431Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2230 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.642440875Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2238 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643181134Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2229 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643444446Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2228 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643458218Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2236 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly
2024-09-26T06:55:32.643512081Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2253 error=Executor error: exchange channel from local upstream actor 2138 closed unexpectedly
2024-09-26T06:55:32.643615174Z ERROR    rw-streaming risingwave_stream::task::stream_manager: actor exit with error actor_id=2232 error=Executor error: exchange channel from local upstream actor 2259 closed unexpectedly

java.lang.RuntimeException: Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=RW_CDC_222'
        at io.debezium.pipeline.JmxUtils.registerMXBean(JmxUtils.java:68)
        at io.debezium.metrics.Metrics.register(Metrics.java:65)
        at io.debezium.relational.history.SchemaHistoryMetrics.started(SchemaHistoryMetrics.java:102)
        at io.debezium.relational.history.AbstractSchemaHistory.start(AbstractSchemaHistory.java:69)
        at io.debezium.relational.HistorizedRelationalDatabaseSchema.<init>(HistorizedRelationalDatabaseSchema.java:50)
        at io.debezium.connector.mysql.MySqlDatabaseSchema.<init>(MySqlDatabaseSchema.java:82)
        at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:98)
        at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:142)
        at io.debezium.embedded.EmbeddedEngine.startSourceTask(EmbeddedEngine.java:918)
        at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:737)
        at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:229)
        at com.risingwave.connector.source.core.DbzCdcEngine.run(DbzCdcEngine.java:66)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.InterruptedException: sleep interrupted
        at java.base/java.lang.Thread.sleep(Native Method)
        at io.debezium.util.Metronome$1.pause(Metronome.java:57)
        at io.debezium.pipeline.JmxUtils.registerMXBean(JmxUtils.java:59)
        ... 14 more

Caused by these errors (recent errors listed first):
  1: gRPC request to meta service failed: Internal error
  2: The cluster is recovering
  3: get error from control stream: worker node 2, gRPC request to stream service failed: Internal error: failed to collect barrier for epoch [7215546658455552]: Actor 2375 exited unexpectedly: Executor error: Sink error: Remote sink error: INTERNAL: Error when exec 22001, message Data truncation: Incorrect datetime value: 'cccc' for column 'column2' at row 1;

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

@cumtcwf cumtcwf added the type/bug Something isn't working label Sep 26, 2024
@github-actions github-actions bot added this to the release-2.1 milestone Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants