Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: Unstable unit test clientSuite.TestConcurrentProcessRangeRequest #179

Open
pingyu opened this issue Jul 26, 2022 · 2 comments
Open

cdc: Unstable unit test clientSuite.TestConcurrentProcessRangeRequest #179

pingyu opened this issue Jul 26, 2022 · 2 comments
Labels
status/stale type/bug Something isn't working

Comments

@pingyu
Copy link
Collaborator

pingyu commented Jul 26, 2022

Bug Report

1. Describe the bug

CI link: https://github.com/tikv/migration/runs/7512462840?check_suite_focus=true

Logs

=== Failed
=== FAIL: cdc/kv Test (276.63s)
Error: 7/26 02:42:20.489 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/26 02:42:21.384 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/26 02:42:22.341 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:45399, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]

----------------------------------------------------------------------
FAIL: client_test.go:[307](https://github.com/tikv/migration/runs/7512462840?check_suite_focus=true#step:4:308)1: clientSuite.TestConcurrentProcessRangeRequest

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:[318](https://github.com/tikv/migration/runs/7512462840?check_suite_focus=true#step:4:319)4:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3184:
    c.Errorf("no more events received")
... Error: no more events received

Error: 7/26 02:44:38.627 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/26 02:44:48.628 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/26 02:44:48.738 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/26 02:44:48.841 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/26 02:44:48.944 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 7/26 02:44:50.581 +00:00] [ERROR] [client_test.go:1714] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestIncompatibleTiKV.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1714\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:00.455 +00:00] [ERROR] [client_test.go:2939] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2939\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:01.221 +00:00] [ERROR] [client_test.go:2996] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2996\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:04.415 +00:00] [ERROR] [client_test.go:3423] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3423\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:05.506 +00:00] [ERROR] [client_test.go:3509] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3509\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:17.478 +00:00] [ERROR] [client_test.go:3328] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3328\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:17.483 +00:00] [ERROR] [client_test.go:3328] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3328\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:36.062 +00:00] [ERROR] [client_test.go:1509] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1509\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:36.232 +00:00] [ERROR] [client_test.go:1509] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1509\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 7/26 02:46:42.236 +00:00] [ERROR] [client_test.go:1[320](https://github.com/tikv/migration/runs/7512462840?check_suite_focus=true#step:4:321)] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1320\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
OOPS: 47 passed, 1 skipped, 1 FAILED

DONE 227 tests, 1 failure in 565.011s
make: *** [Makefile:119: unit_test_in_verify_ci] Error 1
Error: Process completed with exit code 2.

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

  • TiKV:
  • TiKV CDC:
  • TiKV BR:
  • TiKV Online Bulk Load:
@pingyu pingyu added the type/bug Something isn't working label Jul 26, 2022
@pingyu
Copy link
Collaborator Author

pingyu commented Aug 25, 2022

Another case: https://github.com/tikv/migration/runs/8010626295?check_suite_focus=true

=== Failed
=== FAIL: cdc/kv Test (292.92s)
Error: 8/25 07:27:59.222 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/25 07:27:59.652 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/25 07:28:01.014 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:40391, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]

----------------------------------------------------------------------
FAIL: client_test.go:[306](https://github.com/tikv/migration/runs/8010626295?check_suite_focus=true#step:4:307)4: clientSuite.TestConcurrentProcessRangeRequest

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

client_test.go:3177:
    c.Errorf("no more events received")
... Error: no more events received

Error: 8/25 07:30:19.516 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/25 07:30:31.472 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/25 07:30:31.578 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/25 07:30:31.681 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/25 07:30:31.783 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 8/25 07:30:33.192 +00:00] [ERROR] [client_test.go:1710] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestIncompatibleTiKV.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1710\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:31:44.162 +00:00] [ERROR] [client_test.go:2932] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2932\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:31:44.781 +00:00] [ERROR] [client_test.go:2989] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2989\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:31:49.040 +00:00] [ERROR] [client_test.go:3416] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3416\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:31:50.124 +00:00] [ERROR] [client_test.go:3502] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3502\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:32:09.554 +00:00] [ERROR] [client_test.go:3321] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3321\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:32:09.563 +00:00] [ERROR] [client_test.go:3321] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3321\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:32:28.143 +00:00] [ERROR] [client_test.go:1505] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1505\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:32:28.[307](https://github.com/tikv/migration/runs/8010626295?check_suite_focus=true#step:4:308) +00:00] [ERROR] [client_test.go:1505] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1505\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 8/25 07:32:35.907 +00:00] [ERROR] [client_test.go:1[316](https://github.com/tikv/migration/runs/8010626295?check_suite_focus=true#step:4:317)] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1316\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
OOPS: 48 passed, 1 skipped, 1 FAILED

DONE 227 tests, 1 failure in 651.[475](https://github.com/tikv/migration/runs/8010626295?check_suite_focus=true#step:4:476)s
make: *** [Makefile:119: unit_test_in_verify_ci] Error 1
Error: Process completed with exit code 2.

@github-actions
Copy link

This issue is stale because it has been open 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status/stale type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pingyu