br: pipeline wait tiflash synced #43726

3pointer · 2023-05-11T10:07:39Z

What problem does this PR solve?

Issue Number: close #43828

Problem Summary:
Currently if we restore to a cluster that has tiflash replicas. BR only send ingest command to leader and doesn't guarantee learner(tiflash reploica) ready to serve when restore finished. in some worst cases the lag between leader and learner may take hours.

What is changed and how it works?

This PR add a config wait-tiflash-ready to pipeline wait tiflash replica ready to serve when restore finished.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add an option for restore to wait tiflash ready to serve.

ti-chi-bot · 2023-05-11T10:07:41Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Leavrth
YuJuncen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2023-05-11T10:07:41Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

3pointer · 2023-05-15T09:55:40Z

/run-integration-br-tests

3pointer · 2023-05-16T02:22:32Z

/run-integration-br-tests

YuJuncen

rest lgtm

YuJuncen · 2023-05-24T03:41:15Z

br/pkg/restore/client.go

@@ -1601,77 +1603,83 @@ func (rc *Client) switchTiKVMode(ctx context.Context, mode import_sstpb.SwitchMo
 	return nil
 }

+func concurrentHandleTablesCh(


Can we make it generic?

func concurrentHandleTablesCh( ctx context.Context, inCh <-chan T, outCh chan<- T, errCh chan<- error, workers *utils.WorkerPool, processFun func(context.Context, T) error, deferFun func())

what's the benefit of make it generic?

So we don't need to change the outCh in Batcher into chan *CreatedTable

Leavrth · 2023-05-24T03:45:02Z

br/pkg/task/restore.go

+		afterTableCheckesumedCh := client.GoValidateChecksum(
+			ctx, afterTableRestoredCh, mgr.GetStorage().GetClient(), errCh, updateCh, cfg.ChecksumConcurrency)
+		afterTableLoadStatsCh := client.GoUpdateMetaAndLoadStats(ctx, afterTableCheckesumedCh, errCh)
+		postHandleCh = afterTableLoadStatsCh
 	}


does it need to add updateCh.IncBy(len(tables)) in the else statement?

nice catch. normally client.GoUpdateMetaAndLoadStats won't take too much time.
so I just ignore the progress of client.GoUpdateMetaAndLoadStats. and for others I add updateCh to trace progress.

Leavrth · 2023-05-24T05:23:34Z

br/pkg/restore/client.go

+			worker := workers.ApplyWorker()
+			eg.Go(func() error {
+				defer workers.RecycleWorker(worker)


equals to workers.ApplyOnErrorGroup

it's not equal. because we need pass argument(cloneTable) into goroutine.

Leavrth · 2023-05-24T06:58:21Z

br/pkg/restore/client.go

+				progress, err := infosync.CalculateTiFlashProgress(tbl.Table.ID, tbl.Table.TiFlashReplica.Count, tiFlashStores)
+				if err != nil {
+					log.Warn("failed to get tiflash replica progress, wait for next retry", zap.Error(err))
+					continue


need to also sleep to avoid frequent requests?

3pointer · 2023-06-09T04:17:14Z

/merge

3pointer · 2023-06-25T05:38:43Z

/merge

ti-chi-bot · 2023-06-25T05:38:44Z

@3pointer: We have migrated to builtin LGTM and approve plugins for reviewing.

Please use /approve when you want approve this pull request.

The changes announcement: LGTM plugin changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-06-27T02:30:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Leavrth, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Leavrth,YuJuncen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-06-27T02:30:42Z

[LGTM Timeline notifier]

Timeline:

2023-06-25 05:38:00.098361645 +0000 UTC m=+513245.499612089: ☑️ agreed by Leavrth.
2023-06-27 02:30:41.579990728 +0000 UTC m=+674806.981241177: ☑️ agreed by YuJuncen.

tiprow · 2023-06-27T02:32:10Z

@3pointer: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tiprow_fast_test	`c0a7dee`	link	true	`/test tiprow_fast_test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ti-chi-bot · 2023-06-28T06:57:11Z

In response to a cherrypick label: new pull request created to branch release-7.1: #45017.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2023-06-28T06:57:14Z

In response to a cherrypick label: new pull request created to branch release-6.5: #45018.

Signed-off-by: ti-chi-bot <[email protected]>

close #43828

br: pipeline wait tiflash synced

e84eca0

3pointer force-pushed the wait_tiflash_ready branch from fe38f37 to e84eca0 Compare May 11, 2023 10:11

3pointer added 2 commits May 15, 2023 14:45

update

26f98f1

add test

777ac5a

3pointer force-pushed the wait_tiflash_ready branch from 0095a64 to 777ac5a Compare May 15, 2023 09:22

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. labels May 15, 2023

3pointer marked this pull request as ready for review May 15, 2023 09:29

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 15, 2023

bazel

248aebd

fix

1db0d5a

YuJuncen reviewed May 24, 2023

View reviewed changes

Leavrth approved these changes May 24, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 24, 2023

3pointer added 2 commits May 29, 2023 15:49

address comment

75757e1

address comment

c0a7dee

YuJuncen approved these changes Jun 2, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 2, 2023

Leavrth approved these changes Jun 25, 2023

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jun 25, 2023

3pointer removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 25, 2023

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 25, 2023

YuJuncen approved these changes Jun 27, 2023

View reviewed changes

ti-chi-bot bot added the lgtm label Jun 27, 2023

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 27, 2023

ti-chi-bot bot merged commit 0f20315 into pingcap:master Jun 27, 2023

3pointer added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels Jun 28, 2023

ti-chi-bot mentioned this pull request Jun 28, 2023

br: pipeline wait tiflash synced (#43726) #45017

Merged

12 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jun 28, 2023

This is an automated cherry-pick of pingcap#43726

d13b769

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jun 28, 2023

br: pipeline wait tiflash synced (#43726) #45018

Merged

12 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jun 28, 2023

This is an automated cherry-pick of pingcap#43726

9b6b6f4

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot bot pushed a commit that referenced this pull request Jul 10, 2023

br: pipeline wait tiflash synced (#43726) (#45017)

11b3d30

close #43828

ti-chi-bot bot pushed a commit that referenced this pull request Aug 16, 2023

br: pipeline wait tiflash synced (#43726) (#45018)

9d2e744

close #43828

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

br: pipeline wait tiflash synced #43726

br: pipeline wait tiflash synced #43726

3pointer commented May 11, 2023 •

edited

Loading

ti-chi-bot bot commented May 11, 2023 •

edited

Loading

ti-chi-bot bot commented May 11, 2023

3pointer commented May 15, 2023

3pointer commented May 16, 2023

YuJuncen left a comment

YuJuncen May 24, 2023

YuJuncen May 24, 2023

3pointer May 29, 2023

YuJuncen May 30, 2023

Leavrth May 24, 2023

3pointer May 29, 2023

Leavrth May 24, 2023

3pointer May 29, 2023

Leavrth May 24, 2023

3pointer commented Jun 9, 2023

3pointer commented Jun 25, 2023

ti-chi-bot bot commented Jun 25, 2023

ti-chi-bot bot commented Jun 27, 2023

ti-chi-bot bot commented Jun 27, 2023

tiprow bot commented Jun 27, 2023

ti-chi-bot commented Jun 28, 2023

ti-chi-bot commented Jun 28, 2023

br: pipeline wait tiflash synced #43726

br: pipeline wait tiflash synced #43726

Conversation

3pointer commented May 11, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot bot commented May 11, 2023 • edited Loading

ti-chi-bot bot commented May 11, 2023

3pointer commented May 15, 2023

3pointer commented May 16, 2023

YuJuncen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

3pointer commented Jun 9, 2023

3pointer commented Jun 25, 2023

ti-chi-bot bot commented Jun 25, 2023

ti-chi-bot bot commented Jun 27, 2023

ti-chi-bot bot commented Jun 27, 2023

[LGTM Timeline notifier]

tiprow bot commented Jun 27, 2023

ti-chi-bot commented Jun 28, 2023

ti-chi-bot commented Jun 28, 2023

3pointer commented May 11, 2023 •

edited

Loading

ti-chi-bot bot commented May 11, 2023 •

edited

Loading