scheduler: use move-hot-write-leader operator #7852

lhy1024 · 2024-02-26T04:28:52Z

What problem does this PR solve?

Issue Number: Close #7848

What is changed and how does it work?

Check List

Tests

Unit test
manual test with 6 tikv
- tpcc
- oltp_read_write
- oltp_write_only
- oltp_update_index
- oltp_update_non_index
- oltp_insert

Release note

None.

ti-chi-bot · 2024-02-26T04:28:54Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

HuSharp
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov · 2024-02-26T04:38:04Z

Codecov Report

Merging #7852 (b7fc80e) into master (1cd99f1) will decrease coverage by 0.04%.
The diff coverage is 91.11%.

❗ Current head b7fc80e differs from pull request most recent head 5462b89. Consider uploading reports for the commit 5462b89 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7852      +/-   ##
==========================================
- Coverage   77.25%   77.22%   -0.04%     
==========================================
  Files         468      468              
  Lines       60937    60872      -65     
==========================================
- Hits        47078    47009      -69     
- Misses      10302    10307       +5     
+ Partials     3557     3556       -1

Flag	Coverage Δ
unittests	`77.22% <91.11%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: lhy1024 <[email protected]>

rleungx · 2024-03-04T08:53:28Z

pkg/schedule/schedulers/hot_region_test.go

 			// peer in store 1 of the region 2 can transfer to store 5 or store 6 because of the label
 			operatorutil.CheckTransferPeerWithLeaderTransferFrom(re, op, operator.OpHotRegion, 1)
 		} else {
+			re.Equal("move-hot-write-leader", op.Desc())
+			re.Equal(5, op.Len())


why is the length different?

#7872 Move leader without joint will appear as an extra transfer leader, I'll try to adjust the test firstly to make it look more reasonable

Signed-off-by: lhy1024 <[email protected]>

rleungx · 2024-04-11T08:08:07Z

pkg/schedule/schedulers/hot_region_test.go

 		} else {
-			re.Equal("move-hot-write-leader", op.Desc())


no need it?

Signed-off-by: lhy1024 <[email protected]>

HuSharp · 2024-04-11T09:11:29Z

metrics/grafana/pd.json

@@ -5881,6 +5881,21 @@
              "intervalFactor": 1,
              "legendFormat": "store-{{store}}-in",
              "refId": "B"
+            },
+            {
+              "expr": "- sum(delta(pd_scheduler_hot_region_direction{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", store=~\"$store\",type=\"move-leader\",direction=\"out\",rw=\"write\"}[1m]))by (store)",


can you paste the grafana pic?

Suggested change

"expr": "- sum(delta(pd_scheduler_hot_region_direction{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", store=~\"$store\",type=\"move-leader\",direction=\"out\",rw=\"write\"}[1m]))by (store)",

"expr": "sum(delta(pd_scheduler_hot_region_direction{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", store=~\"$store\",type=\"move-leader\",direction=\"out\",rw=\"write\"}[1m]))by (store)",

It should be negative, because its direction is out

HuSharp · 2024-04-11T09:26:39Z

pkg/utils/operatorutil/operator_check.go

@@ -88,32 +104,36 @@ func CheckTransferLearner(re *require.Assertions, op *operator.Operator, kind op
 // CheckTransferPeerWithLeaderTransfer checks if the operator is to transfer
 // peer between the specified source and target stores and it meanwhile
 // transfers the leader out of source store.
+// If targetID is 0, it means the operator is to transfer peer to any store.
 func CheckTransferPeerWithLeaderTransfer(re *require.Assertions, op *operator.Operator, kind operator.OpKind, sourceID, targetID uint64) {


Do we need to add more comments to indicate these functions are used for the test?

Maybe I will refactor them in another pr.

lhy1024 · 2024-04-12T04:24:43Z

/merge

ti-chi-bot · 2024-04-12T04:24:45Z

@lhy1024: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-04-12T04:24:46Z

This pull request has been accepted and is ready to merge.

Commit hash: b7fc80e

ti-chi-bot · 2024-04-12T04:24:59Z

@lhy1024: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-04-12T04:35:30Z

In response to a cherrypick label: new pull request created to branch release-7.1: #8058.

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

close #7848 Signed-off-by: ti-chi-bot <[email protected]> Signed-off-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]>

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-05-07T05:51:25Z

In response to a cherrypick label: new pull request created to branch release-7.5: #8144.

close #7848 Signed-off-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]>

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-06-04T03:18:24Z

In response to a cherrypick label: new pull request created to branch release-6.5: #8242.

ti-chi-bot · 2024-07-29T07:13:30Z

In response to a cherrypick label: new pull request created to branch release-8.1: #8454.

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

close #7848 Signed-off-by: ti-chi-bot <[email protected]> Signed-off-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]>

lhy1024 requested a review from rleungx February 26, 2024 04:28

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 26, 2024

scheduler: use move-hot-write-leader operator

bb3a7bd

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the move-hot-write-leader branch from 1f9e78d to bb3a7bd Compare February 26, 2024 05:02

add metrics

2d2c4f5

Signed-off-by: lhy1024 <[email protected]>

rleungx reviewed Mar 4, 2024

View reviewed changes

lhy1024 added 2 commits March 7, 2024 16:38

use joint consensus in test

293ce17

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into move-hot-write-leader

3c73e3f

Signed-off-by: lhy1024 <[email protected]>

tikv deleted a comment from ti-chi-bot bot Mar 7, 2024

ti-chi-bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. and removed needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. labels Mar 31, 2024

ti-chi-bot added the needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. label Apr 11, 2024

rleungx reviewed Apr 11, 2024

View reviewed changes

pkg/schedule/schedulers/hot_region_test.go Outdated

} else {

re.Equal("move-hot-write-leader", op.Desc())

Copy link

Member

rleungx Apr 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need it?

lhy1024 added 2 commits April 11, 2024 16:58

address comments

797a290

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into move-hot-write-leader

b7fc80e

Signed-off-by: lhy1024 <[email protected]>

rleungx approved these changes Apr 11, 2024

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 11, 2024

HuSharp reviewed Apr 11, 2024

View reviewed changes

HuSharp approved these changes Apr 11, 2024

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 11, 2024

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 12, 2024

Merge branch 'master' into move-hot-write-leader

5462b89

ti-chi-bot bot merged commit 33ae3b6 into tikv:master Apr 12, 2024
22 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Apr 12, 2024

This is an automated cherry-pick of tikv#7852

25469b5

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 12, 2024

scheduler: use move-hot-write-leader operator (#7852) #8058

Merged

1 task

ti-chi-bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label May 7, 2024

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request May 7, 2024

This is an automated cherry-pick of tikv#7852

6c221cc

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request May 7, 2024

scheduler: use move-hot-write-leader operator (#7852) #8144

Merged

1 task

ti-chi-bot bot pushed a commit that referenced this pull request May 20, 2024

scheduler: use move-hot-write-leader operator (#7852) (#8144)

29483b4

close #7848 Signed-off-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]> Co-authored-by: lhy1024 <[email protected]>

lhy1024 added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Jun 4, 2024

lhy1024 deleted the move-hot-write-leader branch June 4, 2024 03:17

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jun 4, 2024

This is an automated cherry-pick of tikv#7852

fd870d5

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jun 4, 2024

scheduler: use move-hot-write-leader operator (#7852) #8242

Closed

1 task

ti-chi-bot added the needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. label Jul 29, 2024

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jul 29, 2024

This is an automated cherry-pick of tikv#7852

c9ef0bd

close tikv#7848 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jul 29, 2024

scheduler: use move-hot-write-leader operator (#7852) #8454

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scheduler: use move-hot-write-leader operator #7852

scheduler: use move-hot-write-leader operator #7852

lhy1024 commented Feb 26, 2024 •

edited

Loading

ti-chi-bot bot commented Feb 26, 2024 •

edited

Loading

codecov bot commented Feb 26, 2024 •

edited

Loading

rleungx Mar 4, 2024

lhy1024 Mar 4, 2024

rleungx Apr 11, 2024

HuSharp Apr 11, 2024

lhy1024 Apr 11, 2024

HuSharp Apr 11, 2024

HuSharp Apr 11, 2024

lhy1024 Apr 11, 2024

lhy1024 commented Apr 12, 2024

ti-chi-bot bot commented Apr 12, 2024

ti-chi-bot bot commented Apr 12, 2024

ti-chi-bot bot commented Apr 12, 2024

ti-chi-bot commented Apr 12, 2024

ti-chi-bot commented May 7, 2024

ti-chi-bot commented Jun 4, 2024

ti-chi-bot commented Jul 29, 2024

	"expr": "- sum(delta(pd_scheduler_hot_region_direction{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", store=~\"$store\",type=\"move-leader\",direction=\"out\",rw=\"write\"}[1m]))by (store)",
	"expr": "sum(delta(pd_scheduler_hot_region_direction{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", store=~\"$store\",type=\"move-leader\",direction=\"out\",rw=\"write\"}[1m]))by (store)",

scheduler: use move-hot-write-leader operator #7852

scheduler: use move-hot-write-leader operator #7852

Conversation

lhy1024 commented Feb 26, 2024 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Feb 26, 2024 • edited Loading

codecov bot commented Feb 26, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lhy1024 commented Apr 12, 2024

ti-chi-bot bot commented Apr 12, 2024

ti-chi-bot bot commented Apr 12, 2024

ti-chi-bot bot commented Apr 12, 2024

ti-chi-bot commented Apr 12, 2024

ti-chi-bot commented May 7, 2024

ti-chi-bot commented Jun 4, 2024

ti-chi-bot commented Jul 29, 2024

lhy1024 commented Feb 26, 2024 •

edited

Loading

ti-chi-bot bot commented Feb 26, 2024 •

edited

Loading

codecov bot commented Feb 26, 2024 •

edited

Loading