Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](merge-on-write) should re-calculate delete bitmaps between segments if BE restart before publish #48775

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Mar 6, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

zhannngchen
zhannngchen previously approved these changes Mar 6, 2025
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice work

Copy link
Contributor

github-actions bot commented Mar 6, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 6, 2025
Copy link
Contributor

github-actions bot commented Mar 6, 2025

PR approved by anyone and no changes requested.

@bobhan1 bobhan1 force-pushed the fix-local-multi-segments-re-calc-in-publish branch from 0480989 to 04e7b90 Compare March 7, 2025 03:54
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 7, 2025
@bobhan1 bobhan1 force-pushed the fix-local-multi-segments-re-calc-in-publish branch from 04e7b90 to c0117f2 Compare March 7, 2025 04:08
@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32556 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c0117f2f83e59e51b61d17d154822a2d15a5209b, data reload: false

------ Round 1 ----------------------------------
q1	17695	5429	5141	5141
q2	2050	318	176	176
q3	10541	1543	729	729
q4	10247	1058	536	536
q5	7676	2382	2394	2382
q6	194	167	135	135
q7	920	764	607	607
q8	9313	1299	1159	1159
q9	5019	4589	4848	4589
q10	6812	2323	1905	1905
q11	477	277	266	266
q12	353	355	226	226
q13	17783	3677	3083	3083
q14	222	231	209	209
q15	538	499	486	486
q16	630	625	574	574
q17	585	873	334	334
q18	7023	6411	6291	6291
q19	1986	948	546	546
q20	303	322	199	199
q21	2777	2108	1994	1994
q22	1036	1004	989	989
Total cold run time: 104180 ms
Total hot run time: 32556 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5246	5253	5118	5118
q2	252	331	228	228
q3	2117	2667	2332	2332
q4	1435	1807	1385	1385
q5	4269	4107	4117	4107
q6	203	164	124	124
q7	2010	1934	1762	1762
q8	2630	2693	2530	2530
q9	7214	7155	7151	7151
q10	2994	3177	2822	2822
q11	579	514	498	498
q12	673	779	620	620
q13	3506	3888	3233	3233
q14	282	291	261	261
q15	517	470	476	470
q16	643	684	637	637
q17	1144	1612	1342	1342
q18	7825	7578	7319	7319
q19	854	797	1005	797
q20	1964	2002	1838	1838
q21	5449	4968	4727	4727
q22	1070	1009	997	997
Total cold run time: 52876 ms
Total hot run time: 50298 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c0117f2f83e59e51b61d17d154822a2d15a5209b, data reload: false

query1	963	373	376	373
query2	6525	1936	1924	1924
query3	6800	226	224	224
query4	26759	23665	23331	23331
query5	4349	670	492	492
query6	312	202	191	191
query7	4609	501	289	289
query8	319	242	238	238
query9	8649	2527	2530	2527
query10	481	310	248	248
query11	15665	15113	14964	14964
query12	157	108	106	106
query13	1646	518	414	414
query14	9508	6217	6268	6217
query15	199	185	176	176
query16	7383	598	472	472
query17	1190	703	554	554
query18	1951	393	292	292
query19	184	184	153	153
query20	120	110	114	110
query21	208	118	100	100
query22	4232	4527	4144	4144
query23	33800	32815	32853	32815
query24	8112	2347	2384	2347
query25	517	458	383	383
query26	1226	268	155	155
query27	2712	491	324	324
query28	4341	2395	2380	2380
query29	746	543	422	422
query30	280	222	185	185
query31	940	850	737	737
query32	76	64	68	64
query33	551	351	299	299
query34	769	847	519	519
query35	789	820	740	740
query36	938	983	862	862
query37	112	97	75	75
query38	4162	4162	3984	3984
query39	1435	1395	1391	1391
query40	204	110	99	99
query41	55	52	53	52
query42	123	104	103	103
query43	495	511	487	487
query44	1275	782	795	782
query45	177	168	161	161
query46	825	1022	618	618
query47	1757	1804	1731	1731
query48	384	412	298	298
query49	805	529	462	462
query50	672	733	402	402
query51	4170	4147	4113	4113
query52	109	107	94	94
query53	226	249	181	181
query54	475	477	410	410
query55	82	82	84	82
query56	282	269	261	261
query57	1131	1123	1119	1119
query58	249	237	246	237
query59	2690	2818	2547	2547
query60	274	258	299	258
query61	121	115	116	115
query62	869	726	699	699
query63	231	187	187	187
query64	4355	1003	651	651
query65	4434	4302	4316	4302
query66	1127	412	317	317
query67	15549	15458	15332	15332
query68	7974	863	514	514
query69	463	298	270	270
query70	1166	1123	1113	1113
query71	456	293	274	274
query72	5487	3610	3712	3610
query73	774	731	342	342
query74	9347	9184	8705	8705
query75	3841	3116	2672	2672
query76	3657	1175	785	785
query77	774	449	278	278
query78	9948	10164	9250	9250
query79	2029	880	582	582
query80	613	522	439	439
query81	481	256	220	220
query82	449	126	91	91
query83	171	172	170	170
query84	232	93	75	75
query85	802	342	298	298
query86	337	305	271	271
query87	4372	4597	4324	4324
query88	3618	2199	2264	2199
query89	389	315	289	289
query90	1930	205	205	205
query91	140	139	109	109
query92	71	60	54	54
query93	1255	1062	589	589
query94	673	408	310	310
query95	351	264	256	256
query96	478	555	275	275
query97	3317	3350	3283	3283
query98	234	206	208	206
query99	1381	1401	1280	1280
Total cold run time: 274071 ms
Total hot run time: 184659 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c0117f2f83e59e51b61d17d154822a2d15a5209b, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.56	0.55	0.55
query6	1.19	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.51
query10	0.57	0.61	0.58
query11	0.16	0.11	0.10
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	2.70	2.85	2.70
query15	0.92	0.84	0.84
query16	0.37	0.40	0.38
query17	1.05	1.01	1.00
query18	0.20	0.19	0.19
query19	1.90	1.78	2.03
query20	0.01	0.01	0.01
query21	15.38	0.87	0.54
query22	0.76	1.30	0.64
query23	14.87	1.40	0.65
query24	7.09	1.96	0.99
query25	0.49	0.29	0.12
query26	0.65	0.16	0.15
query27	0.06	0.04	0.04
query28	9.23	0.91	0.43
query29	12.62	4.09	3.41
query30	0.24	0.09	0.06
query31	2.81	0.62	0.37
query32	3.22	0.55	0.45
query33	2.94	3.04	3.01
query34	15.86	5.08	4.49
query35	4.48	4.51	4.49
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 104.86 s
Total hot run time: 30.97 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 10.00% (2/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 46.86% (12533/26745)
Line Coverage 36.45% (106844/293127)
Region Coverage 35.49% (54570/153761)
Branch Coverage 30.83% (27427/88976)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants