Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](move-memtable) add multi replica fault injection #29348

Merged
merged 1 commit into from
Dec 31, 2023
Merged

[improve](move-memtable) add multi replica fault injection #29348

merged 1 commit into from
Dec 31, 2023

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Dec 31, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@sollhui sollhui force-pushed the multi_replica_fault_injection branch 2 times, most recently from 4d84d64 to 345b161 Compare December 31, 2023 03:00
@sollhui
Copy link
Contributor Author

sollhui commented Dec 31, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring previously approved these changes Dec 31, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sollhui sollhui closed this Dec 31, 2023
@sollhui sollhui force-pushed the multi_replica_fault_injection branch from 345b161 to 56c3487 Compare December 31, 2023 03:12
@sollhui sollhui reopened this Dec 31, 2023
@sollhui
Copy link
Contributor Author

sollhui commented Dec 31, 2023

run buildall

dataroaring
dataroaring approved these changes Dec 31, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 31, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.64% (8617/23517)
Line Coverage: 28.70% (70058/244133)
Region Coverage: 27.68% (36256/130967)
Branch Coverage: 24.39% (18531/75990)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a1132816f4b0d19d668285f45fd584b8ba4d83f5_a1132816f4b0d19d668285f45fd584b8ba4d83f5/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit a1132816f4b0d19d668285f45fd584b8ba4d83f5, data reload: false

------ Round 1 ----------------------------------
q1	17652	5177	5086	5086
q2	2018	161	145	145
q3	10523	1049	1088	1049
q4	10175	787	773	773
q5	7796	2912	2833	2833
q6	206	130	128	128
q7	927	561	499	499
q8	9302	2002	1974	1974
q9	6907	6362	6358	6358
q10	8247	2987	2982	2982
q11	435	217	214	214
q12	390	226	225	225
q13	18111	3663	3611	3611
q14	238	212	210	210
q15	605	541	535	535
q16	499	394	396	394
q17	944	478	486	478
q18	7318	6665	6682	6665
q19	1578	1372	1314	1314
q20	674	357	366	357
q21	2767	2335	2361	2335
q22	388	306	319	306
Total cold run time: 107700 ms
Total hot run time: 38471 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5126	5154	5079	5079
q2	336	250	236	236
q3	3309	3283	3267	3267
q4	2109	1991	1980	1980
q5	5777	5776	5781	5776
q6	215	122	122	122
q7	2296	1873	1931	1873
q8	3355	3457	3433	3433
q9	8792	8819	8711	8711
q10	3805	3820	3853	3820
q11	575	483	481	481
q12	793	654	630	630
q13	7081	3229	3184	3184
q14	293	275	293	275
q15	589	531	530	530
q16	528	507	523	507
q17	1914	1746	1790	1746
q18	8664	8276	8256	8256
q19	1634	1562	1572	1562
q20	2205	1965	1982	1965
q21	5508	5283	5164	5164
q22	536	521	515	515
Total cold run time: 65440 ms
Total hot run time: 59112 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit a1132816f4b0d19d668285f45fd584b8ba4d83f5, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	921	353	334	334
query2	6438	1954	2008	1954
query3	6660	212	205	205
query4	26453	22376	22583	22376
query5	5222	511	551	511
query6	270	193	178	178
query7	4597	291	261	261
query8	242	201	197	197
query9	8288	2725	2798	2725
query10	447	263	260	260
query11	16235	15676	15646	15646
query12	134	81	79	79
query13	1665	338	329	329
query14	12539	7184	7149	7149
query15	235	185	192	185
query16	6468	292	276	276
query17	1797	520	506	506
query18	1938	272	282	272
query19	238	148	139	139
query20	84	82	80	80
query21	181	101	98	98
query22	5060	4778	4553	4553
query23	32140	31480	31359	31359
query24	11855	2772	2884	2772
query25	579	358	351	351
query26	1687	147	148	147
query27	2849	283	281	281
query28	7001	2001	1989	1989
query29	2068	407	414	407
query30	287	144	147	144
query31	982	770	782	770
query32	92	58	60	58
query33	732	283	270	270
query34	894	455	462	455
query35	896	754	755	754
query36	1401	1289	1294	1289
query37	194	78	78	78
query38	3380	3304	3292	3292
query39	1352	1290	1298	1290
query40	311	98	95	95
query41	41	41	38	38
query42	97	90	93	90
query43	525	489	502	489
query44	1076	706	725	706
query45	202	190	189	189
query46	1065	671	653	653
query47	1716	1617	1517	1517
query48	348	267	271	267
query49	1207	330	335	330
query50	813	375	340	340
query51	5366	5285	5338	5285
query52	95	88	86	86
query53	232	166	163	163
query54	1420	592	611	592
query55	103	94	88	88
query56	215	206	206	206
query57	1010	967	992	967
query58	228	212	215	212
query59	2849	2648	2703	2648
query60	266	231	246	231
query61	88	98	85	85
query62	651	458	480	458
query63	176	153	158	153
query64	5867	1765	1725	1725
query65	3351	3277	3269	3269
query66	1317	334	343	334
query67	15835	15418	15152	15152
query68	12923	542	535	535
query69	525	263	260	260
query70	1742	1445	1576	1445
query71	491	241	249	241
query72	5604	3586	3566	3566
query73	2948	317	313	313
query74	6904	6386	6465	6386
query75	5156	2318	2297	2297
query76	6248	1092	1168	1092
query77	656	290	293	290
query78	9069	8764	8562	8562
query79	1252	513	509	509
query80	650	384	377	377
query81	452	214	212	212
query82	208	104	110	104
query83	188	141	141	141
query84	244	55	58	55
query85	952	286	286	286
query86	444	393	421	393
query87	3545	3381	3341	3341
query88	3315	2274	2275	2274
query89	365	265	280	265
query90	1942	209	219	209
query91	127	92	93	92
query92	69	57	55	55
query93	1572	506	439	439
query94	874	196	194	194
query95	484	428	416	416
query96	616	323	316	316
query97	4276	4152	4172	4152
query98	211	198	195	195
query99	1075	865	851	851
Total cold run time: 296098 ms
Total hot run time: 180250 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit a1132816f4b0d19d668285f45fd584b8ba4d83f5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5469	5127	5170	5127
q2	386	168	187	168
q3	1467	1146	1198	1146
q4	1098	829	836	829
q5	3167	3124	3091	3091
q6	224	134	129	129
q7	985	560	501	501
q8	2144	2213	2266	2213
q9	6657	6684	6689	6684
q10	3154	3179	3197	3179
q11	336	230	211	211
q12	387	244	240	240
q13	4414	3677	3663	3663
q14	257	214	208	208
q15	597	559	560	559
q16	449	417	408	408
q17	1049	523	462	462
q18	7053	6779	6707	6707
q19	1629	1560	1510	1510
q20	599	341	332	332
q21	2897	2429	2478	2429
q22	378	335	324	324
Total cold run time: 44796 ms
Total hot run time: 40120 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5169	5107	5101	5101
q2	341	243	244	243
q3	3407	3329	3292	3292
q4	2144	2006	1997	1997
q5	5958	5924	5943	5924
q6	226	126	126	126
q7	2386	1895	1959	1895
q8	3551	3663	3677	3663
q9	9058	8987	9001	8987
q10	3863	3935	3927	3927
q11	605	481	493	481
q12	799	655	635	635
q13	3907	3183	3180	3180
q14	297	268	274	268
q15	609	551	557	551
q16	554	528	518	518
q17	2019	1806	1827	1806
q18	8744	8335	8373	8335
q19	1754	1666	1681	1666
q20	2316	2003	1983	1983
q21	5766	5350	5533	5350
q22	557	494	488	488
Total cold run time: 64030 ms
Total hot run time: 60416 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.45 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17187858072 Bytes

@dataroaring dataroaring merged commit 9a90c1c into apache:master Dec 31, 2023
34 of 42 checks passed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@sollhui @HappenLee
[improve](move-memtable)add multi replica fault injection (apache#29348)
a415091
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@kaijchen kaijchen kaijchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sollhui @doris-robot @kaijchen @dataroaring