Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](move-memtable) cancel load rapidly when stream close wait #29322

Merged
merged 3 commits into from
Dec 31, 2023
Merged

[improve](move-memtable) cancel load rapidly when stream close wait #29322

merged 3 commits into from
Dec 31, 2023

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Dec 29, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@sollhui sollhui force-pushed the load_stream_close_wait branch from e2e4501 to e25d5dd Compare December 29, 2023 11:34
@sollhui
Copy link
Contributor Author

sollhui commented Dec 29, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@sollhui sollhui force-pushed the load_stream_close_wait branch from e25d5dd to cfd895b Compare December 29, 2023 11:44
@sollhui
Copy link
Contributor Author

sollhui commented Dec 29, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit cfd895b847d2a6e28fd78011419c347f82da7e30, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5668	5179	5238	5179
q2	402	193	179	179
q3	1459	1184	1190	1184
q4	1096	819	766	766
q5	3087	3144	3078	3078
q6	243	141	139	139
q7	1007	567	525	525
q8	2169	2289	2234	2234
q9	6717	6659	6669	6659
q10	3177	3118	3171	3118
q11	354	229	227	227
q12	396	246	246	246
q13	4409	3647	3633	3633
q14	258	222	217	217
q15	612	586	571	571
q16	450	426	408	408
q17	1046	569	593	569
q18	7067	6784	6794	6784
q19	1654	1546	1441	1441
q20	566	365	366	365
q21	2891	2469	2533	2469
q22	393	313	316	313
Total cold run time: 45121 ms
Total hot run time: 40304 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5118	5137	5052	5052
q2	342	250	270	250
q3	3381	3299	3304	3299
q4	2183	2008	2004	2004
q5	5943	5905	5900	5900
q6	236	134	135	134
q7	2392	1934	1920	1920
q8	3576	3670	3678	3670
q9	9125	9013	8969	8969
q10	3876	3928	3930	3928
q11	597	479	477	477
q12	821	659	622	622
q13	3908	3223	3156	3156
q14	298	273	279	273
q15	615	578	567	567
q16	551	518	540	518
q17	2028	1819	1780	1780
q18	8840	8415	8322	8322
q19	1763	1734	1677	1677
q20	2281	2012	1986	1986
q21	5729	5351	5351	5351
q22	568	496	468	468
Total cold run time: 64171 ms
Total hot run time: 60323 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.5 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183454787 Bytes

@sollhui sollhui force-pushed the load_stream_close_wait branch from a5330d1 to a4e16ac Compare December 29, 2023 12:41
@sollhui
Copy link
Contributor Author

sollhui commented Dec 29, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit a4e16ace85565abf4f0dc0cc8416a8001867aad1, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5528	5163	5212	5163
q2	392	170	158	158
q3	1479	1209	1216	1209
q4	1125	874	850	850
q5	3136	3012	3143	3012
q6	239	139	139	139
q7	997	579	545	545
q8	2153	2276	2211	2211
q9	6721	6664	6681	6664
q10	3203	3147	3141	3141
q11	353	233	217	217
q12	402	252	247	247
q13	4407	3645	3660	3645
q14	256	227	225	225
q15	639	573	557	557
q16	497	405	398	398
q17	1044	601	545	545
q18	7144	6811	6800	6800
q19	1654	1580	1488	1488
q20	553	376	373	373
q21	2945	2480	2557	2480
q22	402	317	327	317
Total cold run time: 45269 ms
Total hot run time: 40384 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5185	5119	5078	5078
q2	337	251	252	251
q3	3391	3337	3332	3332
q4	2188	2016	2010	2010
q5	5969	5934	5961	5934
q6	237	135	134	134
q7	2386	1939	1932	1932
q8	3563	3631	3677	3631
q9	9062	8994	9027	8994
q10	3875	3920	3902	3902
q11	595	504	492	492
q12	812	682	643	643
q13	3879	3220	3206	3206
q14	306	262	276	262
q15	620	571	561	561
q16	579	507	506	506
q17	2040	1803	1828	1803
q18	8767	8314	8406	8314
q19	1760	1681	1760	1681
q20	2303	2010	1999	1999
q21	5705	5324	5347	5324
q22	557	474	465	465
Total cold run time: 64116 ms
Total hot run time: 60454 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.21 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17188491003 Bytes

@sollhui sollhui force-pushed the load_stream_close_wait branch 2 times, most recently from 2eb9809 to 559b1f3 Compare December 29, 2023 14:34
@sollhui
Copy link
Contributor Author

sollhui commented Dec 29, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 559b1f3a2002e19d1f4620ebe8687c94732dab36, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5411	5151	5139	5139
q2	392	153	179	153
q3	1462	1179	1249	1179
q4	1098	809	883	809
q5	3138	3201	3155	3155
q6	234	142	138	138
q7	979	582	532	532
q8	2170	2298	2305	2298
q9	6752	6702	6651	6651
q10	3218	3144	3131	3131
q11	349	240	227	227
q12	389	237	239	237
q13	4391	3620	3675	3620
q14	246	224	231	224
q15	629	567	578	567
q16	455	420	396	396
q17	1049	601	565	565
q18	7059	6748	6788	6748
q19	1652	1576	1570	1570
q20	612	365	363	363
q21	2919	2440	2548	2440
q22	390	321	323	321
Total cold run time: 44994 ms
Total hot run time: 40463 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5172	5018	5029	5018
q2	354	250	248	248
q3	3358	3300	3291	3291
q4	2186	1984	1959	1959
q5	5962	5916	5953	5916
q6	235	131	129	129
q7	2397	1921	1951	1921
q8	3564	3664	3673	3664
q9	9050	8981	9017	8981
q10	3908	3934	3951	3934
q11	592	498	487	487
q12	821	652	651	651
q13	3899	3218	3172	3172
q14	303	258	269	258
q15	617	567	545	545
q16	553	526	494	494
q17	2002	1820	1769	1769
q18	8798	8406	8324	8324
q19	1765	1726	1727	1726
q20	2293	2005	1987	1987
q21	5808	5398	5294	5294
q22	627	538	563	538
Total cold run time: 64264 ms
Total hot run time: 60306 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.19 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17184236803 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.63% (8610/23507)
Line Coverage: 28.70% (70022/244020)
Region Coverage: 27.69% (36238/130872)
Branch Coverage: 24.40% (18524/75926)
Coverage Report: http://coverage.selectdb-in.cc/coverage/559b1f3a2002e19d1f4620ebe8687c94732dab36_559b1f3a2002e19d1f4620ebe8687c94732dab36/report/index.html

@sollhui sollhui force-pushed the load_stream_close_wait branch from 559b1f3 to c41b931 Compare December 30, 2023 03:18
@sollhui
Copy link
Contributor Author

sollhui commented Dec 30, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@sollhui sollhui force-pushed the load_stream_close_wait branch 3 times, most recently from 2a751a5 to af259c4 Compare December 30, 2023 03:36
@sollhui
Copy link
Contributor Author

sollhui commented Dec 30, 2023

run buildall

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit cc4a005b48a7809750255601de4d4404ea5dcebf, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	923	357	330	330
query2	6416	1995	2011	1995
query3	6660	210	202	202
query4	26959	22479	22309	22309
query5	6042	510	552	510
query6	274	188	187	187
query7	4577	289	282	282
query8	231	203	196	196
query9	8280	2681	2729	2681
query10	465	257	255	255
query11	16361	15570	15642	15570
query12	143	85	77	77
query13	1655	343	336	336
query14	11765	7047	7235	7047
query15	257	192	192	192
query16	6323	271	273	271
query17	1829	513	493	493
query18	1930	271	263	263
query19	267	139	140	139
query20	84	79	80	79
query21	184	97	93	93
query22	5005	4654	4546	4546
query23	32469	31352	31190	31190
query24	11086	2828	2816	2816
query25	571	351	343	343
query26	1459	144	146	144
query27	2861	284	282	282
query28	7309	2034	2027	2027
query29	1161	394	406	394
query30	285	137	146	137
query31	968	776	768	768
query32	90	58	63	58
query33	713	284	273	273
query34	826	439	438	438
query35	901	747	745	745
query36	1302	1205	1099	1099
query37	110	76	78	76
query38	3380	3300	3248	3248
query39	1335	1274	1277	1274
query40	188	94	90	90
query41	39	36	35	35
query42	101	90	93	90
query43	531	502	521	502
query44	1114	732	737	732
query45	197	201	190	190
query46	1079	670	649	649
query47	1726	1576	1482	1482
query48	342	266	267	266
query49	1085	329	328	328
query50	738	329	332	329
query51	5478	5361	5347	5347
query52	95	92	87	87
query53	210	149	141	141
query54	1333	574	592	574
query55	98	93	87	87
query56	225	200	205	200
query57	1049	956	960	956
query58	229	213	201	201
query59	2908	2658	2608	2608
query60	263	232	235	232
query61	88	94	86	86
query62	623	448	441	441
query63	168	157	156	156
query64	4981	1791	1719	1719
query65	3296	3272	3264	3264
query66	968	336	327	327
query67	15782	15020	15233	15020
query68	11673	523	539	523
query69	525	253	268	253
query70	1596	1556	1483	1483
query71	501	232	234	232
query72	5767	3621	3577	3577
query73	2399	318	317	317
query74	7020	6437	6522	6437
query75	4984	2292	2231	2231
query76	6319	1113	1127	1113
query77	701	285	278	278
query78	9126	8860	8537	8537
query79	1067	525	527	525
query80	674	377	366	366
query81	495	210	206	206
query82	223	108	105	105
query83	160	149	135	135
query84	242	54	53	53
query85	963	283	287	283
query86	396	378	372	372
query87	3524	3389	3385	3385
query88	2910	2277	2296	2277
query89	353	263	266	263
query90	1865	202	210	202
query91	120	99	93	93
query92	61	53	54	53
query93	1812	501	500	500
query94	804	190	191	190
query95	481	419	406	406
query96	634	322	321	321
query97	4309	4139	4141	4139
query98	207	199	190	190
query99	1076	842	804	804
Total cold run time: 291377 ms
Total hot run time: 179353 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.79 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17184347875 Bytes

@sollhui
Copy link
Contributor Author

sollhui commented Dec 30, 2023

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit ffce52a625abe3a04536658d1dbfc787d08816ba, data reload: false

------ Round 1 ----------------------------------
q1	17646	5751	5159	5159
q2	2034	158	147	147
q3	10683	1071	1122	1071
q4	10311	790	896	790
q5	7839	3018	2999	2999
q6	209	129	131	129
q7	933	548	538	538
q8	9311	2037	2067	2037
q9	6873	6428	6428	6428
q10	8248	3066	3024	3024
q11	429	217	209	209
q12	386	234	231	231
q13	18041	3714	3769	3714
q14	262	225	221	221
q15	576	553	543	543
q16	463	403	398	398
q17	985	525	490	490
q18	7392	6763	6716	6716
q19	1593	1382	1365	1365
q20	689	346	337	337
q21	2804	2366	2457	2366
q22	375	334	329	329
Total cold run time: 108082 ms
Total hot run time: 39241 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5149	5139	5086	5086
q2	344	256	270	256
q3	3410	3269	3249	3249
q4	2117	2021	1985	1985
q5	5790	5793	5785	5785
q6	214	122	125	122
q7	2283	1857	1875	1857
q8	3358	3452	3472	3452
q9	8903	8783	8749	8749
q10	3809	3841	3842	3841
q11	591	462	481	462
q12	802	661	643	643
q13	5727	3180	3178	3178
q14	288	274	273	273
q15	609	536	550	536
q16	554	490	534	490
q17	1943	1745	1733	1733
q18	8698	8426	8275	8275
q19	1631	1626	1605	1605
q20	2204	1957	1928	1928
q21	5514	5285	5237	5237
q22	519	497	516	497
Total cold run time: 64457 ms
Total hot run time: 59239 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit ffce52a625abe3a04536658d1dbfc787d08816ba, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5364	5144	5119	5119
q2	388	165	157	157
q3	1479	1178	1174	1174
q4	1084	844	799	799
q5	3117	3188	3140	3140
q6	226	132	129	129
q7	981	548	545	545
q8	2174	2226	2205	2205
q9	6688	6652	6655	6652
q10	3171	3181	3168	3168
q11	332	226	217	217
q12	381	229	239	229
q13	4395	3614	3661	3614
q14	255	220	221	220
q15	611	573	556	556
q16	442	396	416	396
q17	1037	579	541	541
q18	7084	6796	6769	6769
q19	1642	1497	1405	1405
q20	602	334	364	334
q21	2921	2527	2456	2456
q22	374	329	329	329
Total cold run time: 44748 ms
Total hot run time: 40154 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5069	5134	5068	5068
q2	331	264	261	261
q3	3370	3312	3302	3302
q4	2142	2008	1985	1985
q5	5938	5878	5897	5878
q6	233	125	124	124
q7	2385	1924	1914	1914
q8	3592	3659	3674	3659
q9	9020	8956	8959	8956
q10	3869	3912	3894	3894
q11	607	486	507	486
q12	817	659	644	644
q13	3863	3207	3213	3207
q14	297	281	282	281
q15	615	552	550	550
q16	563	528	508	508
q17	2021	1801	1842	1801
q18	8724	8269	8388	8269
q19	1748	1707	1701	1701
q20	2277	2001	1970	1970
q21	5769	5442	5356	5356
q22	581	506	529	506
Total cold run time: 63831 ms
Total hot run time: 60320 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.63% (8617/23523)
Line Coverage: 28.69% (70049/244157)
Region Coverage: 27.68% (36253/130979)
Branch Coverage: 24.38% (18529/75990)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ffce52a625abe3a04536658d1dbfc787d08816ba_ffce52a625abe3a04536658d1dbfc787d08816ba/report/index.html

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit ffce52a625abe3a04536658d1dbfc787d08816ba, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	933	363	345	345
query2	6435	2077	1918	1918
query3	6651	207	205	205
query4	26764	22541	22333	22333
query5	5092	549	534	534
query6	277	181	185	181
query7	4578	288	278	278
query8	235	219	209	209
query9	8221	2648	2635	2635
query10	413	271	275	271
query11	16197	15482	15466	15466
query12	138	84	84	84
query13	1669	329	338	329
query14	12237	7241	7221	7221
query15	234	187	191	187
query16	6481	284	274	274
query17	1810	497	492	492
query18	1921	274	256	256
query19	273	145	142	142
query20	83	78	84	78
query21	181	103	96	96
query22	5127	4713	4666	4666
query23	32241	31117	31344	31117
query24	11867	2823	2802	2802
query25	593	348	347	347
query26	1714	144	148	144
query27	2860	276	282	276
query28	7055	2007	1992	1992
query29	2113	408	402	402
query30	289	139	150	139
query31	984	780	783	780
query32	91	65	64	64
query33	722	287	276	276
query34	864	442	459	442
query35	940	765	752	752
query36	1294	1220	1150	1150
query37	197	79	78	78
query38	3386	3313	3258	3258
query39	1322	1305	1295	1295
query40	308	93	94	93
query41	40	36	35	35
query42	102	94	97	94
query43	538	515	502	502
query44	1071	728	731	728
query45	196	183	186	183
query46	1078	662	664	662
query47	1626	1515	1596	1515
query48	346	265	272	265
query49	1202	334	330	330
query50	732	331	354	331
query51	5422	5415	5370	5370
query52	97	87	93	87
query53	219	150	155	150
query54	1417	594	611	594
query55	105	91	88	88
query56	223	209	208	208
query57	1042	968	963	963
query58	235	220	207	207
query59	2706	2582	2641	2582
query60	262	242	250	242
query61	85	84	84	84
query62	655	488	477	477
query63	178	149	147	147
query64	5875	1747	1712	1712
query65	3349	3259	3242	3242
query66	1315	338	332	332
query67	15740	15825	15325	15325
query68	11924	549	529	529
query69	525	267	258	258
query70	1740	1548	1542	1542
query71	518	243	243	243
query72	5605	3590	3573	3573
query73	2712	316	321	316
query74	7109	6469	6479	6469
query75	5026	2316	2308	2308
query76	6209	1135	1072	1072
query77	657	285	276	276
query78	9085	8993	8579	8579
query79	4132	503	507	503
query80	692	380	378	378
query81	460	210	210	210
query82	227	103	106	103
query83	182	137	139	137
query84	246	58	56	56
query85	938	305	277	277
query86	432	404	403	403
query87	3524	3389	3336	3336
query88	3399	2289	2313	2289
query89	363	271	265	265
query90	1886	215	212	212
query91	127	87	91	87
query92	65	57	56	56
query93	2681	442	442	442
query94	848	192	193	192
query95	479	431	405	405
query96	629	323	319	319
query97	4289	4142	4199	4142
query98	226	200	190	190
query99	1102	865	846	846
Total cold run time: 298683 ms
Total hot run time: 180075 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.44 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.8 seconds inserted 10000000 Rows, about 359K ops/s
storage size: 17183605083 Bytes

dataroaring
dataroaring approved these changes Dec 31, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 31, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring force-pushed the load_stream_close_wait branch from ffce52a to a8a53b5 Compare December 31, 2023 06:30
@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit a8a53b53d8ff38d91c43de4113b5a2928407280c, data reload: false

------ Round 1 ----------------------------------
q1	17858	5293	5185	5185
q2	2023	161	151	151
q3	10598	1057	1041	1041
q4	10277	794	798	794
q5	7832	2985	2953	2953
q6	212	130	129	129
q7	945	555	510	510
q8	9288	2039	2055	2039
q9	6873	6391	6407	6391
q10	8279	3087	3074	3074
q11	434	208	216	208
q12	394	232	232	232
q13	18000	3654	3626	3626
q14	249	212	208	208
q15	597	554	538	538
q16	457	403	394	394
q17	999	620	582	582
q18	7425	6732	6666	6666
q19	1605	1375	1340	1340
q20	746	355	339	339
q21	2857	2411	2457	2411
q22	381	324	336	324
Total cold run time: 108329 ms
Total hot run time: 39135 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5220	5122	5153	5122
q2	349	251	254	251
q3	3329	3295	3242	3242
q4	2116	2011	2017	2011
q5	5859	5818	5807	5807
q6	217	130	125	125
q7	2324	1938	1892	1892
q8	3407	3433	3474	3433
q9	8882	8761	8763	8761
q10	3830	3879	3857	3857
q11	596	478	476	476
q12	789	675	646	646
q13	6365	3206	3215	3206
q14	292	279	276	276
q15	586	546	523	523
q16	565	515	502	502
q17	1933	1790	1776	1776
q18	8738	8394	8350	8350
q19	1646	1625	1602	1602
q20	2205	1991	1956	1956
q21	5578	5323	5260	5260
q22	540	503	473	473
Total cold run time: 65366 ms
Total hot run time: 59547 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit a8a53b53d8ff38d91c43de4113b5a2928407280c, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5411	5098	5177	5098
q2	384	159	158	158
q3	1466	1228	1238	1228
q4	1098	873	806	806
q5	3183	3069	3115	3069
q6	221	132	141	132
q7	976	550	520	520
q8	2160	2194	2197	2194
q9	6691	6684	6622	6622
q10	3208	3103	3121	3103
q11	339	226	222	222
q12	386	232	235	232
q13	4383	3667	3659	3659
q14	250	224	222	222
q15	625	556	534	534
q16	473	398	405	398
q17	1045	603	507	507
q18	7108	6775	6762	6762
q19	1652	1530	1442	1442
q20	597	356	343	343
q21	2903	2500	2505	2500
q22	397	316	332	316
Total cold run time: 44956 ms
Total hot run time: 40067 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5144	5095	5125	5095
q2	336	254	264	254
q3	3403	3308	3297	3297
q4	2153	2080	2055	2055
q5	5963	5965	5941	5941
q6	228	123	126	123
q7	2379	1936	1940	1936
q8	3582	3677	3663	3663
q9	9079	9000	8987	8987
q10	3889	3934	3945	3934
q11	593	483	486	483
q12	810	612	636	612
q13	3909	3191	3192	3191
q14	291	295	288	288
q15	612	545	549	545
q16	561	520	521	520
q17	2015	1811	1824	1811
q18	8754	8364	8341	8341
q19	1752	1687	1686	1686
q20	2297	1989	1982	1982
q21	5760	5367	5344	5344
q22	562	506	534	506
Total cold run time: 64072 ms
Total hot run time: 60594 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.62% (8613/23520)
Line Coverage: 28.67% (69988/244100)
Region Coverage: 27.66% (36229/130964)
Branch Coverage: 24.37% (18512/75976)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8a53b53d8ff38d91c43de4113b5a2928407280c_a8a53b53d8ff38d91c43de4113b5a2928407280c/report/index.html

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit a8a53b53d8ff38d91c43de4113b5a2928407280c, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	915	369	342	342
query2	4447	1922	2097	1922
query3	6645	208	203	203
query4	27008	22551	22519	22519
query5	4687	544	544	544
query6	265	215	183	183
query7	4580	265	271	265
query8	234	222	199	199
query9	8245	2549	2575	2549
query10	404	272	258	258
query11	16184	15571	15539	15539
query12	140	80	78	78
query13	1651	331	328	328
query14	11286	7078	7073	7073
query15	220	189	191	189
query16	6165	296	266	266
query17	1786	495	496	495
query18	1850	269	266	266
query19	239	146	136	136
query20	88	80	81	80
query21	168	99	96	96
query22	5161	4618	4744	4618
query23	32084	31266	31315	31266
query24	11835	2849	2797	2797
query25	561	350	347	347
query26	1412	146	148	146
query27	2865	281	288	281
query28	7028	1960	1967	1960
query29	1515	405	386	386
query30	276	143	145	143
query31	984	773	791	773
query32	81	63	61	61
query33	725	275	292	275
query34	871	442	441	441
query35	841	770	775	770
query36	1415	1325	1273	1273
query37	160	79	76	76
query38	3450	3318	3267	3267
query39	1337	1301	1275	1275
query40	288	91	94	91
query41	41	36	36	36
query42	104	92	100	92
query43	529	490	521	490
query44	1044	718	713	713
query45	196	193	184	184
query46	1075	644	641	641
query47	1662	1543	1590	1543
query48	340	255	257	255
query49	1069	328	327	327
query50	770	323	318	318
query51	5398	5226	5376	5226
query52	98	89	90	89
query53	224	163	156	156
query54	1182	571	575	571
query55	107	91	92	91
query56	215	202	202	202
query57	966	980	950	950
query58	232	215	207	207
query59	2786	2625	2637	2625
query60	244	235	236	235
query61	87	84	94	84
query62	649	468	447	447
query63	169	149	149	149
query64	5125	1843	1719	1719
query65	3364	3288	3275	3275
query66	1221	339	323	323
query67	15722	15379	15431	15379
query68	10230	527	527	527
query69	536	259	245	245
query70	1790	1613	1494	1494
query71	500	241	247	241
query72	5498	3607	3559	3559
query73	2061	325	311	311
query74	7062	6513	6508	6508
query75	4964	2287	2319	2287
query76	6152	1126	1155	1126
query77	646	285	269	269
query78	9142	8906	8514	8514
query79	1849	502	508	502
query80	682	383	362	362
query81	458	205	210	205
query82	213	102	103	102
query83	161	140	132	132
query84	246	53	59	53
query85	938	286	273	273
query86	416	398	403	398
query87	3531	3341	3368	3341
query88	4064	2271	2273	2271
query89	366	263	256	256
query90	1952	216	217	216
query91	120	88	93	88
query92	64	57	55	55
query93	3747	504	437	437
query94	886	196	189	189
query95	469	429	424	424
query96	653	322	316	316
query97	4279	4179	4198	4179
query98	206	207	200	200
query99	1088	798	832	798
Total cold run time: 289684 ms
Total hot run time: 179981 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.25 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17183795927 Bytes

dataroaring
dataroaring approved these changes Dec 31, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit bbf58c5 into apache:master Dec 31, 2023
25 of 26 checks passed
dataroaring pushed a commit that referenced this pull request Jan 2, 2024
@sollhui
Revert "[improve](move-memtable) cancel load rapidly when stream clos…
b748743 
…e wait (#29322)" (#29371)

This reverts commit bbf58c5.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 3, 2024
@sollhui @seawinde
Revert "[improve](move-memtable) cancel load rapidly when stream clos…
7cd522a 
…e wait (apache#29322)" (apache#29371)

This reverts commit bbf58c5.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@sollhui @HappenLee
[improve](move-memtable) cancel load rapidly when stream close wait (a…
665eb51 
…pache#29322)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@sollhui @HappenLee
Revert "[improve](move-memtable) cancel load rapidly when stream clos…
d223f81 
…e wait (apache#29322)" (apache#29371)

This reverts commit bbf58c5.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@kaijchen kaijchen kaijchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sollhui @doris-robot @dataroaring @kaijchen