Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[debug](don't merge) test memtable on sink node #29115

Closed
wants to merge 4 commits into from
Closed

[debug](don't merge) test memtable on sink node #29115

wants to merge 4 commits into from

Conversation

kaijchen
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaijchen
Copy link
Contributor Author

run p0 10

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@kaijchen
Copy link
Contributor Author

run p0 10

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@kaijchen
Copy link
Contributor Author

run p0 10

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit f71f588b92b7415bb97d81edc648bde9916c3618, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4779	4414	4420	4414
q2	385	136	135	135
q3	1483	1309	1200	1200
q4	1137	1002	916	916
q5	3181	3152	3141	3141
q6	254	135	135	135
q7	984	506	489	489
q8	2249	2297	2251	2251
q9	6763	6733	6710	6710
q10	3219	3302	3285	3285
q11	322	213	203	203
q12	357	206	205	205
q13	4557	3758	3790	3758
q14	242	213	219	213
q15	571	518	527	518
q16	443	380	382	380
q17	1035	723	579	579
q18	7143	6894	6869	6869
q19	1633	1630	1648	1630
q20	538	323	317	317
q21	3223	2714	2803	2714
q22	363	298	310	298
Total cold run time: 44861 ms
Total hot run time: 40360 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4410	4404	4395	4395
q2	272	168	167	167
q3	3505	3506	3494	3494
q4	2445	2418	2423	2418
q5	5738	5732	5721	5721
q6	247	127	125	125
q7	2384	1880	1862	1862
q8	3612	3621	3616	3616
q9	9075	9007	8976	8976
q10	3924	4004	4025	4004
q11	478	373	355	355
q12	778	599	625	599
q13	4276	3562	3555	3555
q14	285	259	267	259
q15	569	516	517	516
q16	495	451	468	451
q17	1976	1951	1944	1944
q18	8688	8171	8315	8171
q19	1860	1847	1841	1841
q20	2258	1940	1941	1940
q21	6633	6244	6240	6240
q22	537	455	446	446
Total cold run time: 64445 ms
Total hot run time: 61095 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.42 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 20.8 seconds inserted 10000000 Rows, about 480K ops/s
storage size: 17183760429 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8584/23448)
Line Coverage: 28.67% (69771/243357)
Region Coverage: 27.65% (36066/130455)
Branch Coverage: 24.36% (18424/75618)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f71f588b92b7415bb97d81edc648bde9916c3618_f71f588b92b7415bb97d81edc648bde9916c3618/report/index.html

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.60% (8584/23454)
Line Coverage: 28.66% (69775/243448)
Region Coverage: 27.64% (36071/130510)
Branch Coverage: 24.36% (18425/75644)
Coverage Report: http://coverage.selectdb-in.cc/coverage/162a32e7ad16aaf4ef9e00b27bae6db6f707d507_162a32e7ad16aaf4ef9e00b27bae6db6f707d507/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.42 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 20.5 seconds inserted 10000000 Rows, about 487K ops/s
storage size: 17184002085 Bytes

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.59% (8582/23454)
Line Coverage: 28.65% (69747/243449)
Region Coverage: 27.63% (36054/130510)
Branch Coverage: 24.34% (18412/75644)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c2538095440f706b65661909e8d43e299cc3a72_3c2538095440f706b65661909e8d43e299cc3a72/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 162a32e7ad16aaf4ef9e00b27bae6db6f707d507, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4983	4628	4608	4608
q2	362	159	154	154
q3	1476	1262	1188	1188
q4	1137	930	897	897
q5	3160	3165	3167	3165
q6	254	126	124	124
q7	1022	494	505	494
q8	2280	2242	2227	2227
q9	6704	6693	6675	6675
q10	3196	3281	3266	3266
q11	326	213	197	197
q12	348	198	212	198
q13	4153	3448	3439	3439
q14	241	208	221	208
q15	567	521	528	521
q16	447	381	382	381
q17	1041	757	528	528
q18	7097	6738	6949	6738
q19	1618	1630	1645	1630
q20	570	323	319	319
q21	3107	2719	2587	2587
q22	357	302	300	300
Total cold run time: 44446 ms
Total hot run time: 39844 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4570	4571	4558	4558
q2	268	166	168	166
q3	3371	3371	3360	3360
q4	2205	2193	2208	2193
q5	5724	5713	5708	5708
q6	243	117	120	117
q7	2370	1841	1881	1841
q8	3599	3629	3597	3597
q9	9013	8961	8982	8961
q10	3813	3892	3901	3892
q11	492	371	378	371
q12	756	606	602	602
q13	3903	3175	3161	3161
q14	277	256	254	254
q15	569	525	516	516
q16	493	454	498	454
q17	1958	1968	1954	1954
q18	8574	8200	8290	8200
q19	1755	1760	1736	1736
q20	2250	1950	1929	1929
q21	6164	5777	5742	5742
q22	536	463	432	432
Total cold run time: 62903 ms
Total hot run time: 59744 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.4 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 20.7 seconds inserted 10000000 Rows, about 483K ops/s
storage size: 17183835097 Bytes

@dataroaring
Copy link
Contributor

run p0 10

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 3c2538095440f706b65661909e8d43e299cc3a72, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5003	4658	4670	4658
q2	364	149	158	149
q3	1462	1283	1225	1225
q4	1124	943	905	905
q5	3175	3166	3127	3127
q6	254	129	128	128
q7	1022	489	501	489
q8	2274	2263	2244	2244
q9	6713	6689	6684	6684
q10	3199	3253	3272	3253
q11	329	219	211	211
q12	352	206	212	206
q13	4170	3439	3421	3421
q14	241	214	218	214
q15	573	528	527	527
q16	444	387	382	382
q17	1050	758	634	634
q18	7090	6759	6791	6759
q19	1625	1637	1650	1637
q20	551	315	315	315
q21	3227	2739	2763	2739
q22	373	300	309	300
Total cold run time: 44615 ms
Total hot run time: 40207 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4652	4566	4566	4566
q2	268	169	171	169
q3	3369	3366	3358	3358
q4	2241	2216	2212	2212
q5	5738	5721	5769	5721
q6	242	121	119	119
q7	2394	1870	1857	1857
q8	3629	3626	3609	3609
q9	9042	8990	8928	8928
q10	3791	3893	3890	3890
q11	490	361	364	361
q12	780	608	620	608
q13	4011	3199	3165	3165
q14	286	254	242	242
q15	569	517	518	517
q16	506	440	454	440
q17	1983	1946	1948	1946
q18	8689	8121	8282	8121
q19	1753	1766	1755	1755
q20	2244	1934	1913	1913
q21	6162	5796	5802	5796
q22	532	474	463	463
Total cold run time: 63371 ms
Total hot run time: 59756 ms

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8584/23449)
Line Coverage: 28.66% (69769/243441)
Region Coverage: 27.64% (36066/130507)
Branch Coverage: 24.36% (18424/75646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b4dadd3b9e545ebdfc86b64eb3ad971b9f280f32_b4dadd3b9e545ebdfc86b64eb3ad971b9f280f32/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.83 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 20.8 seconds inserted 10000000 Rows, about 480K ops/s
storage size: 17184057060 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.72% (8657/23573)
Line Coverage: 28.77% (70700/245765)
Region Coverage: 27.65% (36468/131900)
Branch Coverage: 24.36% (18658/76584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/37f4a251431f433b3de4eec4efd775bed0715a77_37f4a251431f433b3de4eec4efd775bed0715a77/report/index.html

@dataroaring
Copy link
Contributor

run p0 10

1 similar comment
@dataroaring
Copy link
Contributor

run p0 10

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39056 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c4a9034bc5e3fa63153a1b0045d96b42c29e623, data reload: false

------ Round 1 ----------------------------------
q1	17594	5439	5455	5439
q2	2047	141	132	132
q3	10828	1205	1163	1163
q4	10464	742	820	742
q5	7757	3169	3198	3169
q6	199	122	124	122
q7	866	494	488	488
q8	9221	1934	1961	1934
q9	7298	6421	6357	6357
q10	8219	3061	3005	3005
q11	422	211	208	208
q12	360	190	193	190
q13	17999	3379	3591	3379
q14	266	230	233	230
q15	557	517	506	506
q16	484	373	393	373
q17	969	522	508	508
q18	7740	6957	6838	6838
q19	3312	1320	1420	1320
q20	562	320	290	290
q21	2803	2380	2360	2360
q22	372	312	303	303
Total cold run time: 110339 ms
Total hot run time: 39056 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5540	5278	5277	5277
q2	325	222	210	210
q3	3297	3220	3185	3185
q4	2105	2065	2001	2001
q5	6045	5940	6022	5940
q6	204	115	119	115
q7	2260	1898	1842	1842
q8	3229	3376	3409	3376
q9	8901	8749	8855	8749
q10	3927	3891	3842	3842
q11	550	455	442	442
q12	825	632	628	628
q13	16909	3160	3158	3158
q14	276	246	263	246
q15	553	498	501	498
q16	525	472	465	465
q17	1909	1866	1865	1865
q18	9714	17693	9621	9621
q19	25435	1570	1534	1534
q20	4615	1932	1919	1919
q21	14396	5303	5298	5298
q22	978	543	545	543
Total cold run time: 112518 ms
Total hot run time: 60754 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187032 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c4a9034bc5e3fa63153a1b0045d96b42c29e623, data reload: false

query1	918	343	320	320
query2	6580	1988	1829	1829
query3	6702	201	202	201
query4	30256	22254	22355	22254
query5	4457	454	441	441
query6	241	155	151	151
query7	4598	263	263	263
query8	222	172	183	172
query9	8300	2545	2538	2538
query10	424	240	233	233
query11	16603	15616	15700	15616
query12	120	74	70	70
query13	1696	376	368	368
query14	10563	7131	6917	6917
query15	213	185	181	181
query16	5784	266	256	256
query17	935	478	475	475
query18	1793	257	254	254
query19	174	133	136	133
query20	78	71	62	62
query21	192	130	134	130
query22	5185	4982	4995	4982
query23	31688	30901	30809	30809
query24	12525	2831	2807	2807
query25	571	312	310	310
query26	1788	148	148	148
query27	3213	280	290	280
query28	7116	1820	1822	1820
query29	1951	638	640	638
query30	279	135	138	135
query31	944	743	759	743
query32	83	52	50	50
query33	698	214	212	212
query34	1133	452	470	452
query35	935	788	799	788
query36	1301	1188	1221	1188
query37	91	60	60	60
query38	3350	3217	3235	3217
query39	1312	1275	1259	1259
query40	346	92	87	87
query41	36	35	34	34
query42	90	83	85	83
query43	509	430	477	430
query44	1074	693	705	693
query45	188	182	176	176
query46	1070	651	660	651
query47	1657	1535	1459	1459
query48	399	318	300	300
query49	1195	284	292	284
query50	686	327	309	309
query51	5421	5198	5170	5170
query52	83	87	75	75
query53	325	264	261	261
query54	238	182	200	182
query55	88	75	74	74
query56	186	166	172	166
query57	990	932	907	907
query58	190	169	163	163
query59	2864	2644	2591	2591
query60	217	191	187	187
query61	85	84	80	80
query62	612	359	376	359
query63	276	271	253	253
query64	6135	1795	1751	1751
query65	3341	3258	3227	3227
query66	1365	321	313	313
query67	15915	15377	15124	15124
query68	10063	531	529	529
query69	623	301	292	292
query70	1690	1561	1455	1455
query71	10430	10200	10197	10197
query72	4653	2846	2848	2846
query73	1916	322	319	319
query74	6950	6444	6430	6430
query75	4608	2338	2288	2288
query76	6476	996	954	954
query77	640	230	233	230
query78	9009	8871	8695	8695
query79	1000	504	510	504
query80	626	332	320	320
query81	443	205	198	198
query82	173	78	80	78
query83	159	117	118	117
query84	272	68	68	68
query85	1050	334	333	333
query86	393	399	386	386
query87	3493	3414	3339	3339
query88	2703	2220	2214	2214
query89	417	357	355	355
query90	1998	190	185	185
query91	154	121	131	121
query92	52	43	44	43
query93	923	432	441	432
query94	1149	160	162	160
query95	495	451	448	448
query96	613	319	316	316
query97	4271	4150	4142	4142
query98	198	187	188	187
query99	1021	708	700	700
Total cold run time: 299254 ms
Total hot run time: 187032 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c4a9034bc5e3fa63153a1b0045d96b42c29e623, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.03
query3	0.22	0.05	0.05
query4	1.70	0.06	0.07
query5	0.54	0.53	0.52
query6	1.35	0.63	0.64
query7	0.02	0.02	0.01
query8	0.04	0.02	0.03
query9	0.54	0.49	0.50
query10	0.55	0.56	0.56
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.60	0.59	0.62
query14	0.78	0.81	0.80
query15	0.79	0.77	0.78
query16	0.40	0.39	0.39
query17	1.04	1.01	1.02
query18	0.22	0.25	0.26
query19	1.87	1.78	1.84
query20	0.01	0.01	0.01
query21	15.40	0.58	0.57
query22	2.50	2.75	1.87
query23	17.80	0.88	0.77
query24	2.62	1.12	0.90
query25	0.43	0.13	0.12
query26	0.59	0.15	0.14
query27	0.06	0.07	0.04
query28	12.29	0.77	0.76
query29	12.86	3.09	3.17
query30	0.54	0.49	0.54
query31	2.78	0.34	0.35
query32	3.38	0.49	0.48
query33	3.22	3.19	3.21
query34	15.87	4.35	4.29
query35	4.28	4.27	4.29
query36	1.10	1.08	1.08
query37	0.06	0.04	0.04
query38	0.03	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.06	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 107.1 s
Total hot run time: 30.82 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3c4a9034bc5e3fa63153a1b0045d96b42c29e623 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@kaijchen
Copy link
Contributor Author

run p0 10

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38869 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6f97cb4a6abeb9394aa736c5a86cb4ef9bf83d0, data reload: false

------ Round 1 ----------------------------------
q1	17610	5260	5405	5260
q2	2039	139	134	134
q3	10634	1154	1170	1154
q4	10256	771	829	771
q5	7725	3198	3133	3133
q6	198	119	116	116
q7	868	502	496	496
q8	9188	1952	1963	1952
q9	7216	6401	6404	6401
q10	8208	3090	3033	3033
q11	423	202	200	200
q12	346	188	184	184
q13	17976	3351	3341	3341
q14	249	216	219	216
q15	545	509	520	509
q16	463	370	391	370
q17	961	520	526	520
q18	7341	6909	6739	6739
q19	1538	1318	1402	1318
q20	580	316	302	302
q21	2798	2419	2422	2419
q22	364	316	301	301
Total cold run time: 107526 ms
Total hot run time: 38869 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5547	5127	5323	5127
q2	328	217	211	211
q3	3316	3195	3245	3195
q4	2075	2016	2031	2016
q5	6029	5893	5858	5858
q6	196	112	115	112
q7	2334	1803	1885	1803
q8	3228	3399	3398	3398
q9	8964	8725	8858	8725
q10	3882	3851	3831	3831
q11	547	449	473	449
q12	810	621	610	610
q13	16955	3146	3158	3146
q14	283	249	274	249
q15	547	495	507	495
q16	523	461	454	454
q17	1858	1864	1855	1855
q18	9562	17360	9616	9616
q19	23931	1589	1528	1528
q20	4618	1956	1905	1905
q21	14386	5452	5440	5440
q22	971	537	533	533
Total cold run time: 110890 ms
Total hot run time: 60556 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6f97cb4a6abeb9394aa736c5a86cb4ef9bf83d0, data reload: false

query1	934	330	321	321
query2	6558	1931	1876	1876
query3	6699	199	198	198
query4	31573	22207	22150	22150
query5	4450	376	369	369
query6	241	168	153	153
query7	4598	252	262	252
query8	217	191	170	170
query9	8438	2592	2565	2565
query10	405	224	209	209
query11	16714	15532	15371	15371
query12	114	69	66	66
query13	1673	367	370	367
query14	10426	6841	6829	6829
query15	214	180	177	177
query16	5790	253	257	253
query17	948	471	474	471
query18	1808	258	244	244
query19	168	129	125	125
query20	74	70	68	68
query21	181	130	129	129
query22	4976	4859	4777	4777
query23	31635	30632	30874	30632
query24	12714	2838	2742	2742
query25	562	322	303	303
query26	1834	136	140	136
query27	3200	282	282	282
query28	6988	1832	1821	1821
query29	2147	610	625	610
query30	280	135	136	135
query31	964	726	751	726
query32	71	50	45	45
query33	707	204	198	198
query34	1173	447	459	447
query35	877	755	763	755
query36	1274	1157	1197	1157
query37	126	59	58	58
query38	3390	3223	3255	3223
query39	1320	1259	1254	1254
query40	341	82	84	82
query41	36	35	36	35
query42	97	80	81	80
query43	544	496	509	496
query44	1063	688	680	680
query45	192	178	170	170
query46	1083	643	664	643
query47	1630	1553	1614	1553
query48	385	306	294	294
query49	1221	275	276	275
query50	689	306	301	301
query51	5339	5264	5160	5160
query52	78	75	82	75
query53	331	254	262	254
query54	234	180	180	180
query55	86	78	81	78
query56	164	161	158	158
query57	1055	901	921	901
query58	192	155	149	149
query59	2978	2655	2779	2655
query60	195	177	179	177
query61	78	81	78	78
query62	611	354	366	354
query63	287	258	261	258
query64	6138	1762	1750	1750
query65	3298	3241	3232	3232
query66	1443	309	306	306
query67	15698	15202	15281	15202
query68	11270	534	508	508
query69	615	288	288	288
query70	1720	1525	1487	1487
query71	10479	10195	10195	10195
query72	4810	2813	2805	2805
query73	2130	301	311	301
query74	7033	6455	6367	6367
query75	4827	2273	2293	2273
query76	6731	1041	1002	1002
query77	759	224	227	224
query78	9646	8974	8670	8670
query79	1003	502	485	485
query80	580	316	302	302
query81	445	198	201	198
query82	196	79	77	77
query83	134	114	116	114
query84	271	66	64	64
query85	1073	326	318	318
query86	395	397	379	379
query87	3536	3388	3322	3322
query88	3023	2182	2182	2182
query89	417	361	348	348
query90	1979	186	181	181
query91	161	120	129	120
query92	51	44	42	42
query93	1745	449	441	441
query94	1204	155	155	155
query95	491	445	445	445
query96	618	322	316	316
query97	4262	4122	4141	4122
query98	209	191	179	179
query99	1042	717	711	711
Total cold run time: 304802 ms
Total hot run time: 185861 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6f97cb4a6abeb9394aa736c5a86cb4ef9bf83d0, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.02	0.03
query3	0.23	0.04	0.04
query4	1.72	0.06	0.05
query5	0.54	0.52	0.53
query6	1.27	0.64	0.65
query7	0.02	0.02	0.01
query8	0.04	0.02	0.02
query9	0.54	0.50	0.50
query10	0.56	0.57	0.56
query11	0.12	0.09	0.08
query12	0.11	0.09	0.08
query13	0.62	0.61	0.61
query14	0.77	0.82	0.80
query15	0.79	0.77	0.76
query16	0.39	0.38	0.37
query17	1.03	1.03	1.03
query18	0.21	0.26	0.25
query19	1.90	1.79	1.78
query20	0.01	0.00	0.00
query21	15.40	0.57	0.56
query22	2.71	3.30	2.12
query23	17.06	0.84	0.90
query24	2.80	0.62	0.65
query25	0.40	0.15	0.24
query26	0.43	0.13	0.12
query27	0.06	0.06	0.05
query28	11.92	0.77	0.75
query29	12.51	3.16	3.27
query30	0.54	0.54	0.51
query31	2.78	0.33	0.34
query32	3.39	0.49	0.49
query33	3.23	3.20	3.30
query34	15.67	4.31	4.28
query35	4.32	4.20	4.31
query36	1.12	1.06	1.08
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.15	0.12	0.12
query41	0.06	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.68 s
Total hot run time: 30.84 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f6f97cb4a6abeb9394aa736c5a86cb4ef9bf83d0 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.1 seconds inserted 10000000 Rows, about 763K ops/s

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38650 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd1d1f34b07786d7b1a8fcdaee6d3e1033e8dee4, data reload: false

------ Round 1 ----------------------------------
q1	17663	5246	5403	5246
q2	2053	147	132	132
q3	10803	1131	1192	1131
q4	10428	788	816	788
q5	7744	3093	3112	3093
q6	197	128	127	127
q7	878	507	491	491
q8	9266	1978	1944	1944
q9	7237	6437	6323	6323
q10	8181	3011	3072	3011
q11	410	211	198	198
q12	359	186	187	186
q13	18005	3372	3330	3330
q14	240	223	208	208
q15	542	506	509	506
q16	435	380	379	379
q17	942	602	493	493
q18	7655	7138	6796	6796
q19	4517	1390	1309	1309
q20	620	319	289	289
q21	2755	2370	2444	2370
q22	363	303	300	300
Total cold run time: 111293 ms
Total hot run time: 38650 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5568	5214	5305	5214
q2	323	219	220	219
q3	3355	3248	3282	3248
q4	2086	2042	2067	2042
q5	5892	5942	5926	5926
q6	196	115	119	115
q7	2294	1929	1913	1913
q8	3226	3377	3393	3377
q9	8925	8933	8755	8755
q10	3918	3817	3856	3817
q11	552	453	456	453
q12	821	632	631	631
q13	16938	3188	3162	3162
q14	278	253	252	252
q15	551	492	506	492
q16	516	475	463	463
q17	1826	1872	1842	1842
q18	9389	17404	9867	9867
q19	26849	1602	1500	1500
q20	4579	1945	1914	1914
q21	14437	5310	5281	5281
q22	975	551	590	551
Total cold run time: 113494 ms
Total hot run time: 61034 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186374 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd1d1f34b07786d7b1a8fcdaee6d3e1033e8dee4, data reload: false

query1	935	328	325	325
query2	6573	2015	2042	2015
query3	6702	205	202	202
query4	32817	22103	22261	22103
query5	4448	361	438	361
query6	246	153	154	153
query7	4601	265	262	262
query8	219	175	176	175
query9	8370	2631	2598	2598
query10	421	221	230	221
query11	16594	15469	15544	15469
query12	119	67	64	64
query13	1679	378	366	366
query14	10545	7042	6943	6943
query15	206	180	187	180
query16	5790	262	250	250
query17	946	482	468	468
query18	1783	254	249	249
query19	169	142	124	124
query20	67	67	70	67
query21	186	125	128	125
query22	5002	4796	4827	4796
query23	31663	30806	30742	30742
query24	12571	2788	2879	2788
query25	567	304	308	304
query26	1784	138	149	138
query27	3213	284	285	284
query28	7196	1833	1812	1812
query29	1876	629	619	619
query30	283	135	138	135
query31	928	745	774	745
query32	73	52	48	48
query33	710	208	213	208
query34	1136	456	473	456
query35	868	766	792	766
query36	1355	1188	1220	1188
query37	93	61	57	57
query38	3367	3245	3303	3245
query39	1336	1264	1263	1263
query40	339	91	81	81
query41	38	35	35	35
query42	86	81	85	81
query43	508	492	476	476
query44	1102	688	695	688
query45	190	175	176	175
query46	1068	662	650	650
query47	1661	1538	1627	1538
query48	398	318	314	314
query49	1195	300	282	282
query50	666	309	314	309
query51	5318	5194	5209	5194
query52	84	81	70	70
query53	332	254	255	254
query54	243	181	183	181
query55	77	73	75	73
query56	172	178	168	168
query57	1041	918	928	918
query58	181	150	153	150
query59	2985	2877	2721	2721
query60	210	184	181	181
query61	80	89	83	83
query62	616	357	353	353
query63	282	252	262	252
query64	6115	1800	1751	1751
query65	3349	3307	3274	3274
query66	1316	324	313	313
query67	15706	14858	14970	14858
query68	16009	526	511	511
query69	613	292	291	291
query70	2273	1542	1473	1473
query71	10480	10203	10195	10195
query72	5160	2822	2838	2822
query73	4144	311	312	311
query74	7800	6459	6425	6425
query75	5254	2284	2267	2267
query76	6731	1014	1003	1003
query77	743	229	227	227
query78	9715	8675	8547	8547
query79	2625	491	505	491
query80	581	327	309	309
query81	449	197	202	197
query82	202	80	83	80
query83	257	118	119	118
query84	276	67	67	67
query85	1105	332	330	330
query86	396	412	397	397
query87	3563	3311	3318	3311
query88	3140	2189	2174	2174
query89	438	373	362	362
query90	2163	186	187	186
query91	168	120	129	120
query92	55	45	42	42
query93	1535	423	428	423
query94	1356	160	156	156
query95	505	468	454	454
query96	611	321	321	321
query97	4281	4147	4149	4147
query98	208	196	191	191
query99	1057	689	730	689
Total cold run time: 316688 ms
Total hot run time: 186374 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fd1d1f34b07786d7b1a8fcdaee6d3e1033e8dee4, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.05
query4	1.70	0.06	0.08
query5	0.54	0.52	0.53
query6	1.22	0.63	0.62
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.55	0.52	0.50
query10	0.56	0.57	0.56
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.61	0.60	0.61
query14	0.79	0.81	0.82
query15	0.79	0.77	0.78
query16	0.39	0.36	0.39
query17	1.03	1.01	1.03
query18	0.23	0.25	0.26
query19	1.87	1.76	1.81
query20	0.02	0.01	0.01
query21	15.41	0.61	0.58
query22	2.47	2.60	1.22
query23	17.48	0.78	0.79
query24	2.58	0.37	1.30
query25	0.29	0.21	0.15
query26	0.54	0.14	0.14
query27	0.05	0.06	0.06
query28	11.79	0.77	0.78
query29	12.51	3.19	3.03
query30	0.55	0.49	0.49
query31	2.78	0.34	0.35
query32	3.38	0.48	0.49
query33	3.22	3.22	3.23
query34	15.58	4.31	4.32
query35	4.34	4.36	4.26
query36	1.11	1.08	1.07
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.39 s
Total hot run time: 29.68 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit fd1d1f34b07786d7b1a8fcdaee6d3e1033e8dee4 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees
No one assigned
Labels
meta-change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kaijchen @dataroaring @doris-robot