Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix](scannerscheduler) should minus num_of_scanners before check should schedule #28926 #29331

Merged
merged 2 commits into from
Jan 3, 2024

Conversation

yiguolei
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@yiguolei
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -318,8 +322,8 @@ Status VScanNode::_init_profile() {
return Status::OK();
}

Status VScanNode::_start_scanners(const std::list<VScannerSPtr>& scanners,
const int query_parallel_instance_num) {
void VScanNode::_start_scanners(const std::list<std::shared_ptr<ScannerDelegate>>& scanners,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method '_start_scanners' can be made const [readability-make-member-function-const]

be/src/vec/exec/scan/vscan_node.cpp:325:

-                                 const int query_parallel_instance_num) {
+                                 const int query_parallel_instance_num) const {

public:
VScannerSPtr _scanner;
ScannerDelegate(VScannerSPtr& scanner_ptr) : _scanner(scanner_ptr) {}
~ScannerDelegate() { static_cast<void>(_scanner->close(_scanner->runtime_state())); }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: use '= default' to define a trivial destructor [modernize-use-equals-default]

    ~ScannerDelegate() { static_cast<void>(_scanner->close(_scanner->runtime_state())); }
    ^

Status _start_scanners(const std::list<VScannerSPtr>& scanners,
const int query_parallel_instance_num);
void _start_scanners(const std::list<std::shared_ptr<ScannerDelegate>>& scanners,
const int query_parallel_instance_num);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: parameter 'query_parallel_instance_num' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change
const int query_parallel_instance_num);
int query_parallel_instance_num);

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit daa8f7e189660a5d04ff1b5117487720a652b202, data reload: false

------ Round 1 ----------------------------------
q1	17676	5268	5156	5156
q2	2030	160	151	151
q3	10531	1128	1127	1127
q4	10173	764	834	764
q5	7786	2995	2968	2968
q6	227	142	139	139
q7	954	575	504	504
q8	9269	2044	2071	2044
q9	6865	6428	6388	6388
q10	8256	3094	3109	3094
q11	427	237	227	227
q12	406	238	238	238
q13	18008	3621	3636	3621
q14	243	217	217	217
q15	573	541	521	521
q16	473	421	387	387
q17	969	570	497	497
q18	7396	6743	6785	6743
q19	1593	1321	1370	1321
q20	750	339	344	339
q21	2885	2353	2474	2353
q22	374	335	338	335
Total cold run time: 107864 ms
Total hot run time: 39134 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5100	5185	5129	5129
q2	349	239	241	239
q3	3317	3314	3271	3271
q4	2127	2063	2039	2039
q5	5804	5769	5821	5769
q6	224	129	133	129
q7	2335	1895	1902	1895
q8	3395	3472	3491	3472
q9	8859	8761	8730	8730
q10	3822	3848	3847	3847
q11	590	489	504	489
q12	808	649	645	645
q13	6389	3201	3186	3186
q14	307	276	284	276
q15	582	535	523	523
q16	562	501	509	501
q17	1958	1778	1757	1757
q18	8768	8362	8321	8321
q19	1638	1582	1627	1582
q20	2218	1976	1940	1940
q21	5653	5280	5251	5251
q22	519	470	474	470
Total cold run time: 65324 ms
Total hot run time: 59461 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit daa8f7e189660a5d04ff1b5117487720a652b202, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5457	5136	5130	5130
q2	386	169	167	167
q3	1471	1226	1219	1219
q4	1087	845	805	805
q5	3288	3106	3127	3106
q6	224	141	143	141
q7	988	563	530	530
q8	2159	2264	2203	2203
q9	6720	6639	6699	6639
q10	3201	3161	3143	3143
q11	359	236	230	230
q12	391	247	243	243
q13	4403	3646	3614	3614
q14	258	221	225	221
q15	596	542	535	535
q16	459	403	403	403
q17	1062	534	573	534
q18	7091	6790	6808	6790
q19	1648	1543	1608	1543
q20	627	351	336	336
q21	2856	2490	2533	2490
q22	404	335	342	335
Total cold run time: 45135 ms
Total hot run time: 40357 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5115	5074	5126	5074
q2	339	250	250	250
q3	3364	3332	3324	3324
q4	2147	2082	2067	2067
q5	5946	5901	5925	5901
q6	232	133	140	133
q7	2379	1965	1929	1929
q8	3551	3633	3681	3633
q9	9053	8977	9037	8977
q10	3909	3943	3929	3929
q11	579	464	477	464
q12	821	655	650	650
q13	3878	3168	3195	3168
q14	308	281	272	272
q15	600	537	537	537
q16	555	503	526	503
q17	2028	1819	1824	1819
q18	8775	8354	8352	8352
q19	1760	1722	1719	1719
q20	2294	1978	1976	1976
q21	5652	5336	5379	5336
q22	538	475	465	465
Total cold run time: 63823 ms
Total hot run time: 60478 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit daa8f7e189660a5d04ff1b5117487720a652b202, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	926	365	335	335
query2	6431	2005	1955	1955
query3	6647	226	221	221
query4	27066	22386	22397	22386
query5	4338	590	618	590
query6	273	181	178	178
query7	4588	274	264	264
query8	227	197	187	187
query9	8314	2740	2763	2740
query10	428	288	241	241
query11	16152	15525	15514	15514
query12	142	78	73	73
query13	1647	342	324	324
query14	11572	7254	7247	7247
query15	224	185	194	185
query16	6473	295	292	292
query17	1798	500	506	500
query18	1939	267	256	256
query19	187	134	137	134
query20	81	72	78	72
query21	187	99	97	97
query22	4983	4753	4764	4753
query23	31988	31182	31369	31182
query24	12140	2807	2830	2807
query25	601	349	347	347
query26	1764	143	141	141
query27	2909	277	298	277
query28	7182	1970	1966	1966
query29	2067	414	409	409
query30	282	142	145	142
query31	1056	775	759	759
query32	92	60	59	59
query33	739	266	267	266
query34	911	449	460	449
query35	886	787	782	782
query36	1297	1233	1246	1233
query37	192	88	84	84
query38	3413	3287	3270	3270
query39	1337	1274	1275	1274
query40	295	92	89	89
query41	36	36	35	35
query42	94	90	87	87
query43	526	489	481	481
query44	1106	758	762	758
query45	194	190	181	181
query46	1088	654	670	654
query47	1595	1535	1566	1535
query48	355	265	269	265
query49	1224	321	319	319
query50	766	349	350	349
query51	5367	5229	5364	5229
query52	92	85	78	78
query53	213	157	148	148
query54	1401	597	601	597
query55	104	91	78	78
query56	207	199	198	198
query57	1011	949	946	946
query58	211	197	203	197
query59	2789	2540	2566	2540
query60	247	203	207	203
query61	85	87	88	87
query62	647	451	471	451
query63	157	149	154	149
query64	5937	1765	1726	1726
query65	3335	3255	3283	3255
query66	1384	339	333	333
query67	15508	15122	15794	15122
query68	11030	522	525	522
query69	600	240	263	240
query70	1590	1531	1532	1531
query71	563	207	211	207
query72	5565	3569	3553	3553
query73	1475	325	327	325
query74	6968	6464	6463	6463
query75	5626	2279	2330	2279
query76	5588	1120	1031	1031
query77	874	253	259	253
query78	9087	8572	8582	8572
query79	3871	503	497	497
query80	2392	350	353	350
query81	514	205	207	205
query82	250	111	118	111
query83	309	147	139	139
query84	305	63	56	56
query85	2149	284	274	274
query86	457	399	396	396
query87	3582	3418	3361	3361
query88	3079	2421	2417	2417
query89	344	257	272	257
query90	1932	244	244	244
query91	123	92	95	92
query92	67	55	54	54
query93	1750	504	440	440
query94	660	214	215	214
query95	508	484	461	461
query96	664	340	331	331
query97	4299	4183	4184	4183
query98	222	201	191	191
query99	1144	819	839	819
Total cold run time: 297040 ms
Total hot run time: 180149 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.38 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 28 seconds loaded 2358488459 Bytes, about 80 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17183776994 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.64% (8609/23494)
Line Coverage: 28.71% (70018/243918)
Region Coverage: 27.70% (36235/130815)
Branch Coverage: 24.40% (18518/75884)
Coverage Report: http://coverage.selectdb-in.cc/coverage/daa8f7e189660a5d04ff1b5117487720a652b202_daa8f7e189660a5d04ff1b5117487720a652b202/report/index.html

@yiguolei
Copy link
Contributor Author

run p0 10

1 similar comment
@yiguolei
Copy link
Contributor Author

run p0 10

@yiguolei yiguolei force-pushed the fix_core1 branch 2 times, most recently from 8dc606c to 5810b1f Compare December 30, 2023 08:43
@yiguolei
Copy link
Contributor Author

run compile

@yiguolei
Copy link
Contributor Author

run buildall

@yiguolei
Copy link
Contributor Author

run p0 10

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5810b1f92f2659188e2370677ef62a259d98f8d4, data reload: false

------ Round 1 ----------------------------------
q1	17660	5313	5080	5080
q2	2023	160	146	146
q3	10532	1085	1075	1075
q4	10194	769	831	769
q5	7779	2976	2836	2836
q6	218	133	132	132
q7	937	570	486	486
q8	9297	2038	2046	2038
q9	6839	6409	6413	6409
q10	8280	2998	3018	2998
q11	413	229	221	221
q12	392	233	231	231
q13	17991	3622	3629	3622
q14	244	207	212	207
q15	560	528	525	525
q16	447	409	404	404
q17	967	530	454	454
q18	7386	6718	6634	6634
q19	1577	1409	1419	1409
q20	710	354	339	339
q21	2841	2418	2382	2382
q22	395	331	340	331
Total cold run time: 107682 ms
Total hot run time: 38728 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5133	5140	5028	5028
q2	342	232	232	232
q3	3305	3243	3261	3243
q4	2138	2023	2017	2017
q5	5784	5794	5969	5794
q6	221	128	127	127
q7	2346	1900	1900	1900
q8	3399	3450	3485	3450
q9	8848	8723	8727	8723
q10	3787	3833	3873	3833
q11	610	482	488	482
q12	789	636	636	636
q13	7979	3204	3195	3195
q14	287	273	287	273
q15	564	520	513	513
q16	551	515	509	509
q17	1975	1732	1775	1732
q18	8690	8430	8254	8254
q19	1633	1625	1603	1603
q20	2176	1958	1954	1954
q21	5598	5227	5328	5227
q22	525	477	490	477
Total cold run time: 66680 ms
Total hot run time: 59202 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5810b1f92f2659188e2370677ef62a259d98f8d4, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5474	5150	5144	5144
q2	380	173	169	169
q3	1462	1202	1156	1156
q4	1076	805	803	803
q5	3126	3123	3098	3098
q6	221	133	140	133
q7	985	556	532	532
q8	2138	2246	2259	2246
q9	6686	6693	6640	6640
q10	3150	3115	3118	3115
q11	331	216	206	206
q12	378	238	240	238
q13	4376	3611	3628	3611
q14	252	214	216	214
q15	589	542	527	527
q16	457	397	404	397
q17	1043	546	448	448
q18	7094	6767	6702	6702
q19	1637	1486	1588	1486
q20	541	338	334	334
q21	2869	2470	2445	2445
q22	398	311	337	311
Total cold run time: 44663 ms
Total hot run time: 39955 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5137	5184	5049	5049
q2	339	242	237	237
q3	3375	3322	3309	3309
q4	2171	2017	2029	2017
q5	5918	5900	5896	5896
q6	221	128	133	128
q7	2376	1921	1901	1901
q8	3524	3630	3654	3630
q9	9033	8997	8975	8975
q10	3888	3914	3927	3914
q11	583	495	462	462
q12	797	632	639	632
q13	3878	3206	3155	3155
q14	306	261	281	261
q15	592	534	528	528
q16	550	539	555	539
q17	2018	1824	1785	1785
q18	8756	8313	8454	8313
q19	1735	1644	1663	1644
q20	2287	1986	1982	1982
q21	5747	5298	5281	5281
q22	571	487	499	487
Total cold run time: 63802 ms
Total hot run time: 60125 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 5810b1f92f2659188e2370677ef62a259d98f8d4, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	925	355	342	342
query2	6453	1899	1917	1899
query3	6652	208	205	205
query4	26744	22337	22347	22337
query5	3957	588	538	538
query6	279	196	176	176
query7	4585	287	268	268
query8	238	202	206	202
query9	8448	2813	2807	2807
query10	432	252	250	250
query11	16120	15645	15644	15644
query12	135	76	75	75
query13	1626	323	325	323
query14	11509	7144	7139	7139
query15	221	186	190	186
query16	6370	278	272	272
query17	1855	494	491	491
query18	1924	270	272	270
query19	192	135	143	135
query20	79	75	83	75
query21	190	98	100	98
query22	4886	4533	4816	4533
query23	31994	31139	31377	31139
query24	12110	2805	2806	2805
query25	623	355	349	349
query26	1781	151	148	148
query27	2913	281	287	281
query28	7106	1976	1974	1974
query29	2134	416	388	388
query30	292	151	149	149
query31	973	758	777	758
query32	84	62	58	58
query33	755	290	273	273
query34	881	459	447	447
query35	942	777	752	752
query36	1374	1201	1194	1194
query37	194	70	70	70
query38	3374	3266	3282	3266
query39	1311	1289	1279	1279
query40	305	98	93	93
query41	37	35	34	34
query42	102	77	88	77
query43	522	514	543	514
query44	1085	717	726	717
query45	202	193	183	183
query46	1060	656	649	649
query47	1662	1550	1498	1498
query48	335	264	267	264
query49	1217	335	325	325
query50	740	329	376	329
query51	5382	5309	5262	5262
query52	93	83	86	83
query53	215	149	151	149
query54	1439	600	582	582
query55	88	94	85	85
query56	222	203	203	203
query57	1043	929	867	867
query58	229	200	197	197
query59	2886	2655	2664	2655
query60	229	209	220	209
query61	91	89	88	88
query62	638	457	475	457
query63	182	151	145	145
query64	5989	1710	1704	1704
query65	3353	3273	3267	3267
query66	1400	342	343	342
query67	15566	15509	15401	15401
query68	12671	528	554	528
query69	511	249	248	248
query70	1669	1508	1513	1508
query71	537	219	226	219
query72	5665	3603	3622	3603
query73	2857	323	312	312
query74	6941	6391	6367	6367
query75	5360	2290	2281	2281
query76	6313	1118	1119	1118
query77	680	261	276	261
query78	9093	8730	8556	8556
query79	1049	519	518	518
query80	552	372	363	363
query81	460	209	208	208
query82	209	100	95	95
query83	163	145	144	144
query84	245	54	55	54
query85	997	298	286	286
query86	386	396	382	382
query87	3511	3411	3327	3327
query88	3156	2250	2256	2250
query89	338	273	258	258
query90	1927	201	206	201
query91	127	92	96	92
query92	60	52	59	52
query93	1497	509	443	443
query94	812	192	188	188
query95	461	418	419	418
query96	610	318	314	314
query97	4272	4161	4166	4161
query98	207	195	187	187
query99	1119	806	838	806
Total cold run time: 293428 ms
Total hot run time: 179722 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.45 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17183721619 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.65% (8614/23505)
Line Coverage: 28.70% (70035/244036)
Region Coverage: 27.68% (36242/130916)
Branch Coverage: 24.39% (18523/75940)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5810b1f92f2659188e2370677ef62a259d98f8d4_5810b1f92f2659188e2370677ef62a259d98f8d4/report/index.html

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.65% (8615/23506)
Line Coverage: 28.70% (70035/244040)
Region Coverage: 27.68% (36237/130916)
Branch Coverage: 24.39% (18522/75946)
Coverage Report: http://coverage.selectdb-in.cc/coverage/522057873ea52fde77e4b188cb62ad46f2088969_522057873ea52fde77e4b188cb62ad46f2088969/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 522057873ea52fde77e4b188cb62ad46f2088969, data reload: false

------ Round 1 ----------------------------------
q1	17684	5327	5131	5131
q2	2025	159	149	149
q3	10549	1093	1104	1093
q4	10185	823	870	823
q5	7809	2987	2867	2867
q6	214	135	135	135
q7	923	551	516	516
q8	9316	2039	2043	2039
q9	6851	6431	6429	6429
q10	8223	2987	3020	2987
q11	416	225	229	225
q12	389	228	235	228
q13	17995	3684	3628	3628
q14	250	217	205	205
q15	570	530	514	514
q16	457	403	430	403
q17	961	483	536	483
q18	7410	6814	6771	6771
q19	1574	1350	1324	1324
q20	698	342	326	326
q21	2784	2420	2451	2420
q22	388	337	336	336
Total cold run time: 107671 ms
Total hot run time: 39032 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5122	5031	5040	5031
q2	339	237	241	237
q3	3326	3306	3295	3295
q4	2120	2028	2049	2028
q5	5851	5808	5845	5808
q6	215	128	129	128
q7	2347	1972	1907	1907
q8	3385	3467	3490	3467
q9	8848	8842	8863	8842
q10	3797	3834	3852	3834
q11	583	501	489	489
q12	818	634	641	634
q13	7273	3219	3236	3219
q14	289	266	284	266
q15	584	517	526	517
q16	555	500	497	497
q17	1936	1808	1733	1733
q18	8758	8391	8434	8391
q19	1623	1594	1596	1594
q20	2197	1966	1960	1960
q21	5561	5372	5208	5208
q22	546	489	460	460
Total cold run time: 66073 ms
Total hot run time: 59545 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 522057873ea52fde77e4b188cb62ad46f2088969, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5517	5078	5101	5078
q2	389	163	169	163
q3	1471	1198	1204	1198
q4	1085	817	811	811
q5	3154	3124	3154	3124
q6	222	134	137	134
q7	967	548	533	533
q8	2137	2281	2215	2215
q9	6683	6700	6673	6673
q10	3187	3123	3108	3108
q11	356	217	219	217
q12	378	236	235	235
q13	4414	3664	3634	3634
q14	260	222	213	213
q15	582	537	535	535
q16	476	393	422	393
q17	1047	491	575	491
q18	7044	6779	6805	6779
q19	1628	1509	1586	1509
q20	607	372	343	343
q21	2860	2443	2456	2443
q22	388	308	325	308
Total cold run time: 44852 ms
Total hot run time: 40137 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5124	5077	5166	5077
q2	335	232	240	232
q3	3375	3353	3319	3319
q4	2145	2075	2043	2043
q5	5972	5945	5947	5945
q6	223	126	131	126
q7	2401	1909	1956	1909
q8	3565	3647	3668	3647
q9	9093	9032	9061	9032
q10	3845	3925	3904	3904
q11	575	487	487	487
q12	803	635	664	635
q13	3883	3161	3168	3161
q14	295	266	280	266
q15	591	532	527	527
q16	543	484	503	484
q17	2066	1863	1809	1809
q18	8721	8384	8332	8332
q19	1743	1661	1690	1661
q20	2275	1990	1971	1971
q21	5615	5275	5219	5219
q22	586	482	486	482
Total cold run time: 63774 ms
Total hot run time: 60268 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 522057873ea52fde77e4b188cb62ad46f2088969, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	944	356	339	339
query2	6418	1990	2007	1990
query3	6646	204	200	200
query4	26870	22572	22678	22572
query5	3906	566	533	533
query6	265	184	184	184
query7	4585	290	267	267
query8	227	208	200	200
query9	8345	2749	2684	2684
query10	519	240	241	240
query11	16278	15569	15503	15503
query12	136	79	80	79
query13	1637	330	324	324
query14	11372	7139	7184	7139
query15	220	188	192	188
query16	6414	284	278	278
query17	1837	503	508	503
query18	1962	277	273	273
query19	188	139	137	137
query20	78	81	77	77
query21	190	106	98	98
query22	4973	4558	4530	4530
query23	32091	31229	31270	31229
query24	12247	2873	2837	2837
query25	596	356	361	356
query26	1760	152	145	145
query27	2880	286	286	286
query28	7045	2002	1994	1994
query29	2098	403	397	397
query30	290	145	151	145
query31	974	763	766	763
query32	90	55	61	55
query33	747	259	279	259
query34	948	450	443	443
query35	913	796	796	796
query36	1317	1236	1226	1226
query37	189	70	75	70
query38	3421	3242	3329	3242
query39	1343	1304	1294	1294
query40	301	94	97	94
query41	38	38	34	34
query42	87	83	87	83
query43	522	490	499	490
query44	1114	713	723	713
query45	199	187	189	187
query46	1089	650	630	630
query47	1615	1483	1464	1464
query48	338	269	268	268
query49	1230	313	337	313
query50	808	339	331	331
query51	5318	5301	5217	5217
query52	105	82	89	82
query53	219	147	151	147
query54	1375	572	585	572
query55	103	86	76	76
query56	205	198	200	198
query57	1005	964	888	888
query58	230	203	204	203
query59	2875	2742	2711	2711
query60	224	212	222	212
query61	94	90	92	90
query62	632	448	447	447
query63	163	158	148	148
query64	5891	1758	1728	1728
query65	3378	3260	3262	3260
query66	1385	342	331	331
query67	15774	15227	15059	15059
query68	9678	530	553	530
query69	523	257	257	257
query70	1789	1535	1591	1535
query71	465	212	221	212
query72	5029	3615	3662	3615
query73	1856	318	312	312
query74	6907	6451	6429	6429
query75	4473	2274	2288	2274
query76	6084	1077	1021	1021
query77	657	280	283	280
query78	9109	8761	8541	8541
query79	1047	541	505	505
query80	706	361	365	361
query81	481	214	212	212
query82	228	99	96	96
query83	189	140	139	139
query84	247	60	55	55
query85	984	290	285	285
query86	397	408	379	379
query87	3579	3378	3361	3361
query88	2781	2275	2251	2251
query89	327	269	271	269
query90	1906	198	195	195
query91	121	91	96	91
query92	59	54	52	52
query93	1119	446	436	436
query94	780	189	182	182
query95	467	415	416	415
query96	615	317	327	317
query97	4296	4142	4138	4138
query98	238	206	194	194
query99	1083	804	823	804
Total cold run time: 287496 ms
Total hot run time: 179594 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.51 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17183450595 Bytes

@yiguolei
Copy link
Contributor Author

run p0 10

1 similar comment
@yiguolei
Copy link
Contributor Author

run p0 10

@yiguolei
Copy link
Contributor Author

yiguolei commented Jan 1, 2024

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit b7401deefbe6f8d00019119794d7df67e372c26c, data reload: false

------ Round 1 ----------------------------------
q1	17651	5157	5082	5082
q2	2017	159	143	143
q3	10550	1143	1109	1109
q4	10185	780	806	780
q5	7793	2928	2965	2928
q6	215	129	131	129
q7	933	565	501	501
q8	9338	2031	1955	1955
q9	6825	6430	6380	6380
q10	8254	3092	3043	3043
q11	421	223	206	206
q12	388	233	239	233
q13	17999	3644	3617	3617
q14	246	215	219	215
q15	559	525	520	520
q16	451	390	388	388
q17	965	485	510	485
q18	7320	6786	6703	6703
q19	1577	1359	1335	1335
q20	678	316	318	316
q21	2826	2372	2423	2372
q22	374	336	317	317
Total cold run time: 107565 ms
Total hot run time: 38757 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5120	5060	5023	5023
q2	336	228	244	228
q3	3304	3300	3253	3253
q4	2162	2050	2046	2046
q5	5789	5810	5790	5790
q6	216	123	127	123
q7	2335	1917	1914	1914
q8	3397	3444	3456	3444
q9	8851	8744	8766	8744
q10	3811	3838	3861	3838
q11	580	470	475	470
q12	799	628	645	628
q13	8421	3172	3193	3172
q14	294	261	283	261
q15	577	517	520	517
q16	544	528	496	496
q17	1942	1780	1773	1773
q18	8789	8326	8314	8314
q19	1626	1585	1552	1552
q20	2207	1964	1950	1950
q21	5581	5406	5235	5235
q22	541	490	492	490
Total cold run time: 67222 ms
Total hot run time: 59261 ms

@yiguolei
Copy link
Contributor Author

yiguolei commented Jan 1, 2024

run p0 10

@yiguolei
Copy link
Contributor Author

yiguolei commented Jan 2, 2024

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 6cb7a1e397ded3b70900deaadcd9d220f9894101, data reload: false

------ Round 1 ----------------------------------
q1	17653	5564	5121	5121
q2	2025	158	146	146
q3	10528	1103	1130	1103
q4	10178	829	824	824
q5	7785	2969	2980	2969
q6	212	134	141	134
q7	924	553	508	508
q8	9242	2040	2017	2017
q9	6885	6437	6425	6425
q10	8253	3117	3063	3063
q11	438	226	222	222
q12	393	222	225	222
q13	17994	3592	3614	3592
q14	240	226	209	209
q15	561	520	525	520
q16	466	393	414	393
q17	969	488	490	488
q18	7328	6778	6690	6690
q19	1598	1450	1409	1409
q20	736	312	347	312
q21	2795	2370	2436	2370
q22	382	316	323	316
Total cold run time: 107585 ms
Total hot run time: 39053 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5072	5100	5088	5088
q2	343	231	232	231
q3	3291	3281	3299	3281
q4	2138	2040	2056	2040
q5	5806	5805	5782	5782
q6	216	126	125	125
q7	2322	1913	1948	1913
q8	3397	3453	3512	3453
q9	8798	8768	8751	8751
q10	3787	3861	3870	3861
q11	564	465	483	465
q12	785	637	656	637
q13	9429	3172	3194	3172
q14	309	271	259	259
q15	580	517	519	517
q16	528	465	536	465
q17	1943	1814	1764	1764
q18	8733	8362	8299	8299
q19	1626	1610	1616	1610
q20	2202	1996	1952	1952
q21	5629	5226	5316	5226
q22	542	451	492	451
Total cold run time: 68040 ms
Total hot run time: 59342 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 6cb7a1e397ded3b70900deaadcd9d220f9894101, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	925	358	332	332
query2	6441	1996	1903	1903
query3	6656	215	201	201
query4	26556	22564	22446	22446
query5	4097	554	513	513
query6	260	182	175	175
query7	4587	267	258	258
query8	229	198	192	192
query9	8191	2540	2563	2540
query10	467	237	269	237
query11	16246	15652	15609	15609
query12	127	76	73	73
query13	1638	325	330	325
query14	11163	7061	7194	7061
query15	224	181	192	181
query16	6171	271	259	259
query17	1810	486	481	481
query18	1831	266	266	266
query19	184	135	133	133
query20	77	79	78	78
query21	185	93	93	93
query22	5034	4629	4713	4629
query23	32094	31279	31463	31279
query24	12069	2837	2804	2804
query25	599	348	344	344
query26	1794	137	136	136
query27	2901	274	281	274
query28	6933	1889	1891	1889
query29	2177	384	372	372
query30	292	148	150	148
query31	990	773	781	773
query32	88	57	55	55
query33	733	251	249	249
query34	903	451	434	434
query35	880	748	752	748
query36	1312	1206	1242	1206
query37	184	66	70	66
query38	3409	3265	3266	3265
query39	1327	1323	1289	1289
query40	308	90	88	88
query41	38	35	34	34
query42	96	83	85	83
query43	533	486	494	486
query44	1050	695	700	695
query45	201	192	178	178
query46	1075	645	633	633
query47	1620	1500	1632	1500
query48	335	259	255	255
query49	1215	312	308	308
query50	785	312	321	312
query51	5306	5229	5270	5229
query52	91	81	80	80
query53	212	149	146	146
query54	1385	547	576	547
query55	94	86	74	74
query56	199	186	193	186
query57	1035	927	940	927
query58	215	197	202	197
query59	2766	2617	2653	2617
query60	240	241	256	241
query61	89	101	85	85
query62	652	443	459	443
query63	158	147	146	146
query64	5929	1732	1682	1682
query65	3327	3265	3256	3256
query66	1391	348	329	329
query67	15783	15755	15092	15092
query68	9823	503	520	503
query69	522	257	264	257
query70	1768	1495	1485	1485
query71	471	220	218	218
query72	5332	3585	3560	3560
query73	1895	311	302	302
query74	6962	6380	6457	6380
query75	4645	2313	2280	2280
query76	6214	1129	1098	1098
query77	655	248	293	248
query78	9103	8740	8569	8569
query79	1030	507	496	496
query80	663	342	342	342
query81	451	220	206	206
query82	219	94	85	85
query83	168	140	134	134
query84	251	52	52	52
query85	949	283	274	274
query86	380	369	373	369
query87	3560	3395	3344	3344
query88	2844	2236	2230	2230
query89	345	265	248	248
query90	1868	195	186	186
query91	119	89	94	89
query92	58	50	54	50
query93	1123	434	473	434
query94	814	176	175	175
query95	460	423	396	396
query96	629	319	318	318
query97	4322	4160	4166	4160
query98	214	203	190	190
query99	1086	811	797	797
Total cold run time: 286855 ms
Total hot run time: 178640 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 6cb7a1e397ded3b70900deaadcd9d220f9894101, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5533	5104	5166	5104
q2	390	166	158	158
q3	1447	1257	1228	1228
q4	1083	813	779	779
q5	3114	3072	3067	3067
q6	221	145	133	133
q7	987	549	516	516
q8	2154	2186	2278	2186
q9	6702	6727	6650	6650
q10	3206	3135	3093	3093
q11	343	221	207	207
q12	383	234	235	234
q13	4412	3669	3642	3642
q14	248	219	222	219
q15	582	541	528	528
q16	456	409	395	395
q17	1035	570	507	507
q18	7101	6723	6834	6723
q19	1634	1502	1442	1442
q20	573	342	335	335
q21	2866	2422	2505	2422
q22	413	320	326	320
Total cold run time: 44883 ms
Total hot run time: 39888 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5079	5065	5049	5049
q2	337	236	243	236
q3	3355	3334	3295	3295
q4	2161	2053	2026	2026
q5	5964	5950	5972	5950
q6	221	124	127	124
q7	2381	1892	1944	1892
q8	3565	3684	3654	3654
q9	9016	9054	9039	9039
q10	3882	3931	3932	3931
q11	588	475	484	475
q12	804	642	629	629
q13	3843	3172	3177	3172
q14	306	270	277	270
q15	595	530	537	530
q16	564	513	499	499
q17	2042	1837	1804	1804
q18	8758	8304	8439	8304
q19	1747	1709	1667	1667
q20	2260	1993	1974	1974
q21	5724	5353	5334	5334
q22	548	490	504	490
Total cold run time: 63740 ms
Total hot run time: 60344 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.56 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183948055 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.65% (8615/23506)
Line Coverage: 28.70% (70018/244006)
Region Coverage: 27.68% (36236/130925)
Branch Coverage: 24.38% (18517/75948)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6cb7a1e397ded3b70900deaadcd9d220f9894101_6cb7a1e397ded3b70900deaadcd9d220f9894101/report/index.html

@yiguolei
Copy link
Contributor Author

yiguolei commented Jan 2, 2024

run p0

1 similar comment
@yiguolei
Copy link
Contributor Author

yiguolei commented Jan 3, 2024

run p0

@yiguolei
Copy link
Contributor Author

yiguolei commented Jan 3, 2024

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 2cd9264b70dafee7e16b87eb8d54f7c0350acb0a, data reload: false

------ Round 1 ----------------------------------
q1	17733	5420	5122	5122
q2	2024	157	144	144
q3	10698	1107	1150	1107
q4	10189	803	891	803
q5	7779	3007	2931	2931
q6	212	136	171	136
q7	923	521	507	507
q8	9318	2031	2040	2031
q9	6870	6422	6394	6394
q10	8295	3025	3006	3006
q11	436	208	211	208
q12	388	228	226	226
q13	18020	3610	3668	3610
q14	240	213	208	208
q15	566	546	523	523
q16	474	410	417	410
q17	967	495	494	494
q18	7225	6800	6680	6680
q19	1579	1303	1395	1303
q20	672	325	343	325
q21	2823	2336	2375	2336
q22	365	313	314	313
Total cold run time: 107796 ms
Total hot run time: 38817 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5095	5101	5003	5003
q2	338	229	240	229
q3	3286	3270	3277	3270
q4	2125	2035	2060	2035
q5	5821	5777	5786	5777
q6	222	125	126	125
q7	2312	1896	1930	1896
q8	3358	3439	3458	3439
q9	8784	8797	8758	8758
q10	3786	3852	3856	3852
q11	576	476	473	473
q12	787	641	646	641
q13	7778	3218	3184	3184
q14	286	265	260	260
q15	583	529	514	514
q16	568	522	511	511
q17	1966	1794	1748	1748
q18	8661	8275	8242	8242
q19	1620	1603	1577	1577
q20	2207	1994	1932	1932
q21	5595	5333	5266	5266
q22	525	476	507	476
Total cold run time: 66279 ms
Total hot run time: 59208 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 2cd9264b70dafee7e16b87eb8d54f7c0350acb0a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5583	5104	5128	5104
q2	398	184	150	150
q3	1474	1203	1086	1086
q4	1078	794	807	794
q5	3102	3125	3006	3006
q6	236	142	136	136
q7	974	565	526	526
q8	2147	2274	2206	2206
q9	6697	6662	6654	6654
q10	3170	3142	3122	3122
q11	345	213	221	213
q12	379	243	233	233
q13	4404	3672	3647	3647
q14	257	215	212	212
q15	585	540	525	525
q16	462	406	406	406
q17	1046	590	522	522
q18	7061	6756	6774	6756
q19	1635	1517	1541	1517
q20	599	332	321	321
q21	2931	2560	2425	2425
q22	400	314	314	314
Total cold run time: 44963 ms
Total hot run time: 39875 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5061	5016	5079	5016
q2	338	241	232	232
q3	3365	3338	3308	3308
q4	2140	2075	2052	2052
q5	5965	5953	5949	5949
q6	223	130	127	127
q7	2392	1896	1938	1896
q8	3572	3668	3667	3667
q9	9127	8983	9011	8983
q10	3878	3919	3911	3911
q11	580	474	473	473
q12	798	643	642	642
q13	3865	3226	3204	3204
q14	303	264	275	264
q15	585	534	531	531
q16	552	500	485	485
q17	2040	1811	1790	1790
q18	8713	8433	8425	8425
q19	1754	1651	1690	1651
q20	2272	1970	1973	1970
q21	5774	5377	5370	5370
q22	566	496	494	494
Total cold run time: 63863 ms
Total hot run time: 60440 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 2cd9264b70dafee7e16b87eb8d54f7c0350acb0a, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	946	343	332	332
query2	6442	1970	1938	1938
query3	6656	210	204	204
query4	27151	22649	22285	22285
query5	3896	554	518	518
query6	284	209	179	179
query7	4599	270	263	263
query8	231	199	196	196
query9	8209	2527	2523	2523
query10	396	250	245	245
query11	16103	15655	15479	15479
query12	129	77	77	77
query13	1630	323	314	314
query14	11374	7054	7099	7054
query15	227	180	193	180
query16	6398	269	257	257
query17	1884	500	493	493
query18	1958	267	261	261
query19	192	135	139	135
query20	77	70	75	70
query21	183	99	93	93
query22	5105	4795	4895	4795
query23	31904	31271	31314	31271
query24	12146	2815	2811	2811
query25	607	348	348	348
query26	1772	140	139	139
query27	2917	268	278	268
query28	7024	1881	1861	1861
query29	2115	385	396	385
query30	289	141	149	141
query31	968	761	761	761
query32	86	55	55	55
query33	741	292	269	269
query34	869	440	449	440
query35	896	763	794	763
query36	1338	1195	1173	1173
query37	182	70	72	70
query38	3366	3328	3272	3272
query39	1332	1295	1276	1276
query40	301	89	87	87
query41	37	36	35	35
query42	88	86	84	84
query43	539	448	482	448
query44	1098	685	694	685
query45	197	189	194	189
query46	1061	634	634	634
query47	1653	1591	1589	1589
query48	343	263	253	253
query49	1225	322	318	318
query50	745	329	316	316
query51	5345	5296	5228	5228
query52	86	91	83	83
query53	217	159	144	144
query54	1385	543	574	543
query55	97	86	87	86
query56	185	185	193	185
query57	1036	950	930	930
query58	234	200	194	194
query59	2797	2602	2558	2558
query60	228	243	202	202
query61	86	84	90	84
query62	633	449	464	449
query63	163	145	143	143
query64	5984	1742	1763	1742
query65	3307	3250	3252	3250
query66	1385	354	349	349
query67	15752	15501	15341	15341
query68	10942	527	517	517
query69	505	256	257	256
query70	1755	1550	1506	1506
query71	493	226	226	226
query72	5645	3582	3591	3582
query73	2086	313	321	313
query74	6967	6454	6405	6405
query75	4833	2291	2268	2268
query76	6245	1081	1081	1081
query77	657	251	278	251
query78	9086	8795	8573	8573
query79	1045	509	508	508
query80	555	352	350	350
query81	454	211	207	207
query82	208	92	87	87
query83	161	137	137	137
query84	243	54	53	53
query85	942	299	268	268
query86	501	405	415	405
query87	3520	3357	3327	3327
query88	2796	2248	2250	2248
query89	326	277	246	246
query90	1896	195	194	194
query91	118	88	90	88
query92	60	53	51	51
query93	1667	434	476	434
query94	827	179	178	178
query95	463	413	414	413
query96	624	317	321	317
query97	4274	4168	4184	4168
query98	208	199	189	189
query99	1111	798	811	798
Total cold run time: 290042 ms
Total hot run time: 178987 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.31 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17183835122 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.62% (8612/23515)
Line Coverage: 28.67% (69980/244109)
Region Coverage: 27.65% (36224/131010)
Branch Coverage: 24.35% (18503/75990)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2cd9264b70dafee7e16b87eb8d54f7c0350acb0a_2cd9264b70dafee7e16b87eb8d54f7c0350acb0a/report/index.html

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 3, 2024
Copy link
Contributor

github-actions bot commented Jan 3, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 3, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit bd8113f into apache:master Jan 3, 2024
26 of 28 checks passed
shuke987 added a commit to shuke987/doris that referenced this pull request Jan 4, 2024
shuke987 added a commit to shuke987/doris that referenced this pull request Jan 4, 2024
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants