Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix](scannerscheduler) should minus num_of_scanners before check should schedule #28926

Closed
wants to merge 4 commits into from

Conversation

yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Dec 24, 2023

Proposed changes

  1. remove wait parameter from get block from queue.
  2. should minus num_of_scanners before check should schedule

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.60% (8550/23363)
Line Coverage: 28.67% (69525/242504)
Region Coverage: 27.66% (35945/129943)
Branch Coverage: 24.40% (18378/75324)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cfce1f6eab8abe6aa0bcbd4aaeb6c70e3f1cd4e6_cfce1f6eab8abe6aa0bcbd4aaeb6c70e3f1cd4e6/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit cfce1f6eab8abe6aa0bcbd4aaeb6c70e3f1cd4e6, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4717	4424	4428	4424
q2	373	142	158	142
q3	1458	1226	1279	1226
q4	1103	951	945	945
q5	3183	3190	3185	3185
q6	250	131	131	131
q7	1011	500	483	483
q8	2200	2213	2199	2199
q9	6708	6681	6679	6679
q10	3246	3289	3283	3283
q11	308	184	187	184
q12	353	213	214	213
q13	4544	3851	3842	3842
q14	241	218	222	218
q15	584	529	522	522
q16	443	392	385	385
q17	1017	676	581	581
q18	7055	6816	6679	6679
q19	1539	1460	1402	1402
q20	525	312	297	297
q21	3099	2644	2651	2644
q22	355	283	285	283
Total cold run time: 44312 ms
Total hot run time: 39947 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4336	4340	4336	4336
q2	273	167	176	167
q3	3524	3507	3497	3497
q4	2399	2379	2383	2379
q5	5733	5703	5711	5703
q6	247	123	126	123
q7	2370	1841	1874	1841
q8	3545	3535	3528	3528
q9	9113	9035	9005	9005
q10	3902	4022	4005	4005
q11	494	375	362	362
q12	777	587	590	587
q13	4287	3578	3574	3574
q14	291	255	250	250
q15	575	520	518	518
q16	497	460	466	460
q17	1885	1852	1943	1852
q18	8595	8047	8241	8047
q19	1739	1733	1739	1733
q20	2252	1934	1918	1918
q21	6539	6179	6185	6179
q22	486	442	461	442
Total cold run time: 63859 ms
Total hot run time: 60506 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.17 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183199968 Bytes

@yiguolei
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.60% (8551/23363)
Line Coverage: 28.67% (69529/242503)
Region Coverage: 27.67% (35952/129942)
Branch Coverage: 24.40% (18378/75324)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5312999803a7f7169300755c2a6e9d26bb2e79b7_5312999803a7f7169300755c2a6e9d26bb2e79b7/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5312999803a7f7169300755c2a6e9d26bb2e79b7, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4674	4387	4410	4387
q2	363	151	159	151
q3	1460	1285	1259	1259
q4	1128	907	879	879
q5	3160	3159	3143	3143
q6	247	135	136	135
q7	980	478	495	478
q8	2181	2224	2184	2184
q9	6705	6645	6662	6645
q10	3218	3292	3277	3277
q11	302	188	186	186
q12	355	207	212	207
q13	4582	3789	3824	3789
q14	243	214	216	214
q15	576	533	512	512
q16	437	380	381	380
q17	1017	624	506	506
q18	7046	6800	6914	6800
q19	1527	1453	1392	1392
q20	508	301	317	301
q21	3061	2667	2769	2667
q22	350	277	280	277
Total cold run time: 44120 ms
Total hot run time: 39769 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4361	4358	4319	4319
q2	270	172	169	169
q3	3505	3508	3485	3485
q4	2404	2379	2382	2379
q5	5711	5724	5694	5694
q6	242	125	125	125
q7	2358	1866	1836	1836
q8	3528	3528	3522	3522
q9	9067	8965	8951	8951
q10	3906	4015	4008	4008
q11	494	377	372	372
q12	778	602	596	596
q13	4296	3566	3578	3566
q14	282	262	261	261
q15	568	524	518	518
q16	495	457	467	457
q17	1885	1892	1848	1848
q18	8491	8128	8126	8126
q19	1731	1753	1724	1724
q20	2250	1958	1940	1940
q21	6524	6131	6136	6131
q22	508	425	430	425
Total cold run time: 63654 ms
Total hot run time: 60452 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.39 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183697434 Bytes

@yiguolei
Copy link
Contributor Author

run compile

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yiguolei
Copy link
Contributor Author

run p0

1 similar comment
@yiguolei
Copy link
Contributor Author

run p0

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -318,8 +318,8 @@ Status VScanNode::_init_profile() {
return Status::OK();
}

Status VScanNode::_start_scanners(const std::list<VScannerSPtr>& scanners,
const int query_parallel_instance_num) {
void VScanNode::_start_scanners(const std::list<std::shared_ptr<ScannerDelegate>>& scanners,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method '_start_scanners' can be made const [readability-make-member-function-const]

be/src/vec/exec/scan/vscan_node.cpp:321:

-                                 const int query_parallel_instance_num) {
+                                 const int query_parallel_instance_num) const {

public:
VScannerSPtr _scanner;
ScannerDelegate(VScannerSPtr& scanner_ptr) : _scanner(scanner_ptr) {}
~ScannerDelegate() { static_cast<void>(_scanner->close(_scanner->runtime_state())); }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: use '= default' to define a trivial destructor [modernize-use-equals-default]

    ~ScannerDelegate() { static_cast<void>(_scanner->close(_scanner->runtime_state())); }
    ^

Status _start_scanners(const std::list<VScannerSPtr>& scanners,
const int query_parallel_instance_num);
void _start_scanners(const std::list<std::shared_ptr<ScannerDelegate>>& scanners,
const int query_parallel_instance_num);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: parameter 'query_parallel_instance_num' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change
const int query_parallel_instance_num);
int query_parallel_instance_num);

@yiguolei
Copy link
Contributor Author

run compile

@yiguolei
Copy link
Contributor Author

run p0 10

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8552/23377)
Line Coverage: 28.65% (69531/242686)
Region Coverage: 27.65% (35956/130049)
Branch Coverage: 24.38% (18376/75384)
Coverage Report: http://coverage.selectdb-in.cc/coverage/13abd2ae12b87390e851bf648feee69d3bb46277_13abd2ae12b87390e851bf648feee69d3bb46277/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 13abd2ae12b87390e851bf648feee69d3bb46277, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5556	5100	5013	5013
q2	363	152	160	152
q3	1466	1259	1279	1259
q4	1120	879	887	879
q5	3140	3183	3191	3183
q6	249	143	133	133
q7	1066	486	493	486
q8	2156	2207	2192	2192
q9	6699	6697	6681	6681
q10	3225	3270	3268	3268
q11	302	183	191	183
q12	362	209	209	209
q13	4572	3834	3792	3792
q14	245	218	218	218
q15	580	523	522	522
q16	457	390	388	388
q17	1021	627	518	518
q18	7054	6771	6790	6771
q19	1508	1420	1432	1420
q20	551	364	334	334
q21	3083	2679	2698	2679
q22	349	282	281	281
Total cold run time: 45124 ms
Total hot run time: 40561 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4475	4408	4379	4379
q2	301	181	199	181
q3	3539	3525	3587	3525
q4	2428	2401	2394	2394
q5	5741	5739	5737	5737
q6	242	124	125	124
q7	2403	1861	1865	1861
q8	3666	3689	3699	3689
q9	9092	9031	9062	9031
q10	3922	4014	3999	3999
q11	521	385	389	385
q12	783	602	600	600
q13	4309	3564	3558	3558
q14	293	259	262	259
q15	578	529	515	515
q16	501	469	499	469
q17	2105	2073	2060	2060
q18	8812	8314	8289	8289
q19	1772	1797	1754	1754
q20	2272	1944	1945	1944
q21	6808	6409	6375	6375
q22	513	435	471	435
Total cold run time: 65076 ms
Total hot run time: 61563 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.36 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17183639194 Bytes

@yiguolei
Copy link
Contributor Author

run p0 10

@yiguolei
Copy link
Contributor Author

run p0

1 similar comment
@yiguolei
Copy link
Contributor Author

run p0

@yiguolei
Copy link
Contributor Author

run p0 10

@yiguolei
Copy link
Contributor Author

run buildall

1 similar comment
@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.59% (8554/23377)
Line Coverage: 28.66% (69558/242739)
Region Coverage: 27.65% (35962/130046)
Branch Coverage: 24.39% (18382/75374)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c580fd6ba0ccbc93266e743a49bcc3fa5a6aa99_3c580fd6ba0ccbc93266e743a49bcc3fa5a6aa99/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 3c580fd6ba0ccbc93266e743a49bcc3fa5a6aa99, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5564	4990	5048	4990
q2	370	155	167	155
q3	1467	1260	1214	1214
q4	1104	913	899	899
q5	3166	3148	3161	3148
q6	259	132	136	132
q7	1086	494	479	479
q8	2161	2214	2210	2210
q9	6717	6715	6652	6652
q10	3202	3267	3275	3267
q11	334	202	199	199
q12	354	214	211	211
q13	4582	3823	3806	3806
q14	239	215	214	214
q15	581	527	530	527
q16	437	383	383	383
q17	1008	650	578	578
q18	7075	6740	6994	6740
q19	1533	1409	1400	1400
q20	535	340	310	310
q21	3052	2637	2642	2637
q22	381	303	313	303
Total cold run time: 45207 ms
Total hot run time: 40454 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4421	4398	4395	4395
q2	297	200	188	188
q3	3529	3522	3505	3505
q4	2421	2398	2381	2381
q5	5738	5755	5750	5750
q6	243	123	125	123
q7	2392	1842	1876	1842
q8	3709	3705	3704	3704
q9	9104	9056	9039	9039
q10	3902	4000	4016	4000
q11	545	400	391	391
q12	774	620	609	609
q13	4324	3575	3580	3575
q14	291	252	261	252
q15	570	527	525	525
q16	508	466	458	458
q17	2210	2057	2086	2057
q18	8829	8329	8223	8223
q19	1771	1754	1760	1754
q20	2265	1944	1943	1943
q21	6790	6420	6389	6389
q22	552	482	508	482
Total cold run time: 65185 ms
Total hot run time: 61585 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.64% (8584/23430)
Line Coverage: 28.66% (69740/243318)
Region Coverage: 27.66% (36056/130359)
Branch Coverage: 24.37% (18417/75578)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e3fcf3789bb65c7de4bb03053b10b8bd24123a8_5e3fcf3789bb65c7de4bb03053b10b8bd24123a8/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5e3fcf3789bb65c7de4bb03053b10b8bd24123a8, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5865	5218	5369	5218
q2	384	150	188	150
q3	1570	1287	1221	1221
q4	1159	960	921	921
q5	3231	3193	3265	3193
q6	260	136	135	135
q7	1014	495	486	486
q8	2251	2278	2267	2267
q9	6770	6753	6755	6753
q10	3231	3310	3323	3310
q11	332	210	228	210
q12	354	211	207	207
q13	4600	3810	3821	3810
q14	242	219	219	219
q15	573	531	536	531
q16	454	394	383	383
q17	1050	790	605	605
q18	7124	6876	6763	6763
q19	1655	1665	1657	1657
q20	813	307	340	307
q21	3282	2831	2839	2831
q22	399	312	321	312
Total cold run time: 46613 ms
Total hot run time: 41489 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4475	4471	4437	4437
q2	314	233	230	230
q3	3550	3549	3556	3549
q4	2471	2443	2438	2438
q5	5784	5779	5773	5773
q6	253	127	131	127
q7	2418	1948	1876	1876
q8	3722	3735	3712	3712
q9	9150	9123	9070	9070
q10	3918	4056	4020	4020
q11	595	472	469	469
q12	795	628	643	628
q13	4321	3593	3608	3593
q14	289	265	265	265
q15	585	533	525	525
q16	530	468	484	468
q17	2061	2048	2061	2048
q18	8790	8371	8368	8368
q19	1847	1828	1832	1828
q20	2268	1991	1977	1977
q21	6896	6487	6448	6448
q22	594	495	487	487
Total cold run time: 65626 ms
Total hot run time: 62336 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.5 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17184004415 Bytes

@yiguolei
Copy link
Contributor Author

run p0 10

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8582/23443)
Line Coverage: 28.66% (69746/243352)
Region Coverage: 27.63% (36046/130468)
Branch Coverage: 24.36% (18416/75608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/88448ed73191d677ae3ea8d6ca7b89838eec67e0_88448ed73191d677ae3ea8d6ca7b89838eec67e0/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.58 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17183677296 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 88448ed73191d677ae3ea8d6ca7b89838eec67e0, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4981	4609	4631	4609
q2	360	153	158	153
q3	1463	1291	1185	1185
q4	1121	960	904	904
q5	3181	3150	3176	3150
q6	250	129	129	129
q7	1026	485	507	485
q8	2266	2257	2249	2249
q9	6702	6672	6674	6672
q10	3181	3242	3265	3242
q11	332	216	205	205
q12	349	203	202	202
q13	4150	3372	3409	3372
q14	238	217	211	211
q15	571	517	522	517
q16	439	390	380	380
q17	1039	755	600	600
q18	7183	6803	6958	6803
q19	1635	1626	1655	1626
q20	570	308	320	308
q21	3179	2666	2733	2666
q22	367	288	312	288
Total cold run time: 44583 ms
Total hot run time: 39956 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4587	4580	4553	4553
q2	273	167	168	167
q3	3366	3371	3338	3338
q4	2220	2198	2198	2198
q5	5705	5712	5714	5712
q6	244	118	118	118
q7	2388	1866	1875	1866
q8	3599	3602	3603	3602
q9	8984	8951	9018	8951
q10	3796	3889	3900	3889
q11	476	375	380	375
q12	778	592	594	592
q13	3903	3175	3209	3175
q14	287	259	253	253
q15	580	515	515	515
q16	491	429	456	429
q17	1972	1965	1944	1944
q18	8691	8239	8140	8140
q19	1748	1753	1746	1746
q20	2232	1930	1934	1930
q21	6117	5726	5756	5726
q22	547	464	451	451
Total cold run time: 62984 ms
Total hot run time: 59670 ms

@yiguolei
Copy link
Contributor Author

run p0 10

1 similar comment
@yiguolei
Copy link
Contributor Author

run p0 10

Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5f7bca48fd193a3287f239e12329f329fbc12c3c, data reload: false

------ Round 1 ----------------------------------
q1	18608	5446	5154	5154
q2	2034	164	157	157
q3	10592	1132	1209	1132
q4	10207	828	843	828
q5	7824	3019	2972	2972
q6	221	142	142	142
q7	946	529	553	529
q8	9343	2076	2060	2060
q9	6883	6455	6408	6408
q10	8298	3068	3028	3028
q11	424	237	222	222
q12	408	241	250	241
q13	18009	3670	3660	3660
q14	242	230	206	206
q15	576	538	524	524
q16	477	400	399	399
q17	983	508	501	501
q18	7399	6745	6739	6739
q19	1603	1284	1349	1284
q20	687	344	372	344
q21	2839	2399	2500	2399
q22	410	341	344	341
Total cold run time: 109013 ms
Total hot run time: 39270 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5134	5162	5133	5133
q2	343	245	240	240
q3	3350	3313	3306	3306
q4	2165	2013	2056	2013
q5	5846	5824	5811	5811
q6	224	135	136	135
q7	2353	1933	1922	1922
q8	3361	3507	3496	3496
q9	8803	8825	8765	8765
q10	3844	3861	3887	3861
q11	585	486	480	480
q12	823	651	640	640
q13	6823	3228	3234	3228
q14	290	281	274	274
q15	589	543	526	526
q16	541	521	508	508
q17	1945	1796	1798	1796
q18	8672	8377	8335	8335
q19	1641	1578	1617	1578
q20	2234	1973	1956	1956
q21	5692	5317	5332	5317
q22	548	465	484	465
Total cold run time: 65806 ms
Total hot run time: 59785 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 5f7bca48fd193a3287f239e12329f329fbc12c3c, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5504	5130	5134	5130
q2	387	167	178	167
q3	1459	1108	1120	1108
q4	1083	819	815	815
q5	3170	2986	3091	2986
q6	223	141	142	141
q7	987	556	567	556
q8	2147	2290	2288	2288
q9	6724	6662	6677	6662
q10	3167	3218	3140	3140
q11	351	231	231	231
q12	387	248	244	244
q13	4413	3680	3695	3680
q14	259	220	215	215
q15	597	534	536	534
q16	455	397	408	397
q17	1052	530	534	530
q18	7095	6803	6733	6733
q19	1646	1603	1582	1582
q20	604	361	335	335
q21	2893	2547	2432	2432
q22	408	342	332	332
Total cold run time: 45011 ms
Total hot run time: 40238 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5124	5096	5045	5045
q2	336	237	229	229
q3	3393	3337	3318	3318
q4	2170	2100	2077	2077
q5	5967	5924	5909	5909
q6	230	136	134	134
q7	2396	1938	1926	1926
q8	3565	3666	3678	3666
q9	9050	8952	8948	8948
q10	3884	3925	3935	3925
q11	585	480	478	478
q12	810	650	639	639
q13	3878	3227	3223	3223
q14	303	288	291	288
q15	595	537	540	537
q16	539	506	506	506
q17	1987	1785	1818	1785
q18	8727	8378	8346	8346
q19	1761	1732	1717	1717
q20	2276	1978	1991	1978
q21	5788	5399	5434	5399
q22	555	510	511	510
Total cold run time: 63919 ms
Total hot run time: 60583 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 5f7bca48fd193a3287f239e12329f329fbc12c3c, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	920	362	350	350
query2	6462	2020	2033	2020
query3	6647	225	222	222
query4	24541	22476	22408	22408
query5	4014	587	620	587
query6	285	194	191	191
query7	4552	306	290	290
query8	249	215	210	210
query9	8186	2822	2780	2780
query10	469	264	252	252
query11	16021	15622	15642	15622
query12	141	78	80	78
query13	1552	336	329	329
query14	12141	7592	7730	7592
query15	341	219	225	219
query16	6378	301	292	292
query17	1728	526	515	515
query18	1972	272	264	264
query19	190	158	152	152
query20	82	77	78	77
query21	191	104	104	104
query22	4923	4841	4622	4622
query23	32495	31252	31440	31252
query24	12297	3530	3539	3530
query25	618	378	378	378
query26	1566	160	163	160
query27	2675	296	309	296
query28	6381	1993	1987	1987
query29	1738	441	428	428
query30	309	158	163	158
query31	1091	809	804	804
query32	98	61	64	61
query33	690	267	278	267
query34	840	456	461	456
query35	970	786	860	786
query36	1288	1261	1238	1238
query37	193	79	83	79
query38	3460	3348	3283	3283
query39	1352	1328	1318	1318
query40	309	98	97	97
query41	38	36	35	35
query42	91	95	87	87
query43	521	506	498	498
query44	1125	757	765	757
query45	235	222	209	209
query46	1202	815	798	798
query47	1712	1523	1563	1523
query48	352	297	285	285
query49	1227	318	332	318
query50	1131	801	788	788
query51	5331	5255	5291	5255
query52	105	86	93	86
query53	210	176	178	176
query54	1403	656	669	656
query55	106	88	83	83
query56	225	196	204	196
query57	1004	986	943	943
query58	242	217	213	213
query59	2741	2647	2529	2529
query60	240	220	252	220
query61	86	83	84	83
query62	1381	508	519	508
query63	193	174	180	174
query64	6014	1837	1814	1814
query65	3385	3277	3324	3277
query66	1104	342	342	342
query67	15500	15637	15390	15390
query68	12094	549	523	523
query69	624	256	266	256
query70	1664	1484	1468	1468
query71	581	230	231	230
query72	6202	4068	4037	4037
query73	2055	332	323	323
query74	6905	6562	6497	6497
query75	6148	2284	2288	2284
query76	6024	1132	1090	1090
query77	909	278	300	278
query78	9076	8462	8468	8462
query79	2349	544	544	544
query80	3184	368	370	368
query81	537	216	222	216
query82	229	115	110	110
query83	322	149	144	144
query84	264	55	57	55
query85	1594	295	284	284
query86	403	406	404	404
query87	3626	3383	3443	3383
query88	2889	2384	2386	2384
query89	373	283	286	283
query90	1854	238	243	238
query91	125	93	92	92
query92	70	51	57	51
query93	1518	459	504	459
query94	740	217	215	215
query95	526	475	462	462
query96	655	330	334	330
query97	4307	4204	4140	4140
query98	219	196	191	191
query99	1673	926	930	926
Total cold run time: 296998 ms
Total hot run time: 183724 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.3 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17184007477 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8593/23473)
Line Coverage: 28.67% (69909/243844)
Region Coverage: 27.67% (36179/130758)
Branch Coverage: 24.38% (18492/75834)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5f7bca48fd193a3287f239e12329f329fbc12c3c_5f7bca48fd193a3287f239e12329f329fbc12c3c/report/index.html

@yiguolei yiguolei closed this Dec 29, 2023
yiguolei added a commit that referenced this pull request Jan 3, 2024
shuke987 added a commit to shuke987/doris that referenced this pull request Jan 4, 2024
shuke987 added a commit to shuke987/doris that referenced this pull request Jan 4, 2024
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants