[Feature](exectuor)Add scanbytes/scanrows condition, add move action in be #29337

wangbo · 2023-12-30T05:58:56Z

Add scanbytes/scanrows condition, add move action in be

wangbo · 2023-12-30T05:59:04Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/pipeline/task_scheduler.h

be/src/runtime/query_context.h

be/src/runtime/task_group/task_group_manager.cpp

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

doris-robot · 2023-12-30T06:45:41Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit ae64718ce7faa085cb0a3bd41fdb81ed2f2eae00, data reload: false

------ Round 1 ----------------------------------
q1	17625	5217	5134	5134
q2	2027	159	147	147
q3	10533	1080	1155	1080
q4	10191	785	841	785
q5	7795	2953	2804	2804
q6	218	131	130	130
q7	897	557	539	539
q8	9317	2016	2024	2016
q9	6882	6437	6400	6400
q10	8284	3072	3054	3054
q11	424	231	222	222
q12	384	230	232	230
q13	17989	3635	3644	3635
q14	239	216	214	214
q15	580	539	532	532
q16	452	410	382	382
q17	967	492	514	492
q18	7231	6648	6698	6648
q19	1569	1366	1370	1366
q20	700	338	371	338
q21	2798	2393	2395	2393
q22	389	351	332	332
Total cold run time: 107491 ms
Total hot run time: 38873 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5143	5172	5146	5146
q2	334	250	246	246
q3	3289	3253	3249	3249
q4	2110	2022	2025	2022
q5	5784	5781	5776	5776
q6	222	125	125	125
q7	2330	1877	1904	1877
q8	3368	3425	3487	3425
q9	8771	8744	8742	8742
q10	3795	3811	3836	3811
q11	579	480	487	480
q12	805	644	645	644
q13	8519	3214	3177	3177
q14	298	270	286	270
q15	594	528	529	528
q16	540	519	505	505
q17	1933	1820	1771	1771
q18	8590	8280	8272	8272
q19	1615	1567	1563	1563
q20	2188	1969	1984	1969
q21	5582	5235	5260	5235
q22	531	510	501	501
Total cold run time: 66920 ms
Total hot run time: 59334 ms

doris-robot · 2023-12-30T06:57:16Z

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit ae64718ce7faa085cb0a3bd41fdb81ed2f2eae00, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	934	363	335	335
query2	6418	1953	1882	1882
query3	6645	209	201	201
query4	27485	22619	22512	22512
query5	5860	527	557	527
query6	284	192	184	184
query7	4591	274	278	274
query8	246	204	201	201
query9	8223	2709	2651	2651
query10	440	274	268	268
query11	16287	15600	15472	15472
query12	142	79	76	76
query13	1662	338	326	326
query14	12051	7151	7152	7151
query15	237	187	196	187
query16	6426	291	270	270
query17	1861	510	508	508
query18	1946	276	275	275
query19	274	149	140	140
query20	83	79	84	79
query21	184	97	95	95
query22	5032	4712	4813	4712
query23	31805	31316	31146	31146
query24	11784	2804	2809	2804
query25	581	358	346	346
query26	1697	149	150	149
query27	2850	280	285	280
query28	7097	2004	2001	2001
query29	2044	395	400	395
query30	292	144	150	144
query31	1014	792	776	776
query32	85	61	62	61
query33	732	279	282	279
query34	858	444	444	444
query35	869	768	765	765
query36	1354	1236	1241	1236
query37	190	77	76	76
query38	3438	3276	3288	3276
query39	1316	1281	1276	1276
query40	284	95	91	91
query41	38	36	35	35
query42	97	99	94	94
query43	537	500	488	488
query44	1055	725	729	725
query45	196	193	178	178
query46	1090	658	646	646
query47	1719	1595	1651	1595
query48	336	270	264	264
query49	1189	326	328	326
query50	753	338	329	329
query51	5393	5255	5302	5255
query52	91	90	84	84
query53	213	147	152	147
query54	1352	566	601	566
query55	96	88	89	88
query56	219	204	212	204
query57	1000	960	973	960
query58	228	212	208	208
query59	2837	2590	2773	2590
query60	269	243	243	243
query61	91	90	87	87
query62	616	466	443	443
query63	170	152	159	152
query64	5949	1736	1733	1733
query65	3350	3268	3271	3268
query66	1264	338	326	326
query67	15579	15492	15234	15234
query68	12471	520	533	520
query69	503	280	263	263
query70	1681	1484	1473	1473
query71	486	235	242	235
query72	5633	3624	3544	3544
query73	2916	314	322	314
query74	7012	6382	6482	6382
query75	5145	2299	2299	2299
query76	6199	1134	1157	1134
query77	673	291	275	275
query78	9108	8932	8694	8694
query79	1029	516	519	516
query80	553	390	385	385
query81	475	211	214	211
query82	215	111	108	108
query83	165	140	140	140
query84	247	57	55	55
query85	941	314	285	285
query86	381	374	359	359
query87	3570	3348	3414	3348
query88	2837	2271	2290	2271
query89	350	267	268	267
query90	1850	218	221	218
query91	122	103	93	93
query92	64	59	60	59
query93	1351	456	446	446
query94	865	198	200	198
query95	474	427	409	409
query96	640	317	320	317
query97	4268	4135	4202	4135
query98	215	201	201	201
query99	1096	854	841	841
Total cold run time: 294863 ms
Total hot run time: 180134 ms

doris-robot · 2023-12-30T07:00:38Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.67 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17184069620 Bytes

doris-robot · 2023-12-30T07:22:42Z

TeamCity be ut coverage result:
Function Coverage: 36.63% (8616/23520)
Line Coverage: 28.69% (70052/244174)
Region Coverage: 27.68% (36254/130979)
Branch Coverage: 24.38% (18530/75992)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ae64718ce7faa085cb0a3bd41fdb81ed2f2eae00_ae64718ce7faa085cb0a3bd41fdb81ed2f2eae00/report/index.html

yiguolei · 2024-01-01T09:28:30Z

be/src/runtime/workload_management/workload_action.cpp

@@ -29,7 +30,35 @@ void WorkloadActionCancelQuery::exec(WorkloadQueryInfo* query_info) {
 }

 void WorkloadActionMoveQuery::exec(WorkloadQueryInfo* query_info) {
-    LOG(INFO) << "[workload_schedule]move query action run group=" << _wg_name;
+    QueryContext* query_ctx_ptr = query_info->_query_ctx_share_ptr.get();


如果再执行这个exec 期间，query 结束了，quer context 析构了，怎么版？

queryctx是存成一个sharedptr到WorkloadQueryInfo里了，WorkloadQueryInfo的生命周期是本次调度策略跑完所有的查询才会结束。不过这么一说感觉也有问题，queryctx可能析构的会很晚。感觉可以在每个查询策略匹配完之后就析构WorkloadQueryInfo

Add scanbytes/scanrows condition, add move action in be

ae64718

wangbo added the workload-schedule label Dec 30, 2023

github-actions bot reviewed Dec 30, 2023

View reviewed changes

Update be/src/pipeline/task_scheduler.h

e4475d0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

yiguolei reviewed Jan 1, 2024

View reviewed changes

wangbo closed this Feb 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature](exectuor)Add scanbytes/scanrows condition, add move action in be #29337

[Feature](exectuor)Add scanbytes/scanrows condition, add move action in be #29337

wangbo commented Dec 30, 2023

wangbo commented Dec 30, 2023

github-actions bot left a comment

doris-robot commented Dec 30, 2023

doris-robot commented Dec 30, 2023

doris-robot commented Dec 30, 2023

doris-robot commented Dec 30, 2023

yiguolei Jan 1, 2024

wangbo Jan 2, 2024

[Feature](exectuor)Add scanbytes/scanrows condition, add move action in be #29337

[Feature](exectuor)Add scanbytes/scanrows condition, add move action in be #29337

Conversation

wangbo commented Dec 30, 2023

wangbo commented Dec 30, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

doris-robot commented Dec 30, 2023

doris-robot commented Dec 30, 2023

doris-robot commented Dec 30, 2023

doris-robot commented Dec 30, 2023

yiguolei Jan 1, 2024

Choose a reason for hiding this comment

wangbo Jan 2, 2024

Choose a reason for hiding this comment