[feature](mtmv)support cancel mtmv task command #29252

zddr · 2023-12-28T11:31:11Z

Proposed changes

Issue Number: close #xxx

CANCEL MATERIALIZED VIEW TASK taskId on mvName
CANCEL MATERIALIZED VIEW TASK, tasks("type"="mv") and jobs("type"="mv") support check auth use priv of mv
tasks and jobs add column mvName and mvDbName，you can use select * from tasks("type"="mv") where MvName="xxx" get all tasks of mv
fix desc mv all error
fix p0 The task sequence is incorrect

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

zddr · 2023-12-29T06:21:08Z

run buildall

doris-robot · 2023-12-29T06:58:16Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 36a89ac68c7279df09695766d394e87795942c83, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5457	5188	5150	5150
q2	401	172	189	172
q3	1479	1186	1283	1186
q4	1078	800	876	800
q5	3115	3061	3136	3061
q6	238	143	144	143
q7	986	572	528	528
q8	2146	2193	2231	2193
q9	6666	6697	6678	6678
q10	3152	3104	3146	3104
q11	352	239	234	234
q12	392	241	242	241
q13	4424	3627	3651	3627
q14	266	228	216	216
q15	621	563	559	559
q16	452	411	388	388
q17	1051	509	506	506
q18	7088	6810	6800	6800
q19	1647	1542	1531	1531
q20	631	372	361	361
q21	2913	2510	2506	2506
q22	388	313	332	313
Total cold run time: 44943 ms
Total hot run time: 40297 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5159	5128	5057	5057
q2	343	257	277	257
q3	3391	3329	3323	3323
q4	2144	1992	1998	1992
q5	5937	5967	5952	5952
q6	242	135	136	135
q7	2385	1922	1960	1922
q8	3574	3663	3683	3663
q9	8992	9020	8999	8999
q10	3888	3945	3943	3943
q11	586	498	482	482
q12	810	643	636	636
q13	3889	3198	3173	3173
q14	300	273	288	273
q15	630	575	554	554
q16	558	520	525	520
q17	2063	1823	1788	1788
q18	8749	8521	8369	8369
q19	1742	1695	1752	1695
q20	2300	2013	1993	1993
q21	5637	5287	5346	5287
q22	569	523	519	519
Total cold run time: 63888 ms
Total hot run time: 60532 ms

doris-robot · 2023-12-29T07:04:58Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.68 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17184176800 Bytes

zddr · 2023-12-31T01:00:50Z

run buildall

doris-robot · 2023-12-31T01:20:48Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 8a9f0c5c44a671d6f3019f5a2e99598587ff7ce3, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5479	5165	5219	5165
q2	400	175	159	159
q3	1461	1203	1098	1098
q4	1071	793	777	777
q5	3164	3133	3119	3119
q6	228	138	132	132
q7	943	567	525	525
q8	2148	2209	2210	2209
q9	6726	6674	6654	6654
q10	3176	3161	3151	3151
q11	340	223	209	209
q12	387	236	228	228
q13	4410	3650	3631	3631
q14	264	214	220	214
q15	608	560	565	560
q16	449	422	389	389
q17	1034	562	551	551
q18	7108	6764	6838	6764
q19	1644	1547	1447	1447
q20	558	337	353	337
q21	2881	2489	2466	2466
q22	416	332	316	316
Total cold run time: 44895 ms
Total hot run time: 40101 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5135	5198	5076	5076
q2	335	244	257	244
q3	3378	3304	3284	3284
q4	2180	1993	2017	1993
q5	5921	5918	5941	5918
q6	234	125	127	125
q7	2379	1929	1938	1929
q8	3548	3639	3706	3639
q9	9048	9054	8984	8984
q10	3865	3909	3915	3909
q11	598	471	494	471
q12	809	654	684	654
q13	3889	3206	3214	3206
q14	310	267	263	263
q15	601	560	548	548
q16	593	527	511	511
q17	2057	1816	1829	1816
q18	8757	8410	8377	8377
q19	1750	1698	1686	1686
q20	2288	2031	1983	1983
q21	5710	5319	5302	5302
q22	554	537	494	494
Total cold run time: 63939 ms
Total hot run time: 60412 ms

doris-robot · 2023-12-31T01:33:16Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.33 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17187534833 Bytes

morningman

LGTM

github-actions · 2023-12-31T05:20:51Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-12-31T05:20:53Z

PR approved by anyone and no changes requested.

yiguolei

LGTM

zddr · 2023-12-31T08:58:35Z

run buildall

doris-robot · 2023-12-31T09:30:07Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit a533ed248ff94a38cff1eaaa0ac18814fd43410b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5399	5169	5058	5058
q2	396	159	156	156
q3	1472	1160	1149	1149
q4	1078	797	805	797
q5	3088	3080	3068	3068
q6	228	132	130	130
q7	961	531	544	531
q8	2162	2208	2195	2195
q9	6706	6657	6636	6636
q10	3182	3155	3093	3093
q11	338	224	223	223
q12	375	240	241	240
q13	4434	3639	3619	3619
q14	263	224	226	224
q15	621	549	557	549
q16	450	410	404	404
q17	1035	510	629	510
q18	7078	6724	6790	6724
q19	1639	1484	1566	1484
q20	685	361	363	361
q21	2842	2558	2466	2466
q22	402	322	313	313
Total cold run time: 44834 ms
Total hot run time: 39930 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5091	5088	5057	5057
q2	353	240	255	240
q3	3371	3288	3253	3253
q4	2115	1996	1995	1995
q5	5960	5922	5915	5915
q6	226	122	122	122
q7	2387	1934	1962	1934
q8	3524	3660	3655	3655
q9	9097	8967	8999	8967
q10	3884	3917	3944	3917
q11	593	483	472	472
q12	805	639	639	639
q13	3895	3231	3186	3186
q14	291	269	263	263
q15	608	549	554	549
q16	553	534	565	534
q17	2044	1820	1801	1801
q18	8762	8298	8352	8298
q19	1742	1722	1665	1665
q20	2305	2004	1985	1985
q21	5747	5365	5225	5225
q22	565	496	499	496
Total cold run time: 63918 ms
Total hot run time: 60168 ms

morningman

LGTM

github-actions · 2023-12-31T09:31:24Z

PR approved by at least one committer and no changes requested.

doris-robot · 2023-12-31T09:42:21Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.39 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17188189809 Bytes

wm1581066

LGTM

- `CANCEL MATERIALIZED VIEW TASK taskId on mvName` - CANCEL MATERIALIZED VIEW TASK, tasks("type"="mv") and jobs("type"="mv") support check auth use priv of mv - tasks and jobs add column mvName and mvDbName，you can use `select * from tasks("type"="mv") where MvName="xxx"` get all tasks of mv - fix `desc mv all` error - fix p0 The task sequence is incorrect

zddr added 9 commits December 28, 2023 18:35

1

ee4b820

1

b80dcc2

1

c5b4749

1

a9368ca

1

4f5bef3

1

a279cd6

1

e0d2345

1

7ee3a53

1

36a89ac

1

8a9f0c5

morningman previously approved these changes Dec 31, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 31, 2023

github-actions bot added the reviewed label Dec 31, 2023

yiguolei previously approved these changes Dec 31, 2023

View reviewed changes

1

a533ed2

zddr dismissed stale reviews from yiguolei and morningman via a533ed2 December 31, 2023 08:55

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 31, 2023

morningman approved these changes Dec 31, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 31, 2023

wm1581066 approved these changes Dec 31, 2023

View reviewed changes

morningman merged commit 5985d21 into apache:master Dec 31, 2023
29 of 30 checks passed

zddr deleted the mv_task_auth branch March 28, 2024 02:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](mtmv)support cancel mtmv task command #29252

[feature](mtmv)support cancel mtmv task command #29252

zddr commented Dec 28, 2023 •

edited

Loading

zddr commented Dec 29, 2023

doris-robot commented Dec 29, 2023

doris-robot commented Dec 29, 2023

zddr commented Dec 31, 2023

doris-robot commented Dec 31, 2023

doris-robot commented Dec 31, 2023

morningman left a comment

github-actions bot commented Dec 31, 2023

github-actions bot commented Dec 31, 2023

yiguolei left a comment

zddr commented Dec 31, 2023

doris-robot commented Dec 31, 2023

morningman left a comment

github-actions bot commented Dec 31, 2023

doris-robot commented Dec 31, 2023

wm1581066 left a comment

[feature](mtmv)support cancel mtmv task command #29252

[feature](mtmv)support cancel mtmv task command #29252

Conversation

zddr commented Dec 28, 2023 • edited Loading

Proposed changes

Further comments

zddr commented Dec 29, 2023

doris-robot commented Dec 29, 2023

doris-robot commented Dec 29, 2023

zddr commented Dec 31, 2023

doris-robot commented Dec 31, 2023

doris-robot commented Dec 31, 2023

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 31, 2023

github-actions bot commented Dec 31, 2023

yiguolei left a comment

Choose a reason for hiding this comment

zddr commented Dec 31, 2023

doris-robot commented Dec 31, 2023

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 31, 2023

doris-robot commented Dec 31, 2023

wm1581066 left a comment

Choose a reason for hiding this comment

zddr commented Dec 28, 2023 •

edited

Loading