Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) topn runtime filter only support simplest case #29312

Merged
merged 1 commit into from
Jan 3, 2024
Merged

[fix](Nereids) topn runtime filter only support simplest case #29312

merged 1 commit into from
Jan 3, 2024

Conversation

morrySnow
Copy link
Contributor

only support simple case: select ... from tbl [where ...] order by ... limit ...

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow
[fix](Nereids) topn runtime filter only support simplest case
7055910 
only support simple case: select ... from tbl [where ...] order by ... limit ...
@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 70559108d00cbbc908ca48d53b6019c6d83db45c, data reload: false

------ Round 1 ----------------------------------
q1	17629	5286	5321	5286
q2	2024	162	153	153
q3	10573	1150	1317	1150
q4	10189	900	945	900
q5	7807	3142	3049	3049
q6	234	142	140	140
q7	983	532	544	532
q8	9358	2236	2232	2232
q9	7057	6713	6646	6646
q10	8352	3122	3182	3122
q11	450	240	210	210
q12	436	245	243	243
q13	17993	3603	3629	3603
q14	256	216	212	212
q15	597	571	547	547
q16	462	414	426	414
q17	1018	603	529	529
q18	7590	6777	6760	6760
q19	1784	1580	1522	1522
q20	714	359	391	359
q21	2988	2534	2551	2534
q22	370	330	327	327
Total cold run time: 108864 ms
Total hot run time: 40470 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5264	5251	5204	5204
q2	357	257	279	257
q3	3369	3360	3350	3350
q4	2245	2093	2080	2080
q5	5958	5968	5951	5951
q6	244	136	135	135
q7	2387	1909	1930	1909
q8	3577	3655	3691	3655
q9	9010	8955	8906	8906
q10	3893	3950	3951	3950
q11	612	528	498	498
q12	836	688	657	657
q13	9506	3183	3211	3183
q14	293	283	282	282
q15	609	553	558	553
q16	578	524	525	524
q17	2061	1830	1824	1824
q18	8952	8501	8465	8465
q19	1832	1764	1785	1764
q20	2257	1973	1976	1973
q21	5895	5405	5365	5365
q22	538	491	463	463
Total cold run time: 70273 ms
Total hot run time: 60948 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.39 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17184200364 Bytes

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 70559108d00cbbc908ca48d53b6019c6d83db45c, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	932	364	344	344
query2	4091	1961	2104	1961
query3	6577	232	223	223
query4	27171	22477	22452	22452
query5	4912	558	569	558
query6	298	179	184	179
query7	4584	279	265	265
query8	227	207	214	207
query9	8146	2652	2691	2652
query10	372	247	266	247
query11	16362	15738	15507	15507
query12	139	82	80	80
query13	1638	327	327	327
query14	10725	7431	7404	7404
query15	227	194	194	194
query16	5519	308	302	302
query17	1654	512	499	499
query18	1915	278	270	270
query19	270	145	141	141
query20	87	82	79	79
query21	187	102	96	96
query22	4836	4830	4814	4814
query23	32316	31398	31308	31308
query24	12036	2831	2814	2814
query25	588	349	352	349
query26	1467	151	151	151
query27	2829	284	284	284
query28	7147	1984	1973	1973
query29	1727	415	397	397
query30	287	141	145	141
query31	951	787	795	787
query32	93	65	62	62
query33	725	282	283	282
query34	858	463	449	449
query35	873	774	755	755
query36	1377	1236	1269	1236
query37	192	87	90	87
query38	3365	3284	3237	3237
query39	1313	1305	1283	1283
query40	304	94	94	94
query41	38	36	34	34
query42	99	94	97	94
query43	528	534	512	512
query44	1106	767	788	767
query45	209	196	188	188
query46	1084	655	665	655
query47	1685	1673	1592	1592
query48	337	270	263	263
query49	1209	340	335	335
query50	764	381	344	344
query51	5410	5257	5372	5257
query52	99	96	86	86
query53	217	159	146	146
query54	1488	602	638	602
query55	105	89	89	89
query56	221	203	206	203
query57	1056	976	951	951
query58	238	214	204	204
query59	2925	2663	2680	2663
query60	263	229	232	229
query61	82	82	82	82
query62	654	467	480	467
query63	166	164	162	162
query64	5265	1722	1710	1710
query65	3352	3282	3266	3266
query66	1307	330	333	330
query67	15331	15008	15235	15008
query68	10961	540	512	512
query69	615	277	261	261
query70	1703	1534	1570	1534
query71	565	236	233	233
query72	4846	3541	3505	3505
query73	1436	322	317	317
query74	7018	6526	6472	6472
query75	5590	2283	2273	2273
query76	5338	1125	1098	1098
query77	879	292	291	291
query78	9133	8627	8588	8588
query79	4725	507	519	507
query80	2851	380	373	373
query81	515	210	212	210
query82	315	128	126	126
query83	315	137	136	136
query84	255	53	60	53
query85	2530	280	275	275
query86	404	385	402	385
query87	3591	3377	3352	3352
query88	3268	2451	2447	2447
query89	344	277	273	273
query90	1942	248	242	242
query91	119	92	91	91
query92	72	57	54	54
query93	2765	500	430	430
query94	896	219	212	212
query95	520	475	473	473
query96	655	341	341	341
query97	4287	4180	4175	4175
query98	213	195	193	193
query99	1134	837	840	837
Total cold run time: 294355 ms
Total hot run time: 180998 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 70559108d00cbbc908ca48d53b6019c6d83db45c, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5456	5139	5126	5126
q2	393	186	168	168
q3	1478	1201	1179	1179
q4	1103	821	749	749
q5	3172	3180	3109	3109
q6	232	139	140	139
q7	989	592	532	532
q8	2165	2282	2230	2230
q9	6717	6685	6677	6677
q10	3203	3154	3129	3129
q11	354	237	233	233
q12	395	247	248	247
q13	4455	3682	3668	3668
q14	264	226	232	226
q15	643	580	559	559
q16	459	409	425	409
q17	1043	589	533	533
q18	7132	6823	6857	6823
q19	1646	1559	1494	1494
q20	626	360	378	360
q21	2881	2473	2544	2473
q22	405	332	336	332
Total cold run time: 45211 ms
Total hot run time: 40395 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5153	5142	5114	5114
q2	336	257	235	235
q3	3399	3341	3327	3327
q4	2163	2033	2052	2033
q5	6006	5942	5936	5936
q6	239	139	135	135
q7	2420	1951	1942	1942
q8	3567	3684	3689	3684
q9	9094	9039	9026	9026
q10	3886	3907	3948	3907
q11	597	486	496	486
q12	829	652	632	632
q13	3882	3225	3203	3203
q14	296	276	290	276
q15	636	562	577	562
q16	546	546	527	527
q17	2063	1835	1830	1830
q18	8874	8375	8426	8375
q19	1752	1698	1687	1687
q20	2309	2000	2017	2000
q21	5763	5475	5494	5475
q22	586	513	502	502
Total cold run time: 64396 ms
Total hot run time: 60894 ms

zhiqiang-hhhh
zhiqiang-hhhh approved these changes Dec 31, 2023
View reviewed changes
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

wm1581066
wm1581066 approved these changes Dec 31, 2023
View reviewed changes
Copy link
Collaborator

@wm1581066 wm1581066 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 2, 2024

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit cdf2bb2 into apache:master Jan 3, 2024
29 of 32 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 3, 2024
@morrySnow @seawinde
[fix](Nereids) topn runtime filter only support simplest case (apache…
2df3d37 
…#29312)

only support simple case: select ... from tbl [where ...] order by ... limit ...
@morrySnow morrySnow deleted the fix_topn branch January 3, 2024 05:48
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jan 3, 2024
@morrySnow
[fix](Nereids) topn runtime filter only support simplest case (apache…
44cfcad 
…#29312)

only support simple case: select ... from tbl [where ...] order by ... limit ...
@morrySnow morrySnow mentioned this pull request Jan 3, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jan 3, 2024
@morrySnow
[fix](Nereids) topn runtime filter only support simplest case (#29312) (
8bc3c36 
#29455)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@morrySnow @HappenLee
[fix](Nereids) topn runtime filter only support simplest case (apache…
92ffec8 
…#29312)

only support simple case: select ... from tbl [where ...] order by ... limit ...
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jan 15, 2024
@morrySnow
[fix](Nereids) top-n with top project should hit top-n opt
a7893a9 
in PR apache#29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.
@morrySnow morrySnow mentioned this pull request Jan 15, 2024
Merged
englefly pushed a commit that referenced this pull request Jan 15, 2024
@morrySnow
[fix](Nereids) top-n with top project should hit top-n opt (#29971)
ce89a9f 
in PR #29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 15, 2024
@morrySnow @seawinde
[fix](Nereids) top-n with top project should hit top-n opt (apache#29971
bbad5d5 
)

in PR apache#29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 15, 2024
@morrySnow @seawinde
[fix](Nereids) top-n with top project should hit top-n opt (apache#29971
b6e4040 
)

in PR apache#29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.
yiguolei pushed a commit that referenced this pull request Jan 16, 2024
@morrySnow @Doris-Extras
[fix](Nereids) top-n with top project should hit top-n opt (#29971)
b99b9ef 
in PR #29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jan 29, 2024
@morrySnow
[fix](Nereids) top-n with top project should hit top-n opt (apache#29971
4a22b20 
)

in PR apache#29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.

pick from master apache#29971
commit id ce89a9f
@morrySnow morrySnow mentioned this pull request Jan 29, 2024
Merged
morrySnow added a commit that referenced this pull request Jan 30, 2024
@morrySnow
[fix](Nereids) top-n with top project should hit top-n opt (#29971) (#…
a05505a 
…30521)

in PR #29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.

pick from master #29971
commit id ce89a9f
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morrySnow
[fix](Nereids) top-n with top project should hit top-n opt (apache#29971
8a5c857 
) (apache#30521)

in PR apache#29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.

pick from master apache#29971
commit id ce89a9f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wm1581066 wm1581066 wm1581066 approved these changes

@zhiqiang-hhhh zhiqiang-hhhh zhiqiang-hhhh approved these changes

@englefly englefly englefly approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged dev/3.0.0-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@morrySnow @doris-robot @englefly @zhiqiang-hhhh @wm1581066 @dataroaring