Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Mar 6, 2025

What problem does this PR solve?

Opt materialized view rewrite performance when the num of struct infos are huge

Optimize the recursive algorithm to reduce the number of recursive calls. If a group has already been refreshed, skip subsequent refreshes.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Mar 6, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32335 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2f336c78a945a24e2ba37c3860760751e248f58, data reload: false

------ Round 1 ----------------------------------
q1	17625	5151	5046	5046
q2	2045	289	175	175
q3	10424	1249	742	742
q4	10244	1028	531	531
q5	7687	2335	2349	2335
q6	188	167	133	133
q7	869	767	601	601
q8	9301	1301	1032	1032
q9	4869	5014	4629	4629
q10	6831	2314	1890	1890
q11	475	284	275	275
q12	353	351	216	216
q13	17805	3663	3111	3111
q14	232	235	211	211
q15	540	493	481	481
q16	643	614	574	574
q17	593	858	346	346
q18	6935	6326	6365	6326
q19	1352	928	522	522
q20	323	326	191	191
q21	2922	2095	1971	1971
q22	1068	1011	997	997
Total cold run time: 103324 ms
Total hot run time: 32335 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5133	5137	5428	5137
q2	242	328	247	247
q3	2153	2684	2304	2304
q4	1457	1821	1396	1396
q5	4251	4153	4123	4123
q6	217	162	124	124
q7	1886	1913	1776	1776
q8	2613	2695	2591	2591
q9	7266	7163	7230	7163
q10	3023	3230	2786	2786
q11	568	525	490	490
q12	729	786	581	581
q13	3502	3973	3421	3421
q14	314	302	271	271
q15	524	471	492	471
q16	639	687	650	650
q17	1153	1621	1326	1326
q18	7764	7600	7517	7517
q19	824	800	961	800
q20	2005	2005	1877	1877
q21	5364	4996	4720	4720
q22	1125	1061	1052	1052
Total cold run time: 52752 ms
Total hot run time: 50823 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193109 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f2f336c78a945a24e2ba37c3860760751e248f58, data reload: false

query1	1375	1010	1018	1010
query2	6219	1997	1941	1941
query3	11016	4554	4365	4365
query4	54124	25817	23019	23019
query5	5131	529	506	506
query6	360	215	202	202
query7	4914	497	293	293
query8	319	248	249	248
query9	5793	2556	2607	2556
query10	421	306	255	255
query11	15159	15252	15239	15239
query12	160	118	110	110
query13	1063	515	402	402
query14	10871	6811	7113	6811
query15	213	212	175	175
query16	6998	630	499	499
query17	1092	739	594	594
query18	1494	410	329	329
query19	210	201	178	178
query20	129	127	137	127
query21	210	123	110	110
query22	4450	4503	4409	4409
query23	34268	33417	33668	33417
query24	5764	2477	2447	2447
query25	495	543	403	403
query26	723	287	160	160
query27	1846	508	335	335
query28	2851	2440	2490	2440
query29	572	580	429	429
query30	271	225	193	193
query31	883	850	793	793
query32	75	64	65	64
query33	451	358	314	314
query34	778	886	507	507
query35	794	837	751	751
query36	974	993	913	913
query37	129	97	79	79
query38	4343	4319	4336	4319
query39	1497	1415	1436	1415
query40	205	122	106	106
query41	52	51	49	49
query42	127	109	105	105
query43	518	542	507	507
query44	1345	801	818	801
query45	179	213	169	169
query46	854	1025	661	661
query47	1839	1827	1849	1827
query48	403	445	324	324
query49	711	512	441	441
query50	691	731	421	421
query51	4228	4298	4238	4238
query52	105	108	95	95
query53	247	264	185	185
query54	495	506	424	424
query55	86	85	83	83
query56	296	280	260	260
query57	1158	1211	1085	1085
query58	247	259	236	236
query59	2793	2863	2903	2863
query60	286	273	256	256
query61	127	117	119	117
query62	720	742	679	679
query63	222	189	186	186
query64	1745	1064	760	760
query65	4540	4513	4477	4477
query66	738	390	301	301
query67	15670	15384	15301	15301
query68	7349	885	508	508
query69	531	292	317	292
query70	1276	1163	1106	1106
query71	487	305	268	268
query72	5747	3643	3732	3643
query73	1302	731	342	342
query74	9272	9220	8905	8905
query75	3796	3177	2746	2746
query76	4223	1191	785	785
query77	586	355	276	276
query78	10116	10109	9331	9331
query79	2308	830	601	601
query80	645	596	464	464
query81	478	254	229	229
query82	669	133	98	98
query83	171	169	154	154
query84	285	100	77	77
query85	788	365	307	307
query86	400	294	303	294
query87	4470	4650	4309	4309
query88	3340	2243	2223	2223
query89	412	321	284	284
query90	1953	205	208	205
query91	136	140	108	108
query92	74	61	63	61
query93	1438	1162	579	579
query94	681	426	297	297
query95	351	268	266	266
query96	485	563	284	284
query97	3346	3464	3302	3302
query98	254	200	203	200
query99	1429	1430	1251	1251
Total cold run time: 297860 ms
Total hot run time: 193109 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f2f336c78a945a24e2ba37c3860760751e248f58, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.25	0.06	0.07
query4	1.60	0.10	0.10
query5	0.55	0.54	0.56
query6	1.21	0.73	0.72
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.54
query10	0.57	0.59	0.59
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.63	0.60	0.60
query14	2.66	2.82	2.81
query15	0.93	0.85	0.85
query16	0.38	0.36	0.37
query17	1.05	0.99	1.01
query18	0.21	0.20	0.20
query19	1.90	1.81	1.93
query20	0.01	0.01	0.01
query21	15.37	0.93	0.53
query22	0.76	1.19	0.62
query23	15.00	1.40	0.67
query24	6.77	1.94	0.57
query25	0.52	0.28	0.13
query26	0.61	0.16	0.15
query27	0.06	0.06	0.05
query28	9.31	0.91	0.45
query29	12.60	4.18	3.44
query30	0.24	0.09	0.06
query31	2.80	0.62	0.39
query32	3.23	0.54	0.46
query33	2.97	3.02	3.02
query34	15.68	5.12	4.51
query35	4.48	4.54	4.59
query36	0.69	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.62 s
Total hot run time: 30.87 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants