Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](inverted index) Ensure that col_unique_ids in TabletIndex are not empty. #46648 #46971

Merged
merged 1 commit into from
Jan 14, 2025

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #46648

…ot empty. (#46648)

Problem Summary:
1. Ensure that col_unique_ids in TabletIndex are not empty when
TabletIndex exists after a schema change.
2. Use unique_id to search for and complete index schema information.
@Thearas
Copy link
Contributor

Thearas commented Jan 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 14, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 14, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41227 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97aa19ef1846aca3be98fc9c9e570d8b005820ed, data reload: false

------ Round 1 ----------------------------------
q1	17623	7344	7330	7330
q2	2071	178	164	164
q3	10547	1083	1151	1083
q4	10523	777	687	687
q5	7756	2889	2828	2828
q6	236	151	152	151
q7	978	636	615	615
q8	9369	1975	2065	1975
q9	6616	6440	6373	6373
q10	7044	2269	2303	2269
q11	475	270	266	266
q12	402	215	215	215
q13	17804	3002	3006	3002
q14	241	217	210	210
q15	564	521	529	521
q16	693	595	619	595
q17	1006	594	552	552
q18	7250	6783	6763	6763
q19	1414	1124	1131	1124
q20	483	211	204	204
q21	4043	3318	3373	3318
q22	1096	982	993	982
Total cold run time: 108234 ms
Total hot run time: 41227 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7247	7233	7209	7209
q2	329	235	238	235
q3	2984	2937	2985	2937
q4	2069	1827	1837	1827
q5	5743	5816	5786	5786
q6	224	149	150	149
q7	2322	1806	1824	1806
q8	3395	3610	3518	3518
q9	8966	8999	8898	8898
q10	3581	3591	3503	3503
q11	604	511	495	495
q12	790	622	627	622
q13	10408	3143	3198	3143
q14	304	272	286	272
q15	583	524	538	524
q16	745	680	676	676
q17	1874	1631	1605	1605
q18	8345	7696	7556	7556
q19	1688	1595	1585	1585
q20	2132	1831	1876	1831
q21	5621	5448	5389	5389
q22	1135	1051	1039	1039
Total cold run time: 71089 ms
Total hot run time: 60605 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198050 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97aa19ef1846aca3be98fc9c9e570d8b005820ed, data reload: false

query1	1315	941	948	941
query2	6274	2066	2054	2054
query3	10817	4207	4300	4207
query4	66574	28923	23495	23495
query5	5081	478	474	474
query6	446	185	185	185
query7	5573	311	312	311
query8	298	216	229	216
query9	9098	2747	2713	2713
query10	506	278	266	266
query11	17600	15260	15750	15260
query12	165	107	105	105
query13	1501	450	441	441
query14	10978	7326	6918	6918
query15	218	189	191	189
query16	7081	557	496	496
query17	1080	603	620	603
query18	2088	347	341	341
query19	222	171	162	162
query20	119	117	115	115
query21	215	106	111	106
query22	4734	4279	4402	4279
query23	34888	34479	34267	34267
query24	6336	2984	3035	2984
query25	564	428	435	428
query26	666	177	185	177
query27	1873	355	375	355
query28	4102	2523	2508	2508
query29	713	505	442	442
query30	246	166	158	158
query31	1009	864	825	825
query32	67	53	62	53
query33	413	296	288	288
query34	932	542	570	542
query35	863	774	728	728
query36	1087	944	987	944
query37	130	72	73	72
query38	4186	4031	4044	4031
query39	1544	1454	1465	1454
query40	201	101	101	101
query41	50	48	49	48
query42	117	104	104	104
query43	554	506	508	506
query44	1208	826	829	826
query45	188	169	180	169
query46	1167	795	729	729
query47	2015	1902	1939	1902
query48	490	384	384	384
query49	760	391	406	391
query50	854	445	439	439
query51	7368	7235	7352	7235
query52	98	97	92	92
query53	268	184	185	184
query54	567	451	462	451
query55	81	76	75	75
query56	267	249	234	234
query57	1204	1104	1124	1104
query58	217	223	214	214
query59	3301	3110	3124	3110
query60	275	264	263	263
query61	115	108	141	108
query62	798	668	653	653
query63	224	197	198	197
query64	1396	676	674	674
query65	3278	3166	3214	3166
query66	696	301	299	299
query67	15988	15755	15477	15477
query68	4062	613	585	585
query69	452	265	266	265
query70	1112	1119	1125	1119
query71	390	264	262	262
query72	6522	4021	4041	4021
query73	793	351	356	351
query74	9943	9002	9080	9002
query75	3378	2632	2680	2632
query76	1902	1020	1169	1020
query77	500	284	283	283
query78	10552	9620	9587	9587
query79	2123	619	616	616
query80	1448	438	447	438
query81	534	246	242	242
query82	1261	123	118	118
query83	288	146	146	146
query84	278	88	86	86
query85	1031	306	300	300
query86	414	307	304	304
query87	4529	4304	4345	4304
query88	4009	2386	2343	2343
query89	407	293	291	291
query90	1970	188	190	188
query91	188	154	152	152
query92	74	53	52	52
query93	3008	557	554	554
query94	855	301	308	301
query95	364	257	267	257
query96	646	285	289	285
query97	3314	3198	3184	3184
query98	217	206	200	200
query99	1581	1294	1323	1294
Total cold run time: 323492 ms
Total hot run time: 198050 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97aa19ef1846aca3be98fc9c9e570d8b005820ed, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.53	0.52	0.52
query6	1.14	0.71	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.55	0.52	0.50
query10	0.57	0.56	0.55
query11	0.14	0.10	0.10
query12	0.17	0.11	0.11
query13	0.60	0.60	0.59
query14	2.96	2.94	3.12
query15	0.89	0.83	0.84
query16	0.37	0.37	0.41
query17	1.01	1.06	1.05
query18	0.24	0.22	0.22
query19	1.95	1.83	1.99
query20	0.01	0.01	0.01
query21	15.37	0.56	0.58
query22	2.42	2.06	1.80
query23	17.03	0.88	0.88
query24	3.06	0.98	1.77
query25	0.23	0.15	0.16
query26	0.31	0.15	0.14
query27	0.04	0.05	0.04
query28	10.43	1.09	1.07
query29	12.58	3.31	3.28
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.25	0.47	0.46
query33	2.98	3.00	3.03
query34	17.07	4.44	4.47
query35	4.50	4.55	4.53
query36	0.66	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.67 s
Total hot run time: 33.16 s

@airborne12 airborne12 merged commit 8d15539 into branch-3.0 Jan 14, 2025
20 of 22 checks passed
@airborne12 airborne12 deleted the auto-pick-46648-branch-3.0 branch January 14, 2025 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants