Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](iceberg)support iceberg in batch mode #46398

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Jan 3, 2025

What problem does this PR solve?

  1. use thread pool in planFiles.
  2. support batch mode.
  3. When executing doAs, if the exception type is UndeclaredThrowableException, we should get the specific cause inside. Because the message of UndeclaredThrowableException is null, the message will not be obtained if the exception is captured externally

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@wuwenchi wuwenchi marked this pull request as draft January 3, 2025 12:56
@Thearas
Copy link
Contributor

Thearas commented Jan 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi wuwenchi force-pushed the opt-iceberg-batch-mode branch from f72acc5 to 33d6d4e Compare January 7, 2025 07:25
@wuwenchi
Copy link
Contributor Author

wuwenchi commented Jan 7, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32871 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 33d6d4e5b01a58f04ffb6c6b9344f24b47d37086, data reload: false

------ Round 1 ----------------------------------
q1	17663	6754	6036	6036
q2	2052	310	181	181
q3	10522	1232	765	765
q4	10233	865	429	429
q5	8355	2215	2037	2037
q6	206	189	146	146
q7	911	748	605	605
q8	9243	1360	1225	1225
q9	5342	4942	4953	4942
q10	6784	2328	1847	1847
q11	491	283	263	263
q12	343	372	221	221
q13	17770	3713	3046	3046
q14	239	237	217	217
q15	562	518	497	497
q16	643	611	576	576
q17	584	842	335	335
q18	7324	6528	6415	6415
q19	3625	1086	582	582
q20	306	315	188	188
q21	2857	2244	2007	2007
q22	373	346	311	311
Total cold run time: 106428 ms
Total hot run time: 32871 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6409	6266	6242	6242
q2	237	325	231	231
q3	2228	2649	2307	2307
q4	1469	1841	1383	1383
q5	4344	4822	4952	4822
q6	185	176	143	143
q7	2100	1993	1827	1827
q8	2613	2834	2695	2695
q9	7358	7328	7330	7328
q10	3074	3357	2740	2740
q11	590	514	516	514
q12	692	780	593	593
q13	3493	3799	3254	3254
q14	297	297	284	284
q15	580	512	493	493
q16	656	697	642	642
q17	1226	1760	1249	1249
q18	7789	7622	7320	7320
q19	898	1161	1110	1110
q20	2040	2099	1955	1955
q21	5722	5288	5070	5070
q22	621	638	600	600
Total cold run time: 54621 ms
Total hot run time: 52802 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197112 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 33d6d4e5b01a58f04ffb6c6b9344f24b47d37086, data reload: false

query1	1306	945	914	914
query2	6480	2321	2316	2316
query3	10994	4719	4658	4658
query4	32706	23869	23910	23869
query5	4133	611	450	450
query6	292	205	196	196
query7	4001	510	316	316
query8	319	252	234	234
query9	9395	2787	2769	2769
query10	459	317	251	251
query11	18008	15711	15178	15178
query12	170	113	107	107
query13	1578	557	404	404
query14	9907	6751	8496	6751
query15	249	210	206	206
query16	7973	587	457	457
query17	1612	793	609	609
query18	2004	391	322	322
query19	213	187	163	163
query20	126	121	140	121
query21	208	127	106	106
query22	4674	4657	4515	4515
query23	34396	34922	33631	33631
query24	6342	2377	2317	2317
query25	455	443	380	380
query26	908	264	160	160
query27	2038	491	337	337
query28	5039	2542	2504	2504
query29	561	562	427	427
query30	210	190	151	151
query31	973	911	857	857
query32	73	62	58	58
query33	525	350	291	291
query34	806	862	544	544
query35	834	828	771	771
query36	1047	1059	958	958
query37	124	105	74	74
query38	4257	4334	4308	4308
query39	1538	1515	1455	1455
query40	218	130	108	108
query41	47	44	48	44
query42	129	107	105	105
query43	533	540	502	502
query44	1369	819	837	819
query45	183	180	172	172
query46	936	1066	642	642
query47	1999	1974	1945	1945
query48	388	429	344	344
query49	745	489	421	421
query50	714	701	398	398
query51	7229	7090	7031	7031
query52	103	106	100	100
query53	242	260	188	188
query54	495	503	409	409
query55	84	76	77	76
query56	273	260	241	241
query57	1247	1247	1189	1189
query58	247	223	227	223
query59	3230	3327	3126	3126
query60	282	266	255	255
query61	110	110	106	106
query62	861	853	748	748
query63	236	205	191	191
query64	3410	1018	643	643
query65	3264	3305	3279	3279
query66	769	448	313	313
query67	16729	15821	15591	15591
query68	8108	705	523	523
query69	459	284	255	255
query70	1221	1139	1115	1115
query71	438	282	249	249
query72	6518	3834	3751	3751
query73	668	767	365	365
query74	10238	8973	9232	8973
query75	3913	3179	2653	2653
query76	3714	1194	770	770
query77	766	362	269	269
query78	10173	10233	9456	9456
query79	3497	824	580	580
query80	700	521	415	415
query81	487	268	221	221
query82	564	157	125	125
query83	194	162	144	144
query84	276	96	73	73
query85	790	384	294	294
query86	353	324	307	307
query87	4580	4687	4588	4588
query88	3790	2330	2221	2221
query89	428	338	300	300
query90	1782	189	209	189
query91	135	131	104	104
query92	65	58	52	52
query93	1203	862	529	529
query94	638	410	250	250
query95	325	260	249	249
query96	485	621	286	286
query97	2960	3030	2838	2838
query98	214	197	193	193
query99	1692	1552	1431	1431
Total cold run time: 294379 ms
Total hot run time: 197112 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.32 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 33d6d4e5b01a58f04ffb6c6b9344f24b47d37086, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.43	0.41	0.41
query6	1.16	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.52	0.51
query10	0.57	0.56	0.55
query11	0.15	0.10	0.11
query12	0.14	0.11	0.10
query13	0.61	0.60	0.61
query14	2.84	2.82	2.85
query15	0.89	0.82	0.82
query16	0.38	0.38	0.38
query17	0.98	0.97	1.00
query18	0.22	0.21	0.21
query19	1.86	1.81	2.00
query20	0.01	0.01	0.01
query21	15.36	0.92	0.60
query22	0.75	0.88	0.67
query23	15.19	1.45	0.58
query24	3.10	0.89	2.08
query25	0.21	0.20	0.10
query26	0.27	0.15	0.14
query27	0.05	0.05	0.04
query28	13.69	1.50	1.05
query29	12.56	4.13	3.37
query30	0.25	0.09	0.06
query31	2.80	0.58	0.39
query32	3.23	0.54	0.46
query33	3.13	3.14	3.07
query34	16.70	5.07	4.47
query35	4.55	4.42	4.48
query36	0.63	0.49	0.49
query37	0.09	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.78 s
Total hot run time: 31.32 s

@wuwenchi
Copy link
Contributor Author

wuwenchi commented Jan 8, 2025

run buildall

@wuwenchi wuwenchi marked this pull request as ready for review January 8, 2025 06:28
@doris-robot
Copy link

TPC-H: Total hot run time: 32391 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b10ea0ecb7c7dfa8b2603f32a4f30ba97cf3802b, data reload: false

------ Round 1 ----------------------------------
q1	17624	6166	6018	6018
q2	2052	289	169	169
q3	10434	1246	710	710
q4	10211	848	424	424
q5	7523	2174	1967	1967
q6	210	185	145	145
q7	873	734	616	616
q8	9222	1311	1143	1143
q9	5167	4885	4942	4885
q10	6722	2288	1825	1825
q11	453	273	254	254
q12	360	343	216	216
q13	17775	3583	3108	3108
q14	228	243	213	213
q15	568	510	507	507
q16	629	606	556	556
q17	543	838	319	319
q18	7181	6612	6320	6320
q19	1242	960	560	560
q20	315	321	189	189
q21	2827	2132	1940	1940
q22	366	332	307	307
Total cold run time: 102525 ms
Total hot run time: 32391 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6201	6179	6230	6179
q2	241	331	227	227
q3	2272	2594	2342	2342
q4	1429	1828	1347	1347
q5	4295	4707	4677	4677
q6	190	184	139	139
q7	2105	1997	1851	1851
q8	2636	2791	2642	2642
q9	7290	7151	7217	7151
q10	3015	3378	2806	2806
q11	582	518	499	499
q12	658	737	583	583
q13	3437	3833	3181	3181
q14	291	319	280	280
q15	565	522	511	511
q16	649	671	636	636
q17	1200	1686	1251	1251
q18	7654	7427	7108	7108
q19	820	1039	1105	1039
q20	1872	1966	1888	1888
q21	5449	5008	4794	4794
q22	618	614	583	583
Total cold run time: 53469 ms
Total hot run time: 51714 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187196 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b10ea0ecb7c7dfa8b2603f32a4f30ba97cf3802b, data reload: false

query1	989	398	357	357
query2	6519	2348	2296	2296
query3	6711	214	216	214
query4	33656	23378	23216	23216
query5	4332	643	466	466
query6	294	199	197	197
query7	4638	489	305	305
query8	299	257	244	244
query9	9452	2720	2709	2709
query10	494	326	254	254
query11	17806	15418	14992	14992
query12	176	113	109	109
query13	1679	555	419	419
query14	11064	6944	6489	6489
query15	239	203	190	190
query16	8192	575	432	432
query17	1575	732	558	558
query18	2063	388	293	293
query19	211	180	147	147
query20	114	108	108	108
query21	209	122	127	122
query22	4321	4334	4169	4169
query23	34119	32990	32881	32881
query24	6365	2257	2264	2257
query25	498	448	385	385
query26	1222	270	146	146
query27	2031	451	333	333
query28	5485	2445	2436	2436
query29	732	547	415	415
query30	225	176	149	149
query31	956	860	780	780
query32	92	60	62	60
query33	524	342	287	287
query34	745	810	501	501
query35	764	788	767	767
query36	942	1007	948	948
query37	122	93	78	78
query38	4230	4030	4010	4010
query39	1469	1433	1411	1411
query40	206	110	98	98
query41	46	44	45	44
query42	118	97	99	97
query43	513	528	490	490
query44	1281	812	807	807
query45	179	165	161	161
query46	842	1011	649	649
query47	1798	1840	1783	1783
query48	392	422	320	320
query49	769	466	391	391
query50	609	645	388	388
query51	6998	7085	6957	6957
query52	95	95	91	91
query53	222	247	183	183
query54	462	475	398	398
query55	80	81	78	78
query56	263	254	238	238
query57	1178	1107	1090	1090
query58	236	222	226	222
query59	3127	3197	2798	2798
query60	277	276	246	246
query61	114	113	110	110
query62	815	760	722	722
query63	217	201	197	197
query64	4560	980	641	641
query65	3235	3152	3219	3152
query66	1040	405	298	298
query67	15706	15644	15306	15306
query68	8287	695	523	523
query69	458	287	252	252
query70	1097	1166	1111	1111
query71	455	282	281	281
query72	6444	3867	3825	3825
query73	656	731	360	360
query74	9493	9148	8673	8673
query75	4041	3128	2625	2625
query76	3704	1161	752	752
query77	765	357	283	283
query78	9817	9983	9442	9442
query79	3785	807	588	588
query80	698	507	429	429
query81	504	267	230	230
query82	616	148	131	131
query83	209	160	148	148
query84	236	88	74	74
query85	791	347	308	308
query86	400	319	285	285
query87	4257	4301	4377	4301
query88	4797	2200	2178	2178
query89	400	321	287	287
query90	1829	186	188	186
query91	130	141	105	105
query92	73	57	54	54
query93	2095	848	521	521
query94	658	389	298	298
query95	342	268	253	253
query96	477	601	283	283
query97	2813	2934	2763	2763
query98	230	207	192	192
query99	1433	1540	1370	1370
Total cold run time: 292922 ms
Total hot run time: 187196 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b10ea0ecb7c7dfa8b2603f32a4f30ba97cf3802b, data reload: false

query1	0.05	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.61	0.11	0.10
query5	0.44	0.41	0.42
query6	1.17	0.64	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.50	0.50
query10	0.53	0.56	0.54
query11	0.15	0.09	0.10
query12	0.14	0.11	0.12
query13	0.62	0.62	0.59
query14	2.72	2.72	2.73
query15	0.90	0.82	0.83
query16	0.41	0.42	0.38
query17	1.09	1.04	1.08
query18	0.21	0.22	0.21
query19	1.98	1.87	1.98
query20	0.02	0.02	0.01
query21	15.40	0.96	0.58
query22	0.75	0.74	0.72
query23	15.31	1.38	0.58
query24	2.52	0.68	1.45
query25	0.20	0.25	0.07
query26	0.23	0.15	0.14
query27	0.06	0.04	0.04
query28	13.81	1.57	1.05
query29	12.58	3.94	3.21
query30	0.25	0.09	0.06
query31	2.82	0.61	0.39
query32	3.23	0.56	0.47
query33	3.16	3.12	3.11
query34	16.71	5.19	4.49
query35	4.52	4.51	4.53
query36	0.64	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.68 s
Total hot run time: 31.06 s

@wuwenchi wuwenchi force-pushed the opt-iceberg-batch-mode branch from b10ea0e to 4dd49a2 Compare January 13, 2025 02:27
@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33404 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4dd49a253d8b7e453e91ea8b717c17936e343bfa, data reload: false

------ Round 1 ----------------------------------
q1	17591	6246	6185	6185
q2	2042	333	176	176
q3	10376	1304	751	751
q4	10211	899	467	467
q5	7525	2267	2092	2092
q6	208	186	151	151
q7	918	786	638	638
q8	9250	1433	1306	1306
q9	5322	4930	5026	4930
q10	6793	2306	1884	1884
q11	511	290	257	257
q12	353	377	227	227
q13	17766	3721	3069	3069
q14	244	247	224	224
q15	566	524	497	497
q16	644	620	596	596
q17	615	884	352	352
q18	7138	6501	6452	6452
q19	1211	976	579	579
q20	326	341	197	197
q21	3136	2285	2061	2061
q22	385	355	313	313
Total cold run time: 103131 ms
Total hot run time: 33404 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6339	6288	6283	6283
q2	244	340	244	244
q3	2286	2685	2346	2346
q4	1491	1831	1449	1449
q5	4390	4801	4974	4801
q6	206	184	144	144
q7	2141	1997	1884	1884
q8	2620	2816	2760	2760
q9	7367	7188	7274	7188
q10	3046	3297	2858	2858
q11	605	524	503	503
q12	696	781	614	614
q13	3477	3808	3216	3216
q14	282	302	275	275
q15	571	505	515	505
q16	659	703	633	633
q17	1277	1736	1266	1266
q18	7714	7480	6965	6965
q19	849	916	1163	916
q20	1868	1994	1776	1776
q21	5504	5221	5025	5025
q22	609	616	569	569
Total cold run time: 54241 ms
Total hot run time: 52220 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187592 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4dd49a253d8b7e453e91ea8b717c17936e343bfa, data reload: false

query1	988	383	377	377
query2	6513	2340	2393	2340
query3	6712	224	222	222
query4	34003	23348	23162	23162
query5	4418	623	438	438
query6	298	220	206	206
query7	4622	490	309	309
query8	305	245	242	242
query9	9654	2754	2731	2731
query10	462	315	249	249
query11	17637	15009	15072	15009
query12	153	105	103	103
query13	1642	502	373	373
query14	9553	7078	6533	6533
query15	263	202	191	191
query16	7450	612	439	439
query17	1589	719	559	559
query18	1965	370	290	290
query19	203	176	152	152
query20	116	116	115	115
query21	206	119	104	104
query22	4463	4419	4186	4186
query23	33653	32836	33326	32836
query24	6479	2259	2301	2259
query25	546	461	429	429
query26	1252	269	151	151
query27	2390	431	325	325
query28	5708	2432	2421	2421
query29	667	536	423	423
query30	226	186	163	163
query31	958	863	790	790
query32	101	63	60	60
query33	581	358	306	306
query34	740	833	502	502
query35	774	799	707	707
query36	967	1013	996	996
query37	123	100	80	80
query38	4272	4034	4001	4001
query39	1515	1421	1414	1414
query40	200	113	99	99
query41	55	49	55	49
query42	124	104	103	103
query43	512	542	504	504
query44	1300	818	826	818
query45	178	169	168	168
query46	842	1027	625	625
query47	1829	1848	1773	1773
query48	385	395	311	311
query49	782	483	398	398
query50	615	647	377	377
query51	6952	6876	6699	6699
query52	101	102	91	91
query53	220	255	181	181
query54	475	497	405	405
query55	79	76	80	76
query56	254	269	240	240
query57	1164	1117	1085	1085
query58	237	239	242	239
query59	3064	3051	2822	2822
query60	277	261	247	247
query61	119	123	123	123
query62	890	795	701	701
query63	225	192	187	187
query64	4787	992	666	666
query65	3253	3144	3194	3144
query66	1067	413	324	324
query67	15858	15527	15405	15405
query68	8257	698	515	515
query69	459	287	257	257
query70	1223	1121	1109	1109
query71	448	272	250	250
query72	6165	3853	3880	3853
query73	654	757	362	362
query74	10178	9006	8969	8969
query75	4085	3124	2643	2643
query76	3574	1193	764	764
query77	796	364	275	275
query78	9887	9957	9406	9406
query79	2497	788	602	602
query80	594	510	436	436
query81	515	276	243	243
query82	454	148	118	118
query83	175	175	159	159
query84	312	93	75	75
query85	788	358	294	294
query86	414	317	305	305
query87	4273	4387	4373	4373
query88	5018	2186	2144	2144
query89	395	325	293	293
query90	1863	188	191	188
query91	133	140	103	103
query92	72	58	54	54
query93	1885	868	527	527
query94	667	405	275	275
query95	343	261	261	261
query96	483	616	283	283
query97	2831	2960	2835	2835
query98	228	204	199	199
query99	1414	1508	1369	1369
Total cold run time: 291091 ms
Total hot run time: 187592 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4dd49a253d8b7e453e91ea8b717c17936e343bfa, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.40	0.43	0.41
query6	1.14	0.64	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.50	0.52
query10	0.55	0.56	0.54
query11	0.14	0.10	0.11
query12	0.14	0.10	0.10
query13	0.61	0.61	0.60
query14	2.84	2.77	2.75
query15	0.92	0.83	0.82
query16	0.39	0.39	0.35
query17	0.99	1.02	1.08
query18	0.23	0.22	0.19
query19	1.94	1.78	2.02
query20	0.01	0.01	0.01
query21	15.38	0.91	0.56
query22	0.75	0.72	0.64
query23	15.47	1.43	0.53
query24	2.94	1.84	1.80
query25	0.20	0.24	0.06
query26	0.19	0.14	0.13
query27	0.07	0.05	0.04
query28	14.57	1.46	1.05
query29	12.60	3.99	3.22
query30	0.24	0.09	0.07
query31	2.82	0.59	0.38
query32	3.24	0.54	0.46
query33	3.09	3.15	3.09
query34	16.90	5.00	4.47
query35	4.49	4.45	4.49
query36	0.64	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 106.97 s
Total hot run time: 31.82 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants