Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](maxcompute)support maxcompute timestamp column type. #48768

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Mar 6, 2025

What problem does this PR solve?

Problem Summary:

  1. support maxcompute timestamp column type.
  2. Add the parameter mc.datetime_predicate_push_down to disable predicate pushdown for odps catalog datetime type, because the timestamp precision of odps is 9, while the mapping precision of Doris is 6. If we insert 2023-02-02 00:00:00.123456789 into odps, doris will read it as 2023-02-02 00:00:00.123456. Due to the lack of "789", we cannot push it down correctly. If you don't need such a high precision (greater than 6) on odps, it will not affect your normal use.

Release note

support maxcompute timestamp column type.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Mar 6, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32802 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4350c675e0aa4f29221952f84a140c383b495707, data reload: false

------ Round 1 ----------------------------------
q1	17591	5147	5196	5147
q2	2062	300	166	166
q3	10406	1337	722	722
q4	10225	1054	526	526
q5	8178	2505	2423	2423
q6	198	172	141	141
q7	931	764	611	611
q8	9330	1264	1144	1144
q9	4928	4842	4862	4842
q10	6825	2329	1903	1903
q11	476	287	258	258
q12	351	368	216	216
q13	17746	3735	3094	3094
q14	228	231	210	210
q15	550	496	475	475
q16	616	614	567	567
q17	599	874	355	355
q18	7011	6652	6271	6271
q19	1210	937	568	568
q20	325	332	203	203
q21	2907	2140	1973	1973
q22	1075	1043	987	987
Total cold run time: 103768 ms
Total hot run time: 32802 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5111	5111	5115	5111
q2	248	338	242	242
q3	2164	2728	2311	2311
q4	1472	1804	1397	1397
q5	4243	4144	4201	4144
q6	205	168	126	126
q7	1956	1988	1755	1755
q8	2661	2684	2569	2569
q9	7132	7169	7164	7164
q10	3002	3203	2748	2748
q11	577	517	498	498
q12	680	792	640	640
q13	3517	4285	3232	3232
q14	294	302	273	273
q15	512	488	467	467
q16	641	695	653	653
q17	1124	1621	1338	1338
q18	7773	7575	7465	7465
q19	879	937	962	937
q20	1956	2017	1888	1888
q21	5445	4924	4955	4924
q22	1093	1042	996	996
Total cold run time: 52685 ms
Total hot run time: 50878 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185488 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4350c675e0aa4f29221952f84a140c383b495707, data reload: false

query1	983	386	387	386
query2	6545	1938	1936	1936
query3	6796	215	217	215
query4	26315	23705	22929	22929
query5	4391	665	488	488
query6	316	205	203	203
query7	4605	505	299	299
query8	304	251	235	235
query9	8633	2581	2586	2581
query10	494	318	259	259
query11	15378	15196	14932	14932
query12	161	106	110	106
query13	1663	532	408	408
query14	8763	6637	6692	6637
query15	208	200	183	183
query16	7682	678	480	480
query17	1209	721	566	566
query18	1996	404	308	308
query19	195	192	164	164
query20	123	116	116	116
query21	208	121	104	104
query22	4284	4352	4299	4299
query23	34241	32927	32966	32927
query24	7687	2382	2414	2382
query25	526	451	402	402
query26	1229	263	150	150
query27	2203	502	331	331
query28	3955	2397	2415	2397
query29	730	567	420	420
query30	283	216	185	185
query31	919	863	785	785
query32	74	64	63	63
query33	563	346	349	346
query34	784	866	519	519
query35	817	802	720	720
query36	943	1004	897	897
query37	120	99	74	74
query38	4078	4029	4110	4029
query39	1423	1369	1395	1369
query40	205	119	103	103
query41	55	57	56	56
query42	129	110	105	105
query43	516	541	493	493
query44	1298	800	785	785
query45	174	169	163	163
query46	840	1016	634	634
query47	1781	1778	1705	1705
query48	376	419	294	294
query49	761	509	433	433
query50	674	733	413	413
query51	4166	4178	4136	4136
query52	117	116	97	97
query53	231	261	190	190
query54	488	490	416	416
query55	94	82	80	80
query56	262	262	255	255
query57	1129	1125	1082	1082
query58	244	244	249	244
query59	2606	2730	2741	2730
query60	291	269	314	269
query61	144	120	123	120
query62	807	746	674	674
query63	235	193	198	193
query64	4292	997	713	713
query65	4388	4318	4334	4318
query66	1100	404	309	309
query67	15800	15691	15099	15099
query68	8120	881	513	513
query69	468	300	273	273
query70	1202	1104	1125	1104
query71	459	293	275	275
query72	5590	3561	3721	3561
query73	759	720	349	349
query74	8925	9095	8741	8741
query75	3810	3157	2721	2721
query76	3717	1185	749	749
query77	791	368	289	289
query78	10051	9975	9295	9295
query79	2866	889	590	590
query80	709	527	452	452
query81	478	268	228	228
query82	674	135	99	99
query83	215	175	161	161
query84	286	95	76	76
query85	774	420	293	293
query86	339	323	274	274
query87	4502	4423	4392	4392
query88	3246	2272	2264	2264
query89	397	335	289	289
query90	1920	213	215	213
query91	135	143	115	115
query92	79	63	59	59
query93	1244	1027	584	584
query94	668	417	296	296
query95	354	271	273	271
query96	482	566	280	280
query97	3298	3354	3284	3284
query98	227	212	204	204
query99	1461	1422	1255	1255
Total cold run time: 273276 ms
Total hot run time: 185488 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4350c675e0aa4f29221952f84a140c383b495707, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.60	0.10	0.10
query5	0.56	0.56	0.57
query6	1.22	0.73	0.70
query7	0.02	0.02	0.02
query8	0.03	0.04	0.03
query9	0.58	0.53	0.51
query10	0.57	0.58	0.55
query11	0.16	0.11	0.11
query12	0.15	0.11	0.10
query13	0.62	0.60	0.59
query14	2.66	2.68	2.66
query15	0.93	0.86	0.85
query16	0.38	0.39	0.37
query17	1.02	1.03	1.02
query18	0.20	0.19	0.19
query19	1.93	1.82	1.98
query20	0.01	0.01	0.01
query21	15.38	0.92	0.54
query22	0.75	1.14	0.66
query23	15.00	1.38	0.65
query24	7.05	1.33	1.20
query25	0.53	0.25	0.13
query26	0.65	0.16	0.14
query27	0.05	0.05	0.05
query28	10.26	0.91	0.43
query29	12.60	4.15	3.39
query30	0.25	0.09	0.07
query31	2.82	0.60	0.38
query32	3.22	0.55	0.46
query33	3.04	3.01	3.05
query34	15.79	5.12	4.51
query35	4.57	4.59	4.55
query36	0.66	0.49	0.48
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.12 s
Total hot run time: 31.29 s

@hubgeter
Copy link
Contributor Author

hubgeter commented Mar 6, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32230 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 080377dd04a640ae7358c27711d63d6bbb9493b6, data reload: false

------ Round 1 ----------------------------------
q1	17595	5101	5011	5011
q2	2045	291	166	166
q3	10432	1235	724	724
q4	10240	1022	518	518
q5	7682	2362	2319	2319
q6	189	161	133	133
q7	889	735	602	602
q8	9292	1276	1052	1052
q9	4935	4856	4844	4844
q10	6813	2289	1871	1871
q11	459	271	250	250
q12	338	346	215	215
q13	17748	3631	3043	3043
q14	229	225	207	207
q15	534	494	489	489
q16	641	601	607	601
q17	574	848	334	334
q18	6768	6496	6242	6242
q19	1222	950	521	521
q20	321	326	194	194
q21	2724	2108	1918	1918
q22	1047	1018	976	976
Total cold run time: 102717 ms
Total hot run time: 32230 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5079	5048	5059	5048
q2	230	325	224	224
q3	2157	2701	2281	2281
q4	1429	1813	1360	1360
q5	4252	4114	4099	4099
q6	251	164	122	122
q7	1869	1801	1766	1766
q8	2640	2660	2555	2555
q9	7266	7221	7323	7221
q10	3032	3257	2752	2752
q11	562	523	480	480
q12	707	787	582	582
q13	3448	3948	3315	3315
q14	270	283	270	270
q15	518	486	474	474
q16	665	699	651	651
q17	1142	1595	1312	1312
q18	7908	7632	7524	7524
q19	797	877	1126	877
q20	2002	2049	1878	1878
q21	5304	4824	4814	4814
q22	1123	1066	1026	1026
Total cold run time: 52651 ms
Total hot run time: 50631 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192222 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 080377dd04a640ae7358c27711d63d6bbb9493b6, data reload: false

query1	1350	993	996	993
query2	6095	1967	1949	1949
query3	10980	4381	4469	4381
query4	55040	25786	23288	23288
query5	5131	564	512	512
query6	377	199	192	192
query7	4923	504	293	293
query8	319	251	236	236
query9	5779	2573	2587	2573
query10	459	316	243	243
query11	15094	15115	14880	14880
query12	161	107	111	107
query13	1065	512	407	407
query14	10246	6598	6642	6598
query15	210	194	185	185
query16	7086	649	489	489
query17	1103	731	602	602
query18	1559	428	317	317
query19	201	197	175	175
query20	127	132	126	126
query21	210	129	110	110
query22	4593	4684	4643	4643
query23	34019	33108	33317	33108
query24	5756	2477	2441	2441
query25	471	481	418	418
query26	731	285	154	154
query27	1888	487	336	336
query28	2934	2488	2447	2447
query29	561	563	426	426
query30	276	221	189	189
query31	866	900	830	830
query32	72	64	89	64
query33	492	361	302	302
query34	778	855	506	506
query35	813	830	778	778
query36	965	1009	928	928
query37	126	98	77	77
query38	4248	4292	4129	4129
query39	1530	1433	1452	1433
query40	205	111	101	101
query41	53	48	49	48
query42	126	111	107	107
query43	525	531	514	514
query44	1338	801	817	801
query45	192	177	165	165
query46	869	1032	638	638
query47	1847	1922	1812	1812
query48	395	447	303	303
query49	701	555	439	439
query50	701	756	413	413
query51	4320	4296	4288	4288
query52	106	107	98	98
query53	240	258	192	192
query54	486	532	445	445
query55	82	87	78	78
query56	283	256	257	256
query57	1186	1198	1096	1096
query58	246	238	244	238
query59	3022	2968	2740	2740
query60	299	283	274	274
query61	118	120	121	120
query62	703	746	664	664
query63	231	196	198	196
query64	2130	1055	675	675
query65	4577	4482	4364	4364
query66	757	399	295	295
query67	16342	15892	15294	15294
query68	6860	883	508	508
query69	549	299	269	269
query70	1197	1149	1100	1100
query71	497	313	277	277
query72	5907	3631	3836	3631
query73	1495	740	344	344
query74	9336	8972	9054	8972
query75	3857	3160	2685	2685
query76	4113	1174	744	744
query77	608	365	279	279
query78	10018	10067	9342	9342
query79	2996	831	600	600
query80	653	547	440	440
query81	480	258	224	224
query82	534	123	101	101
query83	275	169	152	152
query84	281	94	71	71
query85	791	351	308	308
query86	381	308	273	273
query87	4463	4511	4321	4321
query88	3734	2217	2199	2199
query89	407	310	281	281
query90	1840	203	201	201
query91	142	137	106	106
query92	71	60	57	57
query93	1856	1059	588	588
query94	654	385	299	299
query95	358	263	255	255
query96	488	562	274	274
query97	3340	3405	3322	3322
query98	226	206	208	206
query99	1434	1371	1274	1274
Total cold run time: 300555 ms
Total hot run time: 192222 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 080377dd04a640ae7358c27711d63d6bbb9493b6, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.23	0.06	0.07
query4	1.62	0.10	0.10
query5	0.57	0.57	0.56
query6	1.18	0.71	0.72
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.60	0.52	0.51
query10	0.60	0.57	0.60
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.59	0.61
query14	2.65	2.71	2.66
query15	0.91	0.85	0.85
query16	0.39	0.38	0.38
query17	1.06	1.03	1.00
query18	0.20	0.20	0.19
query19	1.99	1.96	1.78
query20	0.01	0.01	0.02
query21	15.39	0.87	0.55
query22	0.74	1.17	0.66
query23	14.94	1.37	0.65
query24	7.50	0.65	1.09
query25	0.48	0.29	0.09
query26	0.56	0.16	0.13
query27	0.05	0.05	0.05
query28	9.35	0.91	0.44
query29	12.56	4.08	3.40
query30	0.25	0.08	0.06
query31	2.82	0.60	0.39
query32	3.23	0.55	0.47
query33	2.97	3.00	3.00
query34	15.84	5.18	4.47
query35	4.61	4.55	4.54
query36	0.68	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.56 s
Total hot run time: 30.65 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Mar 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 7, 2025
Copy link
Contributor

github-actions bot commented Mar 7, 2025

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants