[fix](nestedtype)fix nested type with timezone datetime #45885

amorynan · 2024-12-24T15:35:27Z

What problem does this PR solve?

fix nestedtype like struct/map/array nested datetime type with timezone data for load, issue: [Bug] Stream load bug for datetime with timezone in struct #36963

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

amorynan · 2024-12-24T15:35:35Z

run buildall

hello-stephen · 2024-12-24T15:35:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2024-12-24T15:49:14Z

be/src/vec/exec/format/json/new_json_reader.cpp

@@ -1817,6 +1815,10 @@ Status NewJsonReader::_simdjson_write_data_to_column(simdjson::ondemand::value&
    } else {
        return Status::InternalError("Not support load to complex column.");
    }
+    // we should set nullmap at last to avoid column_nullable nullmap and data column size not same


seperate this fix to another PR and add test cases

introduced by #43469

doris-robot · 2024-12-24T17:19:31Z

TPC-H: Total hot run time: 33168 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f4634201e6b5f809de5d745097641a30b27fcf9, data reload: false

------ Round 1 ----------------------------------
q1	17601	6200	6196	6196
q2	2046	329	178	178
q3	10559	1290	753	753
q4	10216	879	460	460
q5	7554	2277	2063	2063
q6	214	190	149	149
q7	914	759	614	614
q8	9275	1426	1241	1241
q9	5342	5007	4951	4951
q10	6782	2337	1895	1895
q11	487	279	255	255
q12	358	380	229	229
q13	17773	3639	2967	2967
q14	247	228	221	221
q15	572	524	493	493
q16	647	619	597	597
q17	580	878	341	341
q18	7188	6579	6396	6396
q19	3095	988	617	617
q20	307	318	198	198
q21	2898	2252	2052	2052
q22	372	341	302	302
Total cold run time: 105027 ms
Total hot run time: 33168 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6426	6318	6261	6261
q2	233	328	244	244
q3	2252	2698	2333	2333
q4	1451	1847	1395	1395
q5	4375	4823	4933	4823
q6	195	181	143	143
q7	2108	2002	1886	1886
q8	2653	2821	2763	2763
q9	7326	7290	7355	7290
q10	3089	3353	2780	2780
q11	573	519	499	499
q12	728	773	624	624
q13	3375	3762	3153	3153
q14	302	328	268	268
q15	575	518	493	493
q16	651	694	658	658
q17	1261	1744	1235	1235
q18	7777	7475	7437	7437
q19	917	1022	1194	1022
q20	2005	2038	1926	1926
q21	5777	5372	4986	4986
q22	633	604	608	604
Total cold run time: 54682 ms
Total hot run time: 52823 ms

doris-robot · 2024-12-24T17:28:27Z

TeamCity be ut coverage result:
Function Coverage: 38.80% (10095/26016)
Line Coverage: 29.79% (85158/285906)
Region Coverage: 28.91% (43488/150432)
Branch Coverage: 25.45% (22175/87144)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5f4634201e6b5f809de5d745097641a30b27fcf9_5f4634201e6b5f809de5d745097641a30b27fcf9/report/index.html

doris-robot · 2024-12-24T17:31:53Z

TPC-DS: Total hot run time: 197308 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f4634201e6b5f809de5d745097641a30b27fcf9, data reload: false

query1	1309	1000	916	916
query2	6404	2478	2286	2286
query3	10976	4671	4615	4615
query4	33117	23901	23525	23525
query5	3788	629	483	483
query6	281	197	174	174
query7	3985	509	315	315
query8	321	253	237	237
query9	9469	2734	2719	2719
query10	430	309	250	250
query11	18118	15466	15216	15216
query12	155	109	106	106
query13	1590	556	422	422
query14	9865	7509	7486	7486
query15	263	205	189	189
query16	8185	597	465	465
query17	1559	792	641	641
query18	2106	434	348	348
query19	213	195	168	168
query20	125	157	115	115
query21	202	123	112	112
query22	4465	4510	4646	4510
query23	34685	33759	33617	33617
query24	6491	2342	2343	2342
query25	486	455	388	388
query26	1051	283	158	158
query27	1994	487	338	338
query28	5520	2532	2478	2478
query29	632	572	438	438
query30	214	212	165	165
query31	995	915	837	837
query32	80	61	61	61
query33	471	355	302	302
query34	805	872	567	567
query35	832	828	751	751
query36	1029	1061	979	979
query37	131	103	79	79
query38	4207	4289	4080	4080
query39	1522	1451	1460	1451
query40	213	125	102	102
query41	44	43	43	43
query42	122	105	111	105
query43	548	530	519	519
query44	1376	822	835	822
query45	191	173	169	169
query46	939	1099	672	672
query47	1999	1966	1938	1938
query48	413	428	335	335
query49	735	521	396	396
query50	689	714	413	413
query51	7337	7300	7127	7127
query52	100	104	99	99
query53	245	270	194	194
query54	519	554	432	432
query55	101	82	77	77
query56	252	265	248	248
query57	1248	1245	1178	1178
query58	252	224	231	224
query59	3293	3448	3238	3238
query60	273	274	250	250
query61	106	100	106	100
query62	907	802	770	770
query63	239	202	192	192
query64	3344	1042	671	671
query65	3383	3285	3257	3257
query66	834	407	330	330
query67	16577	15905	15468	15468
query68	9570	776	527	527
query69	512	298	257	257
query70	1175	1103	1142	1103
query71	450	327	252	252
query72	6382	3932	3951	3932
query73	675	785	363	363
query74	10205	9056	8903	8903
query75	4536	3104	2653	2653
query76	5069	1235	821	821
query77	956	381	269	269
query78	10052	10273	9476	9476
query79	3138	894	578	578
query80	741	537	442	442
query81	494	275	232	232
query82	627	155	123	123
query83	199	164	140	140
query84	281	87	67	67
query85	817	358	299	299
query86	352	323	302	302
query87	4707	4625	4366	4366
query88	3723	2309	2206	2206
query89	421	336	291	291
query90	1883	194	186	186
query91	133	150	103	103
query92	68	54	51	51
query93	1471	930	534	534
query94	654	393	289	289
query95	364	264	255	255
query96	505	620	281	281
query97	2765	2816	2668	2668
query98	222	199	195	195
query99	1758	1537	1446	1446
Total cold run time: 299255 ms
Total hot run time: 197308 ms

doris-robot · 2024-12-24T17:37:21Z

ClickBench: Total hot run time: 31.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f4634201e6b5f809de5d745097641a30b27fcf9, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.41	0.41	0.42
query6	1.15	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.48	0.52
query10	0.56	0.60	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.71	2.88	2.86
query15	0.89	0.83	0.82
query16	0.38	0.37	0.36
query17	1.03	1.00	1.06
query18	0.23	0.20	0.21
query19	1.86	1.76	1.95
query20	0.01	0.01	0.02
query21	15.37	0.96	0.60
query22	0.75	0.84	0.55
query23	15.38	1.43	0.60
query24	2.93	1.38	1.21
query25	0.13	0.28	0.08
query26	0.26	0.14	0.13
query27	0.06	0.04	0.05
query28	14.00	1.50	1.04
query29	12.63	3.93	3.30
query30	0.25	0.10	0.08
query31	2.80	0.59	0.38
query32	3.23	0.55	0.45
query33	3.09	3.09	3.15
query34	16.75	5.06	4.51
query35	4.53	4.45	4.48
query36	0.64	0.48	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.02 s
Total hot run time: 31.49 s

amorynan · 2024-12-25T02:56:59Z

run buildall

zclllyybb

LGTM

github-actions · 2024-12-25T04:26:53Z

PR approved by anyone and no changes requested.

doris-robot · 2024-12-25T04:38:38Z

TPC-H: Total hot run time: 32675 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ad656ed695bec84bb3e1810de524af4df6f08a5, data reload: false

------ Round 1 ----------------------------------
q1	17596	6346	6140	6140
q2	2057	298	171	171
q3	10764	1229	779	779
q4	10229	886	437	437
q5	7757	2176	2008	2008
q6	207	183	150	150
q7	918	750	606	606
q8	9254	1396	1168	1168
q9	5302	4875	4919	4875
q10	6754	2298	1889	1889
q11	486	289	261	261
q12	361	355	224	224
q13	17778	3632	3020	3020
q14	227	234	230	230
q15	577	507	517	507
q16	636	622	592	592
q17	594	863	325	325
q18	7234	6558	6273	6273
q19	2779	988	559	559
q20	315	313	182	182
q21	2881	2172	1968	1968
q22	355	328	311	311
Total cold run time: 105061 ms
Total hot run time: 32675 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6445	6318	6328	6318
q2	238	330	228	228
q3	2272	2684	2336	2336
q4	1381	1854	1388	1388
q5	4368	4791	4910	4791
q6	191	174	139	139
q7	2045	1987	1834	1834
q8	2688	2799	2736	2736
q9	7380	7310	7357	7310
q10	3129	3365	2774	2774
q11	577	517	487	487
q12	650	778	598	598
q13	3412	3784	3136	3136
q14	293	305	282	282
q15	576	528	499	499
q16	657	685	667	667
q17	1254	1743	1258	1258
q18	7692	7494	7222	7222
q19	889	1150	1096	1096
q20	1986	2063	1882	1882
q21	5863	5304	5044	5044
q22	652	603	587	587
Total cold run time: 54638 ms
Total hot run time: 52612 ms

github-actions · 2024-12-25T04:48:43Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-12-25T04:51:02Z

TPC-DS: Total hot run time: 197557 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ad656ed695bec84bb3e1810de524af4df6f08a5, data reload: false

query1	1317	962	937	937
query2	6492	2290	2263	2263
query3	11120	4872	4942	4872
query4	33087	24238	23491	23491
query5	4728	619	476	476
query6	320	208	193	193
query7	3995	506	304	304
query8	293	234	239	234
query9	9382	2746	2720	2720
query10	464	309	262	262
query11	17955	15454	15419	15419
query12	170	106	107	106
query13	1564	567	432	432
query14	10257	6975	7825	6975
query15	235	213	201	201
query16	7827	610	501	501
query17	1578	787	620	620
query18	2081	432	341	341
query19	237	196	177	177
query20	131	120	120	120
query21	212	136	113	113
query22	4651	4724	4420	4420
query23	34850	33723	33583	33583
query24	6563	2330	2367	2330
query25	474	484	406	406
query26	981	288	154	154
query27	2328	484	336	336
query28	5460	2488	2480	2480
query29	653	580	427	427
query30	205	190	151	151
query31	982	924	854	854
query32	73	57	56	56
query33	466	352	303	303
query34	757	876	506	506
query35	827	833	782	782
query36	1016	1070	946	946
query37	116	100	77	77
query38	4357	4140	4138	4138
query39	1580	1479	1627	1479
query40	212	124	105	105
query41	48	42	47	42
query42	120	127	108	108
query43	532	536	500	500
query44	1406	822	827	822
query45	187	176	171	171
query46	894	1070	664	664
query47	1995	1970	1948	1948
query48	394	443	330	330
query49	752	490	384	384
query50	637	687	396	396
query51	7326	7269	7315	7269
query52	109	111	97	97
query53	234	256	190	190
query54	482	495	422	422
query55	84	77	77	77
query56	268	276	276	276
query57	1250	1233	1170	1170
query58	251	225	225	225
query59	3070	3117	3040	3040
query60	278	272	256	256
query61	149	118	112	112
query62	836	805	747	747
query63	227	193	191	191
query64	3740	1029	668	668
query65	3376	3248	3249	3248
query66	774	404	310	310
query67	16628	15850	15588	15588
query68	9258	785	510	510
query69	461	283	257	257
query70	1242	1171	1111	1111
query71	423	317	254	254
query72	5798	3863	4021	3863
query73	660	755	360	360
query74	9960	9384	9230	9230
query75	4549	3182	2655	2655
query76	5243	1189	776	776
query77	1006	377	271	271
query78	10154	10180	9465	9465
query79	3729	910	596	596
query80	766	546	449	449
query81	507	283	234	234
query82	418	158	125	125
query83	194	168	157	157
query84	286	92	71	71
query85	780	361	312	312
query86	352	317	307	307
query87	4619	4549	4435	4435
query88	3593	2227	2219	2219
query89	444	343	297	297
query90	2046	189	189	189
query91	134	141	109	109
query92	73	58	52	52
query93	2151	923	519	519
query94	687	391	291	291
query95	340	307	256	256
query96	492	613	282	282
query97	2702	2857	2728	2728
query98	231	200	194	194
query99	1691	1568	1442	1442
Total cold run time: 301032 ms
Total hot run time: 197557 ms

doris-robot · 2024-12-25T04:55:29Z

TeamCity be ut coverage result:
Function Coverage: 38.82% (10099/26017)
Line Coverage: 29.82% (85261/285949)
Region Coverage: 28.93% (43521/150452)
Branch Coverage: 25.46% (22181/87138)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8ad656ed695bec84bb3e1810de524af4df6f08a5_8ad656ed695bec84bb3e1810de524af4df6f08a5/report/index.html

doris-robot · 2024-12-25T04:56:25Z

ClickBench: Total hot run time: 31.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ad656ed695bec84bb3e1810de524af4df6f08a5, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.60	0.10	0.11
query5	0.44	0.42	0.42
query6	1.16	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.49	0.50
query10	0.56	0.59	0.56
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.61	0.61	0.61
query14	2.85	2.74	2.83
query15	0.91	0.83	0.83
query16	0.37	0.38	0.40
query17	1.07	1.06	1.01
query18	0.22	0.21	0.21
query19	1.88	1.89	2.01
query20	0.01	0.02	0.01
query21	15.37	0.99	0.58
query22	0.75	0.80	0.67
query23	15.29	1.43	0.50
query24	3.62	0.79	1.74
query25	0.17	0.17	0.21
query26	0.22	0.14	0.14
query27	0.07	0.05	0.07
query28	14.44	1.51	1.06
query29	12.60	3.94	3.22
query30	0.24	0.09	0.06
query31	2.83	0.59	0.40
query32	3.23	0.56	0.45
query33	3.16	3.14	3.05
query34	16.75	5.13	4.59
query35	4.48	4.52	4.51
query36	0.65	0.48	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.33 s
Total hot run time: 31.31 s

fix nested type with timezone datetime

5f46342

eldenmoon reviewed Dec 24, 2024

View reviewed changes

recover

8ad656e

zclllyybb approved these changes Dec 25, 2024

View reviewed changes

github-actions bot added the reviewed label Dec 25, 2024

yiguolei approved these changes Dec 25, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](nestedtype)fix nested type with timezone datetime #45885

[fix](nestedtype)fix nested type with timezone datetime #45885

amorynan commented Dec 24, 2024 •

edited

Loading

amorynan commented Dec 24, 2024

hello-stephen commented Dec 24, 2024

eldenmoon Dec 24, 2024

eldenmoon Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

amorynan commented Dec 25, 2024

zclllyybb left a comment

github-actions bot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

github-actions bot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

[fix](nestedtype)fix nested type with timezone datetime #45885

Are you sure you want to change the base?

[fix](nestedtype)fix nested type with timezone datetime #45885

Conversation

amorynan commented Dec 24, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

amorynan commented Dec 24, 2024

hello-stephen commented Dec 24, 2024

eldenmoon Dec 24, 2024

Choose a reason for hiding this comment

eldenmoon Dec 24, 2024

Choose a reason for hiding this comment

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

doris-robot commented Dec 24, 2024

amorynan commented Dec 25, 2024

zclllyybb left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

github-actions bot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

doris-robot commented Dec 25, 2024

amorynan commented Dec 24, 2024 •

edited

Loading