Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](exectutor)Fix topicPublisher thread may NPE when no topic exists #29306

Merged
merged 1 commit into from
Dec 29, 2023
Merged

[Fix](exectutor)Fix topicPublisher thread may NPE when no topic exists #29306

merged 1 commit into from
Dec 29, 2023

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Dec 29, 2023 •
edited
Loading

1 How NPE happens?

 public int getTopicMapSize() {
    return (this.topic_map == null) ? 0 : this.topic_map.size();
  }

  public void putToTopicMap(TTopicInfoType key, java.util.List<TopicInfo> val) {
    if (this.topic_map == null) {
      this.topic_map = new java.util.EnumMap<TTopicInfoType,java.util.List<TopicInfo>>(TTopicInfoType.class);
    }
    this.topic_map.put(key, val);
  }

  @org.apache.thrift.annotation.Nullable
  public java.util.Map<TTopicInfoType,java.util.List<TopicInfo>> getTopicMap() {
    return this.topic_map;
  }

When visiting A thrift Class's collection size, should use getXXXsize(), should not first getXXX then get size, because it may be a null.

2 how to fix?
Currently even request has no topicinfo list, it still needs send an empty rpc to be, because empty may means delete.

@wangbo
Copy link
Contributor Author

wangbo commented Dec 29, 2023

run buildall

yiguolei
yiguolei previously approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 60481a33f5721b032b4169dedea54bdc2c7423e7, data reload: false

------ Round 1 ----------------------------------
q1	17638	5188	5170	5170
q2	2017	162	154	154
q3	10529	1134	1068	1068
q4	10180	821	826	821
q5	7795	2990	2926	2926
q6	218	137	134	134
q7	967	520	569	520
q8	9320	2040	2042	2040
q9	6885	6486	6361	6361
q10	8285	3103	3078	3078
q11	445	235	215	215
q12	402	245	238	238
q13	18003	3632	3664	3632
q14	246	219	209	209
q15	595	563	543	543
q16	513	401	415	401
q17	967	532	511	511
q18	7537	6897	6756	6756
q19	1607	1406	1312	1312
q20	741	372	360	360
q21	2840	2375	2436	2375
q22	376	323	325	323
Total cold run time: 108106 ms
Total hot run time: 39147 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5194	5110	5143	5110
q2	341	262	255	255
q3	3341	3343	3323	3323
q4	2160	1995	2025	1995
q5	5819	5815	5873	5815
q6	221	128	131	128
q7	2348	1934	1983	1934
q8	3429	3466	3468	3466
q9	8914	8828	8842	8828
q10	3818	3889	3920	3889
q11	597	477	498	477
q12	801	638	636	636
q13	11997	3227	3206	3206
q14	312	279	284	279
q15	616	555	561	555
q16	566	510	508	508
q17	1971	1816	1771	1771
q18	8735	8503	8408	8408
q19	1644	1627	1638	1627
q20	2199	1986	1994	1986
q21	5564	5299	5340	5299
q22	555	483	440	440
Total cold run time: 71142 ms
Total hot run time: 59935 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 60481a33f5721b032b4169dedea54bdc2c7423e7, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5467	5154	5146	5146
q2	398	177	164	164
q3	1471	1213	1230	1213
q4	1099	846	820	820
q5	3158	3062	3098	3062
q6	229	138	136	136
q7	978	569	532	532
q8	2158	2195	2248	2195
q9	6686	6716	6654	6654
q10	3182	3152	3166	3152
q11	348	234	221	221
q12	384	246	247	246
q13	4418	3654	3647	3647
q14	260	229	214	214
q15	625	589	571	571
q16	458	401	412	401
q17	1036	518	611	518
q18	7101	6830	6806	6806
q19	1648	1605	1560	1560
q20	626	383	355	355
q21	2854	2485	2451	2451
q22	384	313	329	313
Total cold run time: 44968 ms
Total hot run time: 40377 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5058	5121	5082	5082
q2	350	240	244	240
q3	3470	3310	3276	3276
q4	2151	1999	2031	1999
q5	5932	5932	5951	5932
q6	234	133	128	128
q7	2388	1955	1951	1951
q8	3552	3645	3667	3645
q9	9032	9015	9027	9015
q10	3865	3950	3919	3919
q11	599	511	472	472
q12	808	642	645	642
q13	3865	3212	3221	3212
q14	299	276	278	276
q15	624	556	559	556
q16	563	517	506	506
q17	2028	1820	1800	1800
q18	8774	8438	8374	8374
q19	1750	1715	1703	1703
q20	2278	2011	1983	1983
q21	5773	5252	5282	5252
q22	564	517	498	498
Total cold run time: 63957 ms
Total hot run time: 60461 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 60481a33f5721b032b4169dedea54bdc2c7423e7, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	933	363	341	341
query2	3655	2021	2059	2021
query3	5573	230	221	221
query4	24049	22463	22466	22463
query5	4259	590	570	570
query6	276	212	187	187
query7	3965	292	282	282
query8	250	220	213	213
query9	7147	2777	2677	2677
query10	357	287	282	282
query11	16416	15590	15658	15590
query12	138	82	77	77
query13	1199	335	327	327
query14	10742	7670	7756	7670
query15	350	218	229	218
query16	5429	304	292	292
query17	1621	528	535	528
query18	1929	274	294	274
query19	262	160	161	160
query20	87	79	83	79
query21	187	105	106	105
query22	4916	4902	4864	4864
query23	32296	31716	31655	31655
query24	11915	3502	3523	3502
query25	608	383	392	383
query26	1248	164	162	162
query27	2691	292	296	292
query28	5965	2004	1981	1981
query29	1438	441	443	441
query30	309	148	155	148
query31	1074	821	823	821
query32	102	67	61	61
query33	692	293	286	286
query34	845	475	447	447
query35	1000	807	776	776
query36	1460	1220	1238	1220
query37	199	88	91	88
query38	3434	3330	3315	3315
query39	1360	1321	1304	1304
query40	308	102	100	100
query41	39	38	36	36
query42	97	91	106	91
query43	527	473	519	473
query44	1109	767	778	767
query45	236	229	220	220
query46	1194	820	779	779
query47	1751	1601	1569	1569
query48	379	282	284	282
query49	1206	342	341	341
query50	1077	932	850	850
query51	5473	5332	5205	5205
query52	106	86	94	86
query53	218	196	185	185
query54	1694	689	658	658
query55	106	89	95	89
query56	223	204	213	204
query57	1040	980	959	959
query58	239	224	225	224
query59	2857	2753	2622	2622
query60	267	235	243	235
query61	83	81	86	81
query62	1354	556	566	556
query63	197	184	184	184
query64	5563	1817	1786	1786
query65	3371	3307	3281	3281
query66	1059	339	335	335
query67	15435	15472	15774	15472
query68	12044	544	523	523
query69	631	271	265	265
query70	1628	1615	1529	1529
query71	575	231	237	231
query72	5602	4119	4154	4119
query73	1384	329	323	323
query74	6874	6570	6522	6522
query75	5649	2312	2282	2282
query76	4841	1185	1174	1174
query77	890	306	316	306
query78	9108	8620	8526	8526
query79	3566	536	530	530
query80	2878	396	380	380
query81	538	219	223	219
query82	272	127	120	120
query83	310	142	142	142
query84	255	60	56	56
query85	1844	293	282	282
query86	424	385	378	378
query87	3659	3446	3376	3376
query88	3401	2425	2448	2425
query89	369	302	302	302
query90	1772	246	259	246
query91	125	96	96	96
query92	75	54	57	54
query93	3036	446	488	446
query94	726	223	212	212
query95	519	471	472	471
query96	652	345	336	336
query97	4307	4194	4186	4186
query98	234	201	191	191
query99	1362	901	969	901
Total cold run time: 287134 ms
Total hot run time: 185142 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.81 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17188925447 Bytes

@wangbo wangbo force-pushed the 1229_fix_topic_publisher_npe branch from 60481a3 to 6d72649 Compare December 29, 2023 11:58
@wangbo
Copy link
Contributor Author

wangbo commented Dec 29, 2023

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 6d72649bbcd9f717ebabb11decc286ab4b0c883a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5442	5173	5174	5173
q2	391	193	168	168
q3	1486	1251	1194	1194
q4	1105	914	822	822
q5	3170	3130	3070	3070
q6	230	141	137	137
q7	1008	560	525	525
q8	2160	2300	2248	2248
q9	6707	6652	6645	6645
q10	3188	3131	3146	3131
q11	365	225	232	225
q12	395	243	244	243
q13	4404	3652	3657	3652
q14	264	221	226	221
q15	632	589	565	565
q16	460	427	421	421
q17	1047	597	542	542
q18	7107	6766	6806	6766
q19	1660	1532	1597	1532
q20	749	382	341	341
q21	2913	2494	2476	2476
q22	386	343	342	342
Total cold run time: 45269 ms
Total hot run time: 40439 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5144	5112	5090	5090
q2	340	249	243	243
q3	3376	3330	3287	3287
q4	2139	2018	2015	2015
q5	5945	5932	5879	5879
q6	234	133	135	133
q7	2401	1943	1928	1928
q8	3571	3646	3649	3646
q9	9048	9044	9039	9039
q10	3868	3929	3937	3929
q11	584	500	505	500
q12	822	640	653	640
q13	3869	3189	3183	3183
q14	306	277	285	277
q15	641	579	560	560
q16	583	519	498	498
q17	2035	1784	1798	1784
q18	8855	8389	8392	8389
q19	1741	1727	1740	1727
q20	2286	1988	1977	1977
q21	5681	5377	5435	5377
q22	572	463	514	463
Total cold run time: 64041 ms
Total hot run time: 60564 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 6d72649bbcd9f717ebabb11decc286ab4b0c883a, data reload: false

------ Round 1 ----------------------------------
q1	17665	5309	5155	5155
q2	2026	161	147	147
q3	10520	1162	1192	1162
q4	10172	808	865	808
q5	7807	2967	2957	2957
q6	254	140	143	140
q7	928	516	508	508
q8	9285	2057	2029	2029
q9	6911	6396	6385	6385
q10	8300	3067	3023	3023
q11	444	244	235	235
q12	393	251	243	243
q13	18011	3676	3628	3628
q14	257	213	216	213
q15	595	570	565	565
q16	449	418	393	393
q17	1000	504	493	493
q18	7465	6784	6880	6784
q19	1581	1395	1382	1382
q20	740	341	373	341
q21	2786	2404	2456	2404
q22	362	323	321	321
Total cold run time: 107951 ms
Total hot run time: 39316 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5212	5123	5139	5123
q2	337	251	255	251
q3	3332	3312	3296	3296
q4	2118	2030	2035	2030
q5	5797	5769	5800	5769
q6	223	134	136	134
q7	2335	1914	1899	1899
q8	3395	3446	3478	3446
q9	8948	8777	8780	8777
q10	3839	3880	3903	3880
q11	590	499	497	497
q12	803	641	634	634
q13	6832	3205	3194	3194
q14	295	270	273	270
q15	635	556	573	556
q16	563	495	530	495
q17	1950	1790	1765	1765
q18	8952	8492	8386	8386
q19	1655	1593	1591	1591
q20	2242	1991	1987	1987
q21	5607	5313	5340	5313
q22	537	486	520	486
Total cold run time: 66197 ms
Total hot run time: 59779 ms

yiguolei
yiguolei approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ltm

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 6d72649bbcd9f717ebabb11decc286ab4b0c883a, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	931	352	344	344
query2	6410	2093	2001	2001
query3	6645	223	222	222
query4	26239	22586	22487	22487
query5	4719	569	578	569
query6	291	184	181	181
query7	4574	275	279	275
query8	218	209	199	199
query9	8041	2663	2760	2663
query10	458	251	258	251
query11	16062	15424	15655	15424
query12	133	82	81	81
query13	1645	343	330	330
query14	11606	7165	7309	7165
query15	239	196	206	196
query16	6480	300	292	292
query17	1771	517	516	516
query18	1938	287	274	274
query19	226	143	145	143
query20	84	84	84	84
query21	186	96	97	96
query22	4745	4648	4498	4498
query23	32206	31412	31194	31194
query24	11920	2834	2810	2810
query25	583	358	351	351
query26	1683	150	149	149
query27	2903	289	280	280
query28	7030	1977	1965	1965
query29	1752	419	406	406
query30	288	139	144	139
query31	972	771	775	771
query32	92	62	63	62
query33	731	282	282	282
query34	858	460	452	452
query35	888	790	776	776
query36	1358	1308	1299	1299
query37	112	91	88	88
query38	3421	3321	3338	3321
query39	1329	1292	1286	1286
query40	296	94	91	91
query41	38	38	38	38
query42	97	89	101	89
query43	535	497	499	497
query44	1117	767	791	767
query45	197	188	182	182
query46	1097	664	662	662
query47	1713	1575	1483	1483
query48	348	263	269	263
query49	1183	339	332	332
query50	766	340	341	340
query51	5385	5286	5319	5286
query52	96	92	86	86
query53	219	161	160	160
query54	1378	615	643	615
query55	101	94	94	94
query56	219	200	201	200
query57	1037	969	964	964
query58	232	213	210	210
query59	2895	2638	2714	2638
query60	264	236	241	236
query61	90	87	86	86
query62	647	479	469	469
query63	162	165	163	163
query64	5880	1757	1709	1709
query65	3357	3244	3269	3244
query66	1225	340	334	334
query67	15447	15297	15221	15221
query68	13154	539	512	512
query69	589	271	257	257
query70	1639	1593	1476	1476
query71	566	235	233	233
query72	5550	3626	3612	3612
query73	1618	334	325	325
query74	6894	6396	6544	6396
query75	5669	2311	2292	2292
query76	5791	1179	1095	1095
query77	903	297	294	294
query78	8997	8583	8506	8506
query79	6106	505	503	503
query80	2517	388	378	378
query81	500	203	215	203
query82	348	121	121	121
query83	305	144	140	140
query84	256	59	54	54
query85	2365	296	286	286
query86	395	383	382	382
query87	3580	3321	3347	3321
query88	3174	2443	2448	2443
query89	335	265	262	262
query90	1908	251	239	239
query91	119	94	98	94
query92	71	63	56	56
query93	3130	501	503	501
query94	709	219	216	216
query95	514	467	458	458
query96	654	332	341	332
query97	4237	4158	4158	4158
query98	216	194	189	189
query99	1075	866	858	858
Total cold run time: 301672 ms
Total hot run time: 180573 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.83 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17184094261 Bytes

@yiguolei yiguolei merged commit 03ece43 into apache:master Dec 29, 2023
27 of 28 checks passed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@wangbo @HappenLee
Fix topicPublisher thread may NPE when no topic exists (apache#29306)
d55bdb7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed workload-group workload-schedule
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wangbo @doris-robot @yiguolei @CalvinKirs