Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](auth) after fe restarting, external permissions are lost (#43275) #43339

Merged
merged 2 commits into from
Nov 25, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Nov 6, 2024

pick: #43275

…e#43275)

cause by: apache#39597

This PR checks whether the authorized tableName exists. After
restarting, when replaying the log, the metadata of the external cannot
be obtained, so the replay will report an error, but it does not affect
the startup, only printing the log.

Solution: No longer verify during playback
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Nov 6, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49618 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eacc9711a00c50dd8781f6e36ef3318909820866, data reload: false

------ Round 1 ----------------------------------
q1	17925	4480	4383	4383
q2	2093	158	148	148
q3	10435	1941	2103	1941
q4	10242	1305	1322	1305
q5	8820	3957	3971	3957
q6	236	147	124	124
q7	2037	1632	1599	1599
q8	9306	2767	2717	2717
q9	10534	10030	10276	10030
q10	8681	3570	3533	3533
q11	416	252	245	245
q12	485	295	299	295
q13	18360	3958	4042	3958
q14	364	329	319	319
q15	514	469	462	462
q16	540	453	456	453
q17	1141	982	945	945
q18	7359	6888	6921	6888
q19	1665	1615	1534	1534
q20	526	304	295	295
q21	4444	4088	4080	4080
q22	496	407	408	407
Total cold run time: 116619 ms
Total hot run time: 49618 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4347	4341	4341
q2	320	236	226	226
q3	4153	4169	4164	4164
q4	2750	2748	2756	2748
q5	7211	7202	7131	7131
q6	245	124	122	122
q7	3283	2957	2840	2840
q8	4400	4517	4483	4483
q9	14360	13958	13755	13755
q10	4228	4322	4255	4255
q11	756	677	696	677
q12	1036	836	852	836
q13	6769	3741	3723	3723
q14	444	432	419	419
q15	498	462	464	462
q16	631	590	594	590
q17	3852	3834	3896	3834
q18	8737	8814	8804	8804
q19	1731	1691	1655	1655
q20	2393	2101	2115	2101
q21	8552	8513	8329	8329
q22	1022	953	956	953
Total cold run time: 81734 ms
Total hot run time: 76448 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 213707 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eacc9711a00c50dd8781f6e36ef3318909820866, data reload: false

query1	938	418	382	382
query2	6532	2238	2179	2179
query3	6919	196	201	196
query4	23254	21556	21497	21497
query5	19741	6514	6463	6463
query6	284	223	238	223
query7	4342	304	314	304
query8	260	255	232	232
query9	3100	2654	2597	2597
query10	425	317	312	312
query11	15842	14964	15041	14964
query12	122	75	80	75
query13	1053	437	441	437
query14	17164	13599	13839	13599
query15	367	220	224	220
query16	6486	294	284	284
query17	1908	938	885	885
query18	891	319	310	310
query19	213	155	151	151
query20	102	104	96	96
query21	192	104	97	97
query22	5011	5045	5041	5041
query23	34245	33513	33571	33513
query24	7871	6360	6308	6308
query25	522	431	423	423
query26	1273	169	165	165
query27	2339	295	300	295
query28	6058	2239	2230	2230
query29	2881	2812	2921	2812
query30	246	174	164	164
query31	987	737	766	737
query32	71	63	61	61
query33	453	251	267	251
query34	872	482	480	480
query35	1166	977	956	956
query36	1273	1093	1113	1093
query37	173	64	65	64
query38	3051	2921	2894	2894
query39	1368	1323	1331	1323
query40	307	94	98	94
query41	41	40	39	39
query42	95	94	90	90
query43	653	634	665	634
query44	1174	726	729	726
query45	249	238	228	228
query46	1227	941	966	941
query47	1830	1902	1832	1832
query48	511	430	404	404
query49	641	383	385	383
query50	868	619	602	602
query51	4766	4683	4625	4625
query52	91	76	85	76
query53	224	189	190	189
query54	2690	2451	2495	2451
query55	78	79	81	79
query56	232	219	218	218
query57	1282	1173	1147	1147
query58	228	215	213	213
query59	3489	3432	3495	3432
query60	220	211	216	211
query61	96	93	98	93
query62	860	468	510	468
query63	206	181	176	176
query64	5782	1611	1503	1503
query65	3638	3585	3544	3544
query66	628	445	395	395
query67	16481	15444	15612	15444
query68	10241	663	652	652
query69	504	279	280	279
query70	1636	1385	1485	1385
query71	424	313	317	313
query72	6772	4895	4910	4895
query73	752	327	320	320
query74	6243	5797	5892	5797
query75	5516	3746	3718	3718
query76	5913	1130	1187	1130
query77	1056	273	278	273
query78	12936	11856	11678	11678
query79	8165	639	644	639
query80	1497	413	388	388
query81	493	239	234	234
query82	1638	99	100	99
query83	170	133	136	133
query84	259	72	74	72
query85	880	329	328	328
query86	336	299	303	299
query87	3223	2985	2992	2985
query88	4523	2282	2288	2282
query89	489	296	296	296
query90	1955	223	214	214
query91	155	137	128	128
query92	64	54	52	52
query93	7187	571	540	540
query94	710	210	216	210
query95	2032	2079	2002	2002
query96	644	328	320	320
query97	6687	6451	6517	6451
query98	227	216	209	209
query99	3026	836	939	836
Total cold run time: 324935 ms
Total hot run time: 213707 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eacc9711a00c50dd8781f6e36ef3318909820866, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.04
query4	1.79	0.06	0.06
query5	0.53	0.53	0.52
query6	1.22	0.63	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.48
query10	0.52	0.54	0.52
query11	0.13	0.09	0.09
query12	0.12	0.09	0.09
query13	0.61	0.62	0.61
query14	0.78	0.78	0.77
query15	0.78	0.77	0.77
query16	0.36	0.37	0.38
query17	1.01	1.02	1.05
query18	0.21	0.26	0.24
query19	1.85	1.83	1.86
query20	0.01	0.02	0.02
query21	15.47	0.55	0.55
query22	1.98	2.16	1.50
query23	17.22	0.98	0.99
query24	6.79	0.99	1.02
query25	0.36	0.10	0.06
query26	0.77	0.16	0.15
query27	0.03	0.05	0.04
query28	6.36	0.76	0.73
query29	12.80	2.46	2.43
query30	0.62	0.55	0.53
query31	2.79	0.40	0.37
query32	3.38	0.51	0.50
query33	3.08	3.09	3.05
query34	15.25	4.84	4.82
query35	4.88	4.85	4.86
query36	1.05	1.03	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.15	0.13	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.05 s
Total hot run time: 30.92 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit eacc9711a00c50dd8781f6e36ef3318909820866 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@zddr
Copy link
Contributor Author

zddr commented Nov 6, 2024

run external

@zddr
Copy link
Contributor Author

zddr commented Nov 7, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49101 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2cf436094c8234eb28a32d53699f1759b7f4696, data reload: false

------ Round 1 ----------------------------------
q1	17707	4368	4343	4343
q2	2073	148	146	146
q3	10445	1909	1926	1909
q4	10330	1214	1298	1214
q5	8830	3881	3914	3881
q6	241	123	124	123
q7	1977	1600	1573	1573
q8	9292	2743	2693	2693
q9	10144	9877	9797	9797
q10	8627	3532	3518	3518
q11	420	247	245	245
q12	473	299	304	299
q13	18386	3972	4058	3972
q14	352	331	317	317
q15	500	456	447	447
q16	547	458	461	458
q17	1121	983	967	967
q18	7226	6908	6957	6908
q19	1720	1566	1541	1541
q20	537	300	284	284
q21	4421	4182	4067	4067
q22	516	399	408	399
Total cold run time: 115885 ms
Total hot run time: 49101 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4364	4298	4304	4298
q2	318	228	210	210
q3	4177	4130	4138	4130
q4	2741	2713	2733	2713
q5	7188	7206	7113	7113
q6	240	122	122	122
q7	3218	2871	2859	2859
q8	4355	4476	4471	4471
q9	13609	13548	13588	13548
q10	4246	4233	4244	4233
q11	765	684	692	684
q12	1027	871	858	858
q13	6939	3749	3722	3722
q14	450	430	426	426
q15	498	462	467	462
q16	629	583	592	583
q17	3785	3880	3927	3880
q18	8665	8761	8772	8761
q19	1718	1663	1659	1659
q20	2404	2136	2106	2106
q21	8552	8404	8395	8395
q22	1012	956	911	911
Total cold run time: 80900 ms
Total hot run time: 76144 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211817 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2cf436094c8234eb28a32d53699f1759b7f4696, data reload: false

query1	919	397	407	397
query2	6552	2302	2097	2097
query3	6917	202	203	202
query4	23196	21460	21815	21460
query5	19744	6552	6530	6530
query6	297	228	239	228
query7	4338	299	311	299
query8	254	248	268	248
query9	3067	2633	2569	2569
query10	475	310	312	310
query11	15868	14966	14878	14878
query12	122	73	81	73
query13	1023	439	433	433
query14	17438	13147	13410	13147
query15	383	232	229	229
query16	6466	285	258	258
query17	1755	914	943	914
query18	899	319	323	319
query19	211	150	150	150
query20	110	97	99	97
query21	185	100	95	95
query22	5202	5014	5060	5014
query23	34165	33402	33485	33402
query24	7866	6403	6337	6337
query25	547	429	431	429
query26	1272	162	158	158
query27	2440	292	287	287
query28	6109	2236	2185	2185
query29	2919	2724	2688	2688
query30	247	175	164	164
query31	942	738	746	738
query32	72	60	59	59
query33	456	265	262	262
query34	868	482	464	464
query35	1141	950	914	914
query36	1588	1094	1174	1094
query37	175	63	62	62
query38	3130	2892	2904	2892
query39	1379	1331	1332	1331
query40	313	96	93	93
query41	43	38	36	36
query42	91	80	95	80
query43	673	586	715	586
query44	1222	724	727	724
query45	242	229	232	229
query46	1231	953	968	953
query47	1837	1744	1835	1744
query48	482	420	412	412
query49	650	386	404	386
query50	875	630	588	588
query51	4788	4601	4621	4601
query52	91	84	81	81
query53	225	181	187	181
query54	2668	2517	2454	2454
query55	85	89	79	79
query56	243	199	205	199
query57	1319	1106	1193	1106
query58	216	220	231	220
query59	3623	3303	3475	3303
query60	215	221	212	212
query61	99	94	105	94
query62	864	481	497	481
query63	207	176	173	173
query64	3527	1651	1485	1485
query65	3619	3578	3546	3546
query66	775	407	423	407
query67	16141	15500	16786	15500
query68	8962	678	633	633
query69	492	268	272	268
query70	1713	1500	1542	1500
query71	398	307	323	307
query72	6740	4881	4742	4742
query73	751	314	308	308
query74	6255	5850	5831	5831
query75	4498	3735	3663	3663
query76	4609	1130	1179	1130
query77	600	255	260	255
query78	12445	11366	12599	11366
query79	7200	638	630	630
query80	2516	384	379	379
query81	523	238	237	237
query82	799	95	97	95
query83	181	135	128	128
query84	256	69	70	69
query85	1275	326	314	314
query86	345	295	328	295
query87	3275	2990	3062	2990
query88	4391	2289	2261	2261
query89	358	287	298	287
query90	1706	205	215	205
query91	158	127	132	127
query92	62	53	55	53
query93	3598	564	547	547
query94	691	214	207	207
query95	1993	1948	1902	1902
query96	644	323	324	323
query97	6496	6386	6469	6386
query98	220	198	206	198
query99	2982	815	839	815
Total cold run time: 314233 ms
Total hot run time: 211817 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2cf436094c8234eb28a32d53699f1759b7f4696, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.04
query4	1.79	0.06	0.06
query5	0.53	0.54	0.52
query6	1.30	0.62	0.60
query7	0.01	0.02	0.01
query8	0.03	0.02	0.02
query9	0.51	0.50	0.47
query10	0.54	0.53	0.54
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.79	0.78	0.79
query15	0.77	0.76	0.77
query16	0.36	0.37	0.37
query17	1.05	0.98	1.01
query18	0.21	0.26	0.24
query19	1.93	1.87	1.85
query20	0.02	0.01	0.01
query21	15.47	0.54	0.55
query22	2.31	2.25	1.66
query23	17.14	1.07	1.12
query24	4.05	2.48	3.94
query25	0.37	0.10	0.05
query26	0.84	0.17	0.14
query27	0.03	0.04	0.03
query28	4.73	0.74	0.76
query29	12.60	2.36	2.35
query30	0.59	0.56	0.52
query31	2.81	0.39	0.37
query32	3.39	0.49	0.50
query33	3.14	3.04	3.06
query34	15.25	4.80	4.76
query35	4.86	4.85	4.84
query36	1.07	1.01	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 100.07 s
Total hot run time: 32.41 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d2cf436094c8234eb28a32d53699f1759b7f4696 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

@zddr
Copy link
Contributor Author

zddr commented Nov 11, 2024

run external

@morrySnow
Copy link
Contributor

run p0

@morningman morningman merged commit b061b3e into apache:branch-2.0 Nov 25, 2024
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants