Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](vault) Fix creating storage vault failed with azure backend #46779

Merged
merged 1 commit into from
Jan 14, 2025
Merged

[fix](vault) Fix creating storage vault failed with azure backend #46779

merged 1 commit into from
Jan 14, 2025

Conversation

SWJTU-ZhangLei
Copy link
Contributor

@SWJTU-ZhangLei SWJTU-ZhangLei commented Jan 10, 2025 •
edited
Loading

  • fix create azure storage vault should use AzureResource instead of S3Resource
  • show storage vaults display use_path_style info

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32550 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca85849e0737a2dc545c0e5345a7a25fba19e9a1, data reload: false

------ Round 1 ----------------------------------
q1	17601	6235	6007	6007
q2	2046	299	173	173
q3	10417	1234	705	705
q4	10218	868	442	442
q5	7505	2159	1922	1922
q6	212	181	148	148
q7	885	745	614	614
q8	9233	1357	1145	1145
q9	5164	4898	4903	4898
q10	6757	2310	1867	1867
q11	466	290	265	265
q12	346	361	217	217
q13	17760	3690	3117	3117
q14	229	232	207	207
q15	564	510	493	493
q16	623	622	607	607
q17	582	837	342	342
q18	7144	6500	6389	6389
q19	1213	957	550	550
q20	319	330	195	195
q21	2859	2170	1936	1936
q22	362	329	311	311
Total cold run time: 102505 ms
Total hot run time: 32550 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6242	6244	6245	6244
q2	242	330	231	231
q3	2263	2664	2290	2290
q4	1435	1825	1400	1400
q5	4306	4704	4683	4683
q6	190	184	145	145
q7	2089	1948	1840	1840
q8	2617	2756	2686	2686
q9	7228	7126	7229	7126
q10	3025	3332	2851	2851
q11	607	528	501	501
q12	704	734	608	608
q13	3394	3798	3209	3209
q14	301	310	306	306
q15	572	494	498	494
q16	662	710	662	662
q17	1209	1722	1245	1245
q18	7641	7333	6832	6832
q19	780	1059	1097	1059
q20	1895	1970	1816	1816
q21	5644	5118	4904	4904
q22	602	611	599	599
Total cold run time: 53648 ms
Total hot run time: 51731 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187301 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca85849e0737a2dc545c0e5345a7a25fba19e9a1, data reload: false

query1	982	373	358	358
query2	6528	2353	2454	2353
query3	6719	221	213	213
query4	33556	23561	23199	23199
query5	4270	620	465	465
query6	306	204	203	203
query7	4625	487	311	311
query8	302	253	240	240
query9	9532	2709	2698	2698
query10	466	312	240	240
query11	18494	15299	14976	14976
query12	154	104	100	100
query13	1663	495	378	378
query14	10022	6593	7508	6593
query15	225	197	180	180
query16	7818	635	456	456
query17	1551	745	554	554
query18	2020	382	290	290
query19	241	176	154	154
query20	124	114	111	111
query21	212	122	101	101
query22	4047	4345	4084	4084
query23	34116	32806	32876	32806
query24	6372	2259	2236	2236
query25	477	448	381	381
query26	1202	274	153	153
query27	2008	455	328	328
query28	5144	2443	2449	2443
query29	649	532	409	409
query30	233	182	161	161
query31	968	864	757	757
query32	78	63	59	59
query33	508	351	321	321
query34	753	847	516	516
query35	821	802	720	720
query36	1013	1025	946	946
query37	121	96	77	77
query38	4114	4016	3923	3923
query39	1468	1562	1386	1386
query40	213	120	106	106
query41	59	55	64	55
query42	158	109	101	101
query43	514	530	502	502
query44	1306	826	810	810
query45	177	170	164	164
query46	852	1034	636	636
query47	1820	1862	1730	1730
query48	380	394	321	321
query49	785	488	396	396
query50	616	635	386	386
query51	6913	7065	6905	6905
query52	100	105	93	93
query53	229	256	183	183
query54	473	483	410	410
query55	82	79	84	79
query56	254	246	235	235
query57	1168	1154	1107	1107
query58	248	229	228	228
query59	3103	3103	3006	3006
query60	282	267	244	244
query61	116	120	114	114
query62	824	797	751	751
query63	233	193	184	184
query64	4218	1092	728	728
query65	3270	3166	3181	3166
query66	1078	474	321	321
query67	15897	15736	15351	15351
query68	7544	702	513	513
query69	454	300	257	257
query70	1190	1099	1132	1099
query71	457	289	253	253
query72	6081	3765	3920	3765
query73	670	740	359	359
query74	9996	9194	8699	8699
query75	3779	3165	2645	2645
query76	3669	1184	778	778
query77	766	393	286	286
query78	9894	9882	9398	9398
query79	3838	817	569	569
query80	689	515	437	437
query81	475	273	241	241
query82	615	154	125	125
query83	205	177	149	149
query84	298	98	80	80
query85	770	344	304	304
query86	357	326	306	306
query87	4552	4480	4344	4344
query88	4115	2194	2160	2160
query89	418	334	292	292
query90	1943	190	185	185
query91	174	135	111	111
query92	71	57	53	53
query93	1082	871	523	523
query94	654	382	302	302
query95	336	263	248	248
query96	501	611	282	282
query97	3028	2956	2739	2739
query98	227	200	202	200
query99	1612	1525	1408	1408
Total cold run time: 289730 ms
Total hot run time: 187301 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca85849e0737a2dc545c0e5345a7a25fba19e9a1, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.25	0.07	0.06
query4	1.61	0.11	0.10
query5	0.44	0.41	0.42
query6	1.16	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.54	0.50
query10	0.56	0.56	0.55
query11	0.15	0.11	0.10
query12	0.15	0.12	0.11
query13	0.60	0.60	0.59
query14	2.83	2.76	2.72
query15	0.89	0.82	0.82
query16	0.38	0.39	0.38
query17	1.03	0.97	1.06
query18	0.23	0.21	0.21
query19	1.95	1.87	1.98
query20	0.01	0.01	0.02
query21	15.36	0.89	0.59
query22	0.76	0.85	0.77
query23	15.11	1.39	0.52
query24	3.04	0.91	2.08
query25	0.17	0.17	0.08
query26	0.18	0.15	0.14
query27	0.07	0.04	0.06
query28	14.15	1.48	1.05
query29	12.58	4.06	3.30
query30	0.25	0.09	0.07
query31	2.85	0.62	0.39
query32	3.23	0.54	0.47
query33	3.05	3.07	3.17
query34	16.90	5.12	4.51
query35	4.50	4.48	4.49
query36	0.63	0.49	0.47
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 106.33 s
Total hot run time: 31.32 s

@SWJTU-ZhangLei
[fix](vault) Fix creating storage vault failed with azure backend
30f785c 
* fix create azure storage vault should use `AzureResource` instead of `S3Resource`
* show storage vaults display `use_path_style` info
@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 32866 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30f785c36e244340e3fe1ecaa29ddf5a170c62f8, data reload: false

------ Round 1 ----------------------------------
q1	17575	6142	6007	6007
q2	2061	308	186	186
q3	10506	1235	738	738
q4	10235	889	438	438
q5	8082	2204	1979	1979
q6	207	180	146	146
q7	900	775	629	629
q8	9252	1412	1251	1251
q9	5227	4847	4942	4847
q10	6731	2315	1862	1862
q11	482	283	268	268
q12	338	373	219	219
q13	17805	3607	3085	3085
q14	248	233	222	222
q15	567	523	506	506
q16	647	640	585	585
q17	596	859	340	340
q18	6961	6609	6499	6499
q19	1545	951	571	571
q20	320	329	200	200
q21	3032	2243	1982	1982
q22	365	338	306	306
Total cold run time: 103682 ms
Total hot run time: 32866 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6305	6457	6201	6201
q2	245	330	235	235
q3	2220	2802	2293	2293
q4	1431	1844	1417	1417
q5	4358	4761	4848	4761
q6	187	176	142	142
q7	2050	2036	1788	1788
q8	2629	2895	2758	2758
q9	7319	7262	7291	7262
q10	3061	3337	2727	2727
q11	592	536	505	505
q12	698	762	614	614
q13	3459	3830	3325	3325
q14	291	295	306	295
q15	569	513	495	495
q16	669	694	661	661
q17	1226	1764	1263	1263
q18	7855	7559	7494	7494
q19	845	1199	1056	1056
q20	1988	2026	1849	1849
q21	5705	5151	5032	5032
q22	628	625	601	601
Total cold run time: 54330 ms
Total hot run time: 52774 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195490 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30f785c36e244340e3fe1ecaa29ddf5a170c62f8, data reload: false

query1	1317	972	938	938
query2	6421	2436	2331	2331
query3	10954	4836	4882	4836
query4	33207	23796	23257	23257
query5	4377	629	469	469
query6	315	203	191	191
query7	3981	508	301	301
query8	296	243	225	225
query9	9345	2749	2718	2718
query10	469	308	252	252
query11	18290	15615	15046	15046
query12	164	108	109	108
query13	1615	527	400	400
query14	11474	7307	7233	7233
query15	230	213	198	198
query16	7842	627	458	458
query17	1578	779	616	616
query18	2077	437	338	338
query19	226	192	184	184
query20	123	117	120	117
query21	208	126	103	103
query22	4695	4588	4235	4235
query23	35231	34649	33444	33444
query24	6325	2367	2292	2292
query25	463	453	385	385
query26	1108	250	154	154
query27	2074	458	336	336
query28	5626	2524	2522	2522
query29	568	531	419	419
query30	218	188	157	157
query31	1018	882	816	816
query32	73	59	60	59
query33	497	380	318	318
query34	754	898	511	511
query35	812	810	765	765
query36	1012	1037	959	959
query37	128	105	80	80
query38	4242	4326	4295	4295
query39	1496	1475	1419	1419
query40	209	120	102	102
query41	52	53	49	49
query42	126	108	102	102
query43	530	537	487	487
query44	1420	874	846	846
query45	185	177	171	171
query46	882	1056	654	654
query47	1917	1935	1893	1893
query48	403	418	347	347
query49	769	507	401	401
query50	670	678	407	407
query51	7145	6998	7021	6998
query52	104	106	95	95
query53	227	253	188	188
query54	492	546	423	423
query55	87	81	83	81
query56	273	271	263	263
query57	1225	1226	1150	1150
query58	261	237	243	237
query59	3263	3305	3170	3170
query60	278	274	247	247
query61	118	155	113	113
query62	866	785	748	748
query63	226	191	186	186
query64	4047	1043	647	647
query65	3277	3364	3234	3234
query66	775	427	320	320
query67	16117	15870	15427	15427
query68	8159	694	514	514
query69	423	282	268	268
query70	1194	1106	1146	1106
query71	434	282	260	260
query72	6361	3914	3798	3798
query73	669	740	356	356
query74	10324	8935	8714	8714
query75	4214	3131	2613	2613
query76	3628	1174	777	777
query77	752	374	269	269
query78	9952	10254	9218	9218
query79	2572	821	613	613
query80	573	518	446	446
query81	477	288	239	239
query82	461	156	122	122
query83	176	173	154	154
query84	241	93	74	74
query85	802	340	374	340
query86	385	293	303	293
query87	4366	4326	4307	4307
query88	4768	2223	2199	2199
query89	396	329	296	296
query90	1738	189	190	189
query91	138	135	102	102
query92	64	58	51	51
query93	1873	882	522	522
query94	625	395	290	290
query95	324	262	249	249
query96	498	619	282	282
query97	2786	2947	2794	2794
query98	226	200	191	191
query99	1424	1489	1373	1373
Total cold run time: 297816 ms
Total hot run time: 195490 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30f785c36e244340e3fe1ecaa29ddf5a170c62f8, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.42	0.44	0.39
query6	1.16	0.65	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.49
query10	0.56	0.56	0.54
query11	0.15	0.11	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.74	2.75	2.74
query15	0.90	0.82	0.82
query16	0.36	0.39	0.38
query17	1.06	1.05	1.02
query18	0.23	0.20	0.21
query19	1.83	1.75	1.99
query20	0.02	0.01	0.01
query21	15.37	0.91	0.56
query22	0.75	0.95	0.71
query23	15.05	1.34	0.54
query24	3.23	1.19	1.77
query25	0.24	0.14	0.24
query26	0.23	0.14	0.13
query27	0.05	0.05	0.05
query28	14.04	1.48	1.05
query29	12.61	3.88	3.25
query30	0.25	0.08	0.06
query31	2.83	0.61	0.39
query32	3.22	0.55	0.47
query33	3.09	3.11	3.10
query34	16.82	5.13	4.47
query35	4.46	4.46	4.50
query36	0.85	0.48	0.49
query37	0.10	0.06	0.05
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.4 s
Total hot run time: 31.43 s

dataroaring
dataroaring approved these changes Jan 14, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit a2a674f into apache:master Jan 14, 2025
27 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 14, 2025
@SWJTU-ZhangLei
[fix](vault) Fix creating storage vault failed with azure backend (#4…
dac5360 
…6779)

* fix create azure storage vault should use `AzureResource` instead of
`S3Resource`
* show storage vaults display `use_path_style` info
@github-actions github-actions bot mentioned this pull request Jan 14, 2025
Merged
gavinchou pushed a commit that referenced this pull request Jan 14, 2025
@github-actions @SWJTU-ZhangLei
branch-3.0: [fix](vault) Fix creating storage vault failed with azure…
2e3c75a 
… backend #46779 (#46937)

Cherry-picked from #46779

Co-authored-by: Lei Zhang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@SWJTU-ZhangLei @hello-stephen @doris-robot @gavinchou @dataroaring @wm1581066