Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](disk) pick disk randomly when usage is less than 0.7 #29368

Merged
merged 5 commits into from
Jan 2, 2024

Conversation

dataroaring
Copy link
Contributor

@dataroaring dataroaring commented Jan 1, 2024

Proposed changes

When creating a tablet, firstly we try to pick a disk from disks whose usage is below 70% randomly. If all disks are beyonds 70%, then we try to pick a disk from disks whose usage is below 85% randomly. If all disks are beyond 85%, then we try to pick a disk from them randomly.
Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

Copy link
Contributor

github-actions bot commented Jan 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

github-actions bot commented Jan 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor Author

run buildall

@dataroaring
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Jan 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

github-actions bot commented Jan 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit c3ff5dd5d7bbd1c06721d24bfffa1feae19589f8, data reload: false

------ Round 1 ----------------------------------
q1	17623	5265	5228	5228
q2	2028	161	149	149
q3	10516	1116	1162	1116
q4	10219	768	831	768
q5	7822	2938	2934	2934
q6	210	135	140	135
q7	944	560	491	491
q8	9252	2020	2015	2015
q9	6875	6455	6357	6357
q10	8285	3041	3082	3041
q11	432	222	215	215
q12	391	237	245	237
q13	17988	3639	3657	3639
q14	238	220	207	207
q15	581	543	535	535
q16	500	397	401	397
q17	949	485	562	485
q18	7334	6789	6739	6739
q19	1594	1414	1303	1303
q20	686	344	364	344
q21	2792	2325	2447	2325
q22	390	322	329	322
Total cold run time: 107649 ms
Total hot run time: 38982 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5144	5148	5136	5136
q2	349	268	247	247
q3	3311	3273	3263	3263
q4	2118	1990	2007	1990
q5	5818	5778	5813	5778
q6	216	126	131	126
q7	2340	1882	1908	1882
q8	3385	3461	3460	3460
q9	8794	8745	8738	8738
q10	3839	3867	3845	3845
q11	579	513	503	503
q12	801	661	638	638
q13	6494	3239	3248	3239
q14	300	286	269	269
q15	603	542	538	538
q16	534	504	476	476
q17	1934	1784	1757	1757
q18	8732	8366	8295	8295
q19	1629	1595	1575	1575
q20	2198	1965	1969	1965
q21	5638	5273	5289	5273
q22	570	510	531	510
Total cold run time: 65326 ms
Total hot run time: 59503 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit c3ff5dd5d7bbd1c06721d24bfffa1feae19589f8, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5454	5162	5159	5159
q2	393	160	160	160
q3	1467	1177	1201	1177
q4	1097	838	851	838
q5	3129	3120	3111	3111
q6	228	135	136	135
q7	976	561	522	522
q8	2167	2296	2245	2245
q9	6761	6681	6644	6644
q10	3159	3212	3176	3176
q11	343	217	222	217
q12	378	234	237	234
q13	4385	3605	3656	3605
q14	256	219	217	217
q15	614	554	554	554
q16	440	424	401	401
q17	1049	544	488	488
q18	7121	6699	6812	6699
q19	1634	1529	1589	1529
q20	568	360	422	360
q21	2875	2491	2454	2454
q22	392	314	340	314
Total cold run time: 44886 ms
Total hot run time: 40239 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5094	5093	5066	5066
q2	345	265	267	265
q3	3364	3297	3330	3297
q4	2141	2004	1966	1966
q5	5930	5891	5896	5891
q6	227	127	122	122
q7	2405	1896	1953	1896
q8	3581	3657	3656	3656
q9	8983	8970	8949	8949
q10	3911	3904	3906	3904
q11	597	487	507	487
q12	796	644	628	628
q13	3901	3221	3195	3195
q14	311	291	264	264
q15	627	553	537	537
q16	566	514	516	514
q17	2015	1776	1818	1776
q18	8680	8347	8446	8347
q19	1735	1681	1651	1651
q20	2298	1995	1989	1989
q21	5809	5331	5363	5331
q22	546	475	479	475
Total cold run time: 63862 ms
Total hot run time: 60206 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit c3ff5dd5d7bbd1c06721d24bfffa1feae19589f8, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	945	355	338	338
query2	6439	2020	1868	1868
query3	6648	208	201	201
query4	28464	22650	22488	22488
query5	5202	525	578	525
query6	283	190	177	177
query7	4576	292	277	277
query8	226	208	210	208
query9	8314	2539	2548	2539
query10	446	284	237	237
query11	16280	15596	15564	15564
query12	143	81	78	78
query13	1681	335	332	332
query14	12069	7209	7303	7209
query15	242	189	190	189
query16	6545	290	279	279
query17	1849	503	498	498
query18	1942	284	270	270
query19	279	146	144	144
query20	90	79	81	79
query21	180	97	97	97
query22	4971	4743	4777	4743
query23	31804	31267	31069	31069
query24	11882	2853	2830	2830
query25	596	359	368	359
query26	1689	151	153	151
query27	2849	284	294	284
query28	7091	1957	1959	1957
query29	2030	402	414	402
query30	299	147	146	146
query31	963	762	775	762
query32	85	64	65	64
query33	735	297	271	271
query34	852	459	449	449
query35	917	785	773	773
query36	1353	1319	1190	1190
query37	192	79	78	78
query38	3378	3292	3277	3277
query39	1320	1293	1295	1293
query40	287	98	94	94
query41	39	36	35	35
query42	97	89	96	89
query43	545	494	510	494
query44	1090	718	731	718
query45	194	189	182	182
query46	1095	681	641	641
query47	1689	1601	1578	1578
query48	353	267	265	265
query49	1183	335	329	329
query50	739	338	347	338
query51	5386	5266	5262	5262
query52	103	89	89	89
query53	226	161	154	154
query54	1385	586	594	586
query55	98	91	88	88
query56	220	208	209	208
query57	995	961	980	961
query58	231	216	213	213
query59	2773	2614	2557	2557
query60	257	248	234	234
query61	90	88	87	87
query62	647	470	470	470
query63	176	154	152	152
query64	5934	1745	1780	1745
query65	3351	3253	3296	3253
query66	1319	347	340	340
query67	15648	15481	15784	15481
query68	11301	528	527	527
query69	528	256	268	256
query70	1689	1542	1432	1432
query71	506	230	233	230
query72	5612	3592	3602	3592
query73	2260	310	338	310
query74	7068	6469	6440	6440
query75	4811	2278	2244	2244
query76	6334	1124	1156	1124
query77	664	293	287	287
query78	9119	8743	8679	8679
query79	1392	500	526	500
query80	642	372	371	371
query81	458	208	206	206
query82	214	104	103	103
query83	164	138	133	133
query84	248	54	56	54
query85	924	286	277	277
query86	370	370	391	370
query87	3618	3420	3396	3396
query88	3195	2278	2299	2278
query89	347	251	268	251
query90	1860	214	210	210
query91	122	92	113	92
query92	62	55	56	55
query93	1739	491	437	437
query94	796	195	188	188
query95	475	426	416	416
query96	626	325	331	325
query97	4264	4157	4200	4157
query98	209	195	187	187
query99	1149	824	778	778
Total cold run time: 294765 ms
Total hot run time: 180243 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.62% (8614/23522)
Line Coverage: 28.69% (70020/244084)
Region Coverage: 27.66% (36227/130964)
Branch Coverage: 24.36% (18511/75976)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c3ff5dd5d7bbd1c06721d24bfffa1feae19589f8_c3ff5dd5d7bbd1c06721d24bfffa1feae19589f8/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.69 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17189160963 Bytes

Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Jan 1, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.62% (8613/23521)
Line Coverage: 28.68% (70005/244083)
Region Coverage: 27.67% (36235/130963)
Branch Coverage: 24.37% (18512/75976)
Coverage Report: http://coverage.selectdb-in.cc/coverage/82a970a91a8c96edbd0bd395dfe706077fcf6787_82a970a91a8c96edbd0bd395dfe706077fcf6787/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 82a970a91a8c96edbd0bd395dfe706077fcf6787, data reload: false

------ Round 1 ----------------------------------
q1	17659	5087	5135	5087
q2	2028	159	154	154
q3	10524	1055	1086	1055
q4	10188	764	856	764
q5	7775	2920	2875	2875
q6	206	135	133	133
q7	951	506	513	506
q8	9308	2003	1994	1994
q9	6815	6396	6370	6370
q10	8219	3005	3042	3005
q11	433	224	215	215
q12	388	229	237	229
q13	18008	3632	3634	3632
q14	241	210	216	210
q15	580	539	533	533
q16	451	391	413	391
q17	970	552	458	458
q18	7295	6669	6666	6666
q19	1569	1315	1350	1315
q20	689	322	340	322
q21	2795	2364	2429	2364
q22	382	329	331	329
Total cold run time: 107474 ms
Total hot run time: 38607 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5131	5061	5074	5061
q2	357	230	244	230
q3	3329	3300	3241	3241
q4	2194	1986	1945	1945
q5	5815	5790	5769	5769
q6	215	125	126	125
q7	2317	1889	1889	1889
q8	3351	3429	3466	3429
q9	8774	8734	8688	8688
q10	3799	3820	3839	3820
q11	598	480	483	480
q12	802	659	641	641
q13	7798	3178	3205	3178
q14	305	268	276	268
q15	602	530	525	525
q16	542	491	489	489
q17	1976	1781	1779	1779
q18	8576	8371	8289	8289
q19	1613	1531	1594	1531
q20	2196	1967	1966	1966
q21	5538	5224	5283	5224
q22	543	540	468	468
Total cold run time: 66371 ms
Total hot run time: 59035 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 82a970a91a8c96edbd0bd395dfe706077fcf6787, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5568	5182	5178	5178
q2	386	157	157	157
q3	1455	1241	1185	1185
q4	1074	844	805	805
q5	3147	3000	3113	3000
q6	224	140	138	138
q7	993	567	515	515
q8	2159	2259	2250	2250
q9	6686	6689	6622	6622
q10	3170	3113	3119	3113
q11	346	226	209	209
q12	386	233	235	233
q13	4402	3674	3622	3622
q14	255	207	217	207
q15	603	545	547	545
q16	468	403	396	396
q17	1040	524	560	524
q18	7053	6809	6771	6771
q19	1641	1568	1574	1568
q20	626	337	347	337
q21	2861	2473	2461	2461
q22	403	328	347	328
Total cold run time: 44946 ms
Total hot run time: 40164 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5146	5163	5065	5065
q2	332	255	266	255
q3	3376	3306	3320	3306
q4	2134	2018	2025	2018
q5	5958	5952	5964	5952
q6	225	129	123	123
q7	2421	1907	2017	1907
q8	3581	3643	3643	3643
q9	9015	8987	9040	8987
q10	3889	3919	3913	3913
q11	581	508	500	500
q12	805	639	643	639
q13	3911	3188	3182	3182
q14	294	288	277	277
q15	612	543	532	532
q16	568	501	511	501
q17	2047	1808	1809	1808
q18	8780	8373	8391	8373
q19	1738	1716	1678	1678
q20	2263	2025	1979	1979
q21	5738	5309	5263	5263
q22	589	525	491	491
Total cold run time: 64003 ms
Total hot run time: 60392 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 82a970a91a8c96edbd0bd395dfe706077fcf6787, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	925	356	336	336
query2	6433	2037	2121	2037
query3	6645	209	202	202
query4	26662	22430	22296	22296
query5	4347	550	566	550
query6	270	184	196	184
query7	4576	290	277	277
query8	226	208	192	192
query9	8221	2785	2652	2652
query10	434	268	259	259
query11	16153	15367	15469	15367
query12	130	74	73	73
query13	1628	327	332	327
query14	11199	7307	7115	7115
query15	231	191	197	191
query16	6581	281	281	281
query17	2076	523	511	511
query18	2433	274	278	274
query19	321	145	171	145
query20	138	80	84	80
query21	488	101	94	94
query22	5119	5104	4945	4945
query23	32128	31356	31190	31190
query24	12076	2848	2835	2835
query25	584	351	342	342
query26	1720	151	148	148
query27	2861	283	282	282
query28	7040	1972	1954	1954
query29	2077	382	379	379
query30	297	139	146	139
query31	955	778	788	778
query32	91	63	59	59
query33	738	280	266	266
query34	876	454	453	453
query35	858	776	755	755
query36	1309	1224	1177	1177
query37	188	75	74	74
query38	3432	3299	3280	3280
query39	1322	1292	1270	1270
query40	302	93	88	88
query41	38	35	35	35
query42	93	94	94	94
query43	548	491	495	491
query44	1095	718	741	718
query45	204	187	182	182
query46	1090	660	659	659
query47	1718	1536	1575	1536
query48	344	264	260	260
query49	1214	334	334	334
query50	727	332	372	332
query51	5403	5289	5267	5267
query52	99	83	95	83
query53	224	150	147	147
query54	1403	586	638	586
query55	100	91	90	90
query56	208	199	200	199
query57	1081	963	957	957
query58	222	198	199	198
query59	2792	2652	2640	2640
query60	255	231	229	229
query61	84	83	82	82
query62	670	481	461	461
query63	171	151	149	149
query64	5907	1750	1718	1718
query65	3328	3282	3271	3271
query66	1379	329	328	328
query67	15690	15711	15408	15408
query68	11763	529	531	529
query69	542	251	253	251
query70	1696	1527	1584	1527
query71	495	230	225	225
query72	5570	3527	3597	3527
query73	2479	317	320	317
query74	6996	6427	6519	6427
query75	4800	2281	2276	2276
query76	6271	1127	1148	1127
query77	646	273	257	257
query78	9042	8991	8653	8653
query79	1236	500	498	498
query80	662	379	366	366
query81	454	209	203	203
query82	208	100	99	99
query83	159	141	136	136
query84	249	54	52	52
query85	988	288	269	269
query86	394	380	383	380
query87	3615	3381	3402	3381
query88	3537	2276	2309	2276
query89	361	259	278	259
query90	1896	212	212	212
query91	120	92	96	92
query92	64	52	51	51
query93	2505	438	438	438
query94	935	192	190	190
query95	471	419	418	418
query96	628	322	320	320
query97	4309	4203	4135	4135
query98	216	195	192	192
query99	1102	851	834	834
Total cold run time: 294586 ms
Total hot run time: 180228 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.33 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183676659 Bytes

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2024
Copy link
Contributor

github-actions bot commented Jan 2, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 4581618 into apache:master Jan 2, 2024
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants