Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](segment iterator) fix wrong segment cache remove #29333

Merged
merged 1 commit into from
Dec 30, 2023
Merged

[Fix](segment iterator) fix wrong segment cache remove #29333

merged 1 commit into from
Dec 30, 2023

Conversation

airborne12
Copy link
Member

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

qidaye
qidaye approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

eldenmoon
eldenmoon approved these changes Dec 29, 2023
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye
Copy link
Contributor

qidaye commented Dec 29, 2023

run buildall

yiguolei
yiguolei approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8606/23505)
Line Coverage: 28.68% (69984/244002)
Region Coverage: 27.68% (36222/130858)
Branch Coverage: 24.39% (18514/75922)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ba42ca239d6b13701f1cdbb2328c172a2f60fbad_ba42ca239d6b13701f1cdbb2328c172a2f60fbad/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit ba42ca239d6b13701f1cdbb2328c172a2f60fbad, data reload: false

------ Round 1 ----------------------------------
q1	17630	5056	5070	5056
q2	2014	155	144	144
q3	10529	1119	1146	1119
q4	10183	796	863	796
q5	7781	2916	2975	2916
q6	209	131	130	130
q7	926	554	496	496
q8	9321	1989	2011	1989
q9	6816	6365	6351	6351
q10	8195	3014	3000	3000
q11	431	222	231	222
q12	386	241	240	240
q13	18008	3672	3609	3609
q14	246	211	206	206
q15	576	536	525	525
q16	447	394	382	382
q17	950	452	475	452
q18	7210	6654	6665	6654
q19	1589	1358	1359	1358
q20	705	343	351	343
q21	2754	2378	2419	2378
q22	377	325	313	313
Total cold run time: 107283 ms
Total hot run time: 38679 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5098	5083	5064	5064
q2	343	249	259	249
q3	3292	3254	3256	3254
q4	2116	1981	2032	1981
q5	5800	5774	5765	5765
q6	213	122	124	122
q7	2339	1907	1935	1907
q8	3345	3435	3457	3435
q9	8856	8735	8766	8735
q10	3772	3863	3869	3863
q11	570	481	478	478
q12	786	644	643	643
q13	10572	3237	3208	3208
q14	286	274	277	274
q15	601	532	555	532
q16	545	514	511	511
q17	1939	1774	1740	1740
q18	8673	8271	8324	8271
q19	1613	1599	1636	1599
q20	2194	1976	1959	1959
q21	5610	5302	5234	5234
q22	556	496	488	488
Total cold run time: 69119 ms
Total hot run time: 59312 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit ba42ca239d6b13701f1cdbb2328c172a2f60fbad, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5447	5198	5140	5140
q2	385	169	162	162
q3	1479	1189	1186	1186
q4	1079	887	806	806
q5	3151	3130	3092	3092
q6	223	135	135	135
q7	994	563	519	519
q8	2149	2261	2269	2261
q9	6671	6650	6598	6598
q10	3176	3135	3109	3109
q11	352	221	220	220
q12	386	236	242	236
q13	4399	3613	3653	3613
q14	246	214	217	214
q15	606	573	547	547
q16	446	401	408	401
q17	1040	594	582	582
q18	7114	6793	6945	6793
q19	1638	1518	1600	1518
q20	613	354	335	335
q21	2861	2516	2456	2456
q22	405	333	347	333
Total cold run time: 44860 ms
Total hot run time: 40256 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5158	5063	5047	5047
q2	339	257	254	254
q3	3377	3325	3286	3286
q4	2148	2016	1997	1997
q5	5947	5941	5906	5906
q6	224	128	130	128
q7	2369	1901	1891	1891
q8	3566	3652	3681	3652
q9	9102	8971	9010	8971
q10	3884	3922	3925	3922
q11	590	486	511	486
q12	791	665	635	635
q13	3870	3242	3171	3171
q14	303	268	279	268
q15	619	567	543	543
q16	541	508	512	508
q17	2029	1815	1834	1815
q18	8741	8315	8420	8315
q19	1727	1614	1702	1614
q20	2283	2012	2023	2012
q21	5747	5391	5282	5282
q22	580	480	519	480
Total cold run time: 63935 ms
Total hot run time: 60183 ms

@doris-robot
Copy link

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit ba42ca239d6b13701f1cdbb2328c172a2f60fbad, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	927	360	335	335
query2	6412	1992	2078	1992
query3	6649	210	203	203
query4	26850	22531	22518	22518
query5	4788	519	541	519
query6	272	184	193	184
query7	4602	281	268	268
query8	232	211	217	211
query9	8304	2487	2540	2487
query10	410	242	254	242
query11	16338	15583	15546	15546
query12	144	80	81	80
query13	1660	323	327	323
query14	12619	7110	7052	7052
query15	264	189	199	189
query16	6499	293	277	277
query17	1789	501	499	499
query18	1939	286	270	270
query19	259	144	145	144
query20	83	79	81	79
query21	183	100	94	94
query22	4887	4536	4608	4536
query23	32409	31507	31464	31464
query24	11833	2886	2886	2886
query25	595	369	365	365
query26	1703	153	155	153
query27	2865	277	279	277
query28	7377	1935	1937	1935
query29	2082	435	421	421
query30	299	157	160	157
query31	999	801	814	801
query32	95	67	64	64
query33	733	316	312	312
query34	863	465	476	465
query35	943	825	788	788
query36	1433	1348	1325	1325
query37	193	85	83	83
query38	3574	3445	3415	3415
query39	1387	1344	1296	1296
query40	308	95	95	95
query41	38	40	35	35
query42	100	94	97	94
query43	559	538	514	514
query44	1342	736	737	736
query45	201	213	209	209
query46	1097	629	687	629
query47	1752	1654	1616	1616
query48	357	271	264	264
query49	1739	332	328	328
query50	811	333	339	333
query51	5462	5293	5260	5260
query52	98	86	94	86
query53	215	149	147	147
query54	1390	581	621	581
query55	106	90	87	87
query56	217	205	209	205
query57	1029	967	976	967
query58	234	218	212	212
query59	2855	2734	2689	2689
query60	266	242	247	242
query61	85	84	85	84
query62	650	455	478	455
query63	169	154	153	153
query64	5906	1723	1724	1723
query65	3336	3254	3265	3254
query66	1341	332	331	331
query67	15896	15266	15312	15266
query68	10931	517	526	517
query69	552	266	269	266
query70	1714	1568	1565	1565
query71	511	234	237	234
query72	5610	3577	3642	3577
query73	2200	320	316	316
query74	7335	6576	6482	6482
query75	4815	2259	2258	2258
query76	6317	1085	1133	1085
query77	670	292	297	292
query78	9120	8768	8543	8543
query79	1100	515	502	502
query80	583	366	371	366
query81	457	210	213	210
query82	209	105	101	101
query83	166	136	136	136
query84	247	55	53	53
query85	996	286	277	277
query86	380	373	366	366
query87	3567	3404	3375	3375
query88	3270	2292	2309	2292
query89	328	262	265	262
query90	1934	217	218	217
query91	123	93	94	93
query92	66	56	58	56
query93	2444	483	485	483
query94	847	196	195	195
query95	495	429	414	414
query96	639	325	324	324
query97	4295	4139	4175	4139
query98	222	199	194	194
query99	1110	915	818	818
Total cold run time: 296305 ms
Total hot run time: 180858 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.97 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17183621006 Bytes

xinyiZzz
xinyiZzz approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 5c9e0d9 into apache:master Dec 30, 2023
34 of 36 checks passed
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Dec 30, 2023
@airborne12
[Fix](segment iterator) fix wrong segment cache remove (apache#29333)
ab65216
xiaokang pushed a commit that referenced this pull request Dec 30, 2023
@airborne12
[Fix](segment iterator) fix wrong segment cache remove (#29333) (#29336)
351e9e3
@xiaokang xiaokang mentioned this pull request Dec 30, 2023
Merged
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@airborne12 @HappenLee
[Fix](segment iterator) fix wrong segment cache remove (apache#29333)
a2c1c19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

@qidaye qidaye qidaye approved these changes

@xinyiZzz xinyiZzz xinyiZzz approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@airborne12 @qidaye @doris-robot @yiguolei @xinyiZzz @eldenmoon @dataroaring @xiaokang