Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](move-memtable) log rpc failures in stream file writer #29267

Merged
merged 3 commits into from
Jan 4, 2024
Merged

[improve](move-memtable) log rpc failures in stream file writer #29267

merged 3 commits into from
Jan 4, 2024

Conversation

kaijchen
Copy link
Contributor

Proposed changes

Followup #29257

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8590/23480)
Line Coverage: 28.65% (69880/243867)
Region Coverage: 27.66% (36172/130770)
Branch Coverage: 24.38% (18490/75854)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5f00b9767f5dbad8a6b8279ce2b426a7164b04ee_5f00b9767f5dbad8a6b8279ce2b426a7164b04ee/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5f00b9767f5dbad8a6b8279ce2b426a7164b04ee, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5380	5167	5148	5148
q2	393	161	158	158
q3	1470	1237	1120	1120
q4	1093	829	827	827
q5	3101	3074	3129	3074
q6	235	141	136	136
q7	978	545	563	545
q8	2152	2219	2253	2219
q9	6897	6813	6867	6813
q10	3210	3151	3165	3151
q11	355	244	222	222
q12	388	244	242	242
q13	4390	3649	3614	3614
q14	269	223	222	222
q15	639	571	576	571
q16	456	411	399	399
q17	1050	584	542	542
q18	7040	6714	6787	6714
q19	1650	1581	1473	1473
q20	614	356	345	345
q21	2931	2472	2493	2472
q22	406	325	320	320
Total cold run time: 45097 ms
Total hot run time: 40327 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5105	5093	5101	5093
q2	344	246	262	246
q3	3375	3327	3310	3310
q4	2172	2021	2010	2010
q5	5982	5948	5944	5944
q6	238	136	131	131
q7	2404	1920	1959	1920
q8	3556	3655	3622	3622
q9	9055	9008	8998	8998
q10	3892	3941	3928	3928
q11	583	487	487	487
q12	835	638	651	638
q13	3891	3220	3156	3156
q14	303	291	276	276
q15	620	556	551	551
q16	562	531	522	522
q17	2049	1801	1800	1800
q18	8752	8326	8354	8326
q19	1779	1690	1708	1690
q20	2282	2008	1989	1989
q21	5768	5379	5338	5338
q22	558	524	497	497
Total cold run time: 64105 ms
Total hot run time: 60472 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.34 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17183867833 Bytes

dataroaring
dataroaring previously approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

dataroaring
dataroaring approved these changes Dec 31, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 31, 2023
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.62% (8614/23520)
Line Coverage: 28.68% (70018/244112)
Region Coverage: 27.68% (36246/130970)
Branch Coverage: 24.38% (18522/75984)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6592be9e47c3a4adb0d96da6770d4e3df92fcb72_6592be9e47c3a4adb0d96da6770d4e3df92fcb72/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 6592be9e47c3a4adb0d96da6770d4e3df92fcb72, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5456	5169	5191	5169
q2	418	167	161	161
q3	1461	1175	1247	1175
q4	1094	839	846	839
q5	3159	3184	2978	2978
q6	226	138	132	132
q7	951	553	534	534
q8	2160	2210	2180	2180
q9	6732	6704	6658	6658
q10	3176	3130	3146	3130
q11	345	213	224	213
q12	386	235	242	235
q13	4437	3632	3683	3632
q14	255	223	210	210
q15	620	556	553	553
q16	480	389	401	389
q17	1039	517	467	467
q18	7095	6792	6853	6792
q19	1639	1519	1509	1509
q20	592	363	326	326
q21	2921	2530	2524	2524
q22	400	338	318	318
Total cold run time: 45042 ms
Total hot run time: 40124 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5148	5100	5012	5012
q2	337	252	250	250
q3	3372	3308	3288	3288
q4	2162	2001	1994	1994
q5	5971	5925	5942	5925
q6	233	124	126	124
q7	2350	1952	1930	1930
q8	3576	3637	3666	3637
q9	9018	9006	8971	8971
q10	3882	3922	3930	3922
q11	575	494	494	494
q12	794	675	657	657
q13	3883	3216	3196	3196
q14	292	265	261	261
q15	608	543	539	539
q16	548	490	534	490
q17	2038	1823	1835	1823
q18	8691	8326	8477	8326
q19	1765	1678	1683	1678
q20	2269	2000	1990	1990
q21	5785	5340	5327	5327
q22	561	488	491	488
Total cold run time: 63858 ms
Total hot run time: 60322 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.42 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183928150 Bytes

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit f08124af3acd59bfc66a44c286f7267fa2e1c80a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5516	5205	5203	5203
q2	391	166	156	156
q3	1471	1185	1237	1185
q4	1112	908	804	804
q5	3154	3079	3151	3079
q6	225	137	140	137
q7	1003	555	496	496
q8	2164	2223	2200	2200
q9	6751	6723	6653	6653
q10	3157	3137	3228	3137
q11	346	231	209	209
q12	386	238	243	238
q13	4402	3751	3678	3678
q14	250	223	228	223
q15	607	546	537	537
q16	451	419	427	419
q17	1049	527	503	503
q18	7124	6754	6823	6754
q19	1655	1549	1515	1515
q20	585	348	360	348
q21	2924	2442	2484	2442
q22	400	337	328	328
Total cold run time: 45123 ms
Total hot run time: 40244 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5164	5196	5122	5122
q2	341	247	271	247
q3	3376	3339	3321	3321
q4	2160	2025	2033	2025
q5	5938	5926	5931	5926
q6	232	129	129	129
q7	2359	1977	1902	1902
q8	3591	3685	3704	3685
q9	9068	9079	9101	9079
q10	3883	3928	3931	3928
q11	611	501	487	487
q12	812	645	665	645
q13	3912	3247	3237	3237
q14	304	292	287	287
q15	622	541	547	541
q16	552	528	538	528
q17	2036	1839	1806	1806
q18	8805	8414	8584	8414
q19	1739	1696	1684	1684
q20	2319	1993	1983	1983
q21	5784	5421	5322	5322
q22	585	487	524	487
Total cold run time: 64193 ms
Total hot run time: 60785 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.84 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.7 seconds inserted 10000000 Rows, about 361K ops/s
storage size: 17183962637 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.63% (8615/23520)
Line Coverage: 28.68% (70017/244127)
Region Coverage: 27.67% (36249/131004)
Branch Coverage: 24.37% (18520/76008)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f08124af3acd59bfc66a44c286f7267fa2e1c80a_f08124af3acd59bfc66a44c286f7267fa2e1c80a/report/index.html

@dataroaring dataroaring merged commit 35150bb into apache:master Jan 4, 2024
24 of 26 checks passed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@kaijchen @HappenLee
[improve](move-memtable) log rpc failures in stream file writer (apac…
7da5bae 
…he#29267)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kaijchen @doris-robot @dataroaring @gavinchou