feat: Faster UDP/IO on Apple platforms #1993

larseggert · 2024-09-20T10:25:25Z

This uses Apple's private sendmsg_x and recvmsg_x system calls for multi-packet UDP I/O.

WIP

Ralith · 2024-09-20T19:40:32Z

Is there interest in seeing TX support via sendmsg_x?

We found there wasn't much performance benefit, and was considerable difficulty taking advantage of, sendmmsg-style batching. IIRC the _x functions on macOS have more to offer than that, though. Will this unblock segmentation offload or other incidental optimizations?

quinn-udp/build.rs

larseggert · 2024-09-23T06:02:47Z

Bench on main:

test large_data_10_streams  ... bench:  27,558,791 ns/iter (+/- 13,459,810) = 380 MB/s
test large_data_1_stream    ... bench:  24,324,266 ns/iter (+/- 19,219,937) = 43 MB/s
test small_data_100_streams ... bench:  19,437,900 ns/iter (+/- 20,065,941)
test small_data_1_stream    ... bench:  11,465,128 ns/iter (+/- 8,699,934)

Bench with this PR:

test large_data_10_streams  ... bench:  28,829,216 ns/iter (+/- 15,924,956) = 363 MB/s
test large_data_1_stream    ... bench:  14,354,999 ns/iter (+/- 20,039,122) = 73 MB/s
test small_data_100_streams ... bench:  14,061,741 ns/iter (+/- 17,311,517)
test small_data_1_stream    ... bench:  19,194,441 ns/iter (+/- 5,012,070)

Surprised that large_data_10_streams and small_data_1_stream are slower...

Ralith · 2024-09-23T06:34:07Z

Those tests tend to be extremely noisy, as the huge variance suggests. A targeted quinn-udp benchmark might be more useful.

larseggert · 2024-09-23T06:42:37Z

We've also found on neqo that multi-packet RX without multi-packet TX has limited benefits, since the RX batch size will be very small.

larseggert · 2024-09-23T07:29:18Z

I added sendmsg_x support, mostly to see what the performance difference would be. But it seems that none of the benches or tests call send with a Transmit struct where segment_size is not None?

mxinden · 2024-09-23T09:36:20Z

A targeted quinn-udp benchmark might be more useful.

How about using the throughput.rs benchmark @larseggert?

https://github.com/quinn-rs/quinn/blob/main/quinn-udp/benches/throughput.rs

larseggert · 2024-09-23T12:32:18Z

With @mxinden's benchmark. Baseline:

gso_true/throughput     time:   [58.076 ms 58.230 ms 58.387 ms]
                        thrpt:  [171.27 MiB/s 171.73 MiB/s 172.19 MiB/s]

Only sendmsg_x:

gso_true/throughput     time:   [15.143 ms 15.189 ms 15.236 ms]
                        thrpt:  [656.35 MiB/s 658.36 MiB/s 660.37 MiB/s]
                 change:
                        time:   [-74.028% -73.915% -73.808%] (p = 0.00 < 0.05)
                        thrpt:  [+281.80% +283.36% +285.04%]
                        Performance has improved.

Both sendmsg_x and recvmsg_x:

gso_true/throughput     time:   [12.632 ms 12.682 ms 12.731 ms]
                        thrpt:  [785.46 MiB/s 788.53 MiB/s 791.61 MiB/s]
                 change:
                        time:   [-78.321% -78.221% -78.112%] (p = 0.00 < 0.05)
                        thrpt:  [+356.88% +359.16% +361.27%]
                        Performance has improved.

Both sendmsg_x and recvmsg_x with BATCH_SIZE of 64:

gso_true/throughput     time:   [11.640 ms 11.682 ms 11.725 ms]
                        thrpt:  [852.85 MiB/s 856.00 MiB/s 859.07 MiB/s]
                 change:
                        time:   [-80.030% -79.938% -79.844%] (p = 0.00 < 0.05)
                        thrpt:  [+396.13% +398.45% +400.75%]
                        Performance has improved.

djc

LGTM.

Please squash all of the changes into a single commit?

quinn-udp/benches/throughput.rs

quinn-udp/src/unix.rs

mxinden

Impressive results. Great to see the MacOS _x syscalls work for QUIC UDP IO.

quinn-udp/src/unix.rs

Ralith

Can we enable real GSO/GRO using these interfaces?

quinn-udp/benches/throughput.rs

larseggert · 2024-09-24T04:51:53Z

No. They are the equivalent of the mmsg Linux calls. AFAIK Apple doesn't have GSO/GRO via the socket interface.

quinn-udp/src/cmsg/unix.rs

quinn-udp/src/unix.rs

quinn-udp/benches/throughput.rs

…o feat-apple-datapath

larseggert · 2024-10-01T16:24:34Z

Are you waiting on anything from me on this?

mxinden

quinn-udp/benches/throughput.rs will need more changes to still support non-apple platform. @larseggert I believe we will need to either run it multi-threaded, or use some kind of executor, e.g. tokio. I can prepare a commit in the next couple of days. Sorry for missing this in earlier reviews.

Changes itself look good to me.

quinn-udp/benches/throughput.rs

mxinden · 2024-10-03T18:52:14Z

quinn-udp/benches/throughput.rs

@@ -27,8 +27,12 @@ pub fn criterion_benchmark(c: &mut Criterion) {
    // Reverse non-blocking flag set by `UdpSocketState` to make the test non-racy
    recv.set_nonblocking(false).unwrap();

-    let mut receive_buffer = vec![0; MAX_BUFFER_SIZE];
-    let mut meta = RecvMeta::default();
+    let mut receive_buffers = vec![[0; SEGMENT_SIZE]; BATCH_SIZE];


We set the recv socket to blocking above, to prevent potential race conditions.

recv.set_nonblocking(false).unwrap();

This is problematic on Linux, given that:

A blocking recvmmsg() call blocks until vlen messages have been
received or until the timeout expires. A nonblocking call reads
as many messages as are available (up to the limit specified by
vlen) and returns immediately.

https://man7.org/linux/man-pages/man2/recvmmsg.2.html

Given that quinn-udp does not do sendmmsg, send_state.send will never send BATCH_SIZE messages and thus recv_state.recv will never return.

@mxinden do you have a suggestion for a fix?

@larseggert will be fixed with larseggert#1.

Thanks!

Also, since all the tests are green, I guess quinn is missing a unit test to detect this breakage...

Above referenced pull request addressing this issue moved to #2010.

quinn-udp/src/unix.rs

Co-authored-by: Max Inden <[email protected]>

quinn-udp/src/unix.rs

Done.

djc · 2024-10-09T14:32:56Z

@Ralith can you do another round on this one?

larseggert · 2024-10-10T05:16:33Z

Once @mxinden's fix to the bench is in, I will rebase and squash this PR.

AndrewDryga · 2024-10-10T17:43:56Z

Hey guys 👋, is there any chance Apple won't approve apps that are using those private syscalls in the App Store? They are notorious for doing so and even de-listing apps for using anything "undocumented". See 2.5.1 here: https://developer.apple.com/app-store/review/guidelines/

See one of such cases here: https://9to5mac.com/2019/11/04/electron-app-rejections/

How they will find out? Apple employs automated tools to scan apps for the usage of private APIs. If sendmsg_x and recvmsg_x are detected, the app is at risk of being flagged.

larseggert · 2024-10-10T17:49:20Z

The use of the private syscalls is now behind a non-default feature.

AndrewDryga · 2024-10-10T17:51:22Z

@larseggert should we add a big fat warning saying that if you enable this flag you will violate Apple ToS so it's only should be enabled if app is not distributed via App Store (or notarized for EU)?

larseggert · 2024-10-10T17:55:06Z

I have no opinion here - we don't distribute via the App Store.

Could you make a suggestion on what you think would be good to add?

thomaseizinger · 2024-10-10T20:19:43Z

The use of the private syscalls is now behind a non-default feature.

Thank you for making it optional! It can be an issue to toggle code-paths like these with cargo features because they get aggregated across the entire dependency tree. Simply adding another dependency that also happens to depend on quinn-udp can thus activate this silently. (As a rule of thumb, cargo features should only add new APIs, not modify existing ones so this doesn't happen.)

For some prior art, curve25519-dalek had to solve similar challenges: https://github.com/dalek-cryptography/curve25519-dalek/tree/main/curve25519-dalek#backends

It looks like it is already abstracted away pretty well. Could we change this to a "plain" rustc cfg that needs to be set at build-time?
I am happy for it to be opt-out. Most apps using this probably aren't in the app store and toggling it off is a one-liner in the build process. Ultimately, I got no opinion though on opt-in/opt-out.

larseggert added 4 commits September 20, 2024 13:24

feat: Faster UDP/IO on Apple platforms

aa19b97

WIP

no_main on non-Apple platforms

c36d954

Empty main on non-Apple platforms

adf614d

Use libc::msghdr and libc::iovec on non-Apple platforms

c8dba51

Ralith reviewed Sep 20, 2024

View reviewed changes

quinn-udp/build.rs Outdated Show resolved Hide resolved

larseggert added 2 commits September 23, 2024 08:50

Static bindings

c642dbb

Undo

0cc5f78

larseggert added 3 commits September 23, 2024 10:25

sendmsg_x

a5523fd

No panic

450b754

fmt

de86e7f

larseggert marked this pull request as ready for review September 23, 2024 09:10

Merge branch 'main' into feat-apple-datapath

8979a14

Many fixes

fec6063

larseggert added 2 commits September 23, 2024 15:32

bench fix

83ae3e7

Fix typo

05dc111

djc approved these changes Sep 23, 2024

View reviewed changes

quinn-udp/benches/throughput.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

larseggert added 3 commits September 23, 2024 16:03

Remove commented-out code

f0ff0d7

Address review comment

ff73229

Remove comment

8df7cab

mxinden reviewed Sep 23, 2024

View reviewed changes

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

Ralith previously requested changes Sep 23, 2024

View reviewed changes

quinn-udp/benches/throughput.rs Outdated Show resolved Hide resolved

djc reviewed Sep 24, 2024

View reviewed changes

quinn-udp/src/cmsg/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

Ralith reviewed Sep 25, 2024

View reviewed changes

quinn-udp/benches/throughput.rs Outdated Show resolved Hide resolved

quinn-udp/benches/throughput.rs Outdated Show resolved Hide resolved

larseggert added 2 commits September 25, 2024 08:39

Undo

8e88f60

Merge branch 'feat-apple-datapath' of github.com:larseggert/quinn int…

f1b8448

…o feat-apple-datapath

larseggert requested a review from Ralith October 1, 2024 16:24

larseggert mentioned this pull request Oct 3, 2024

perf: don't allocate in UDP send & recv path mozilla/neqo#2093

Draft

mxinden reviewed Oct 3, 2024

View reviewed changes

mxinden mentioned this pull request Oct 8, 2024

GSO on macOS #1554

Open

larseggert and others added 6 commits October 8, 2024 17:43

Update quinn-udp/benches/throughput.rs

3b5ed21

Co-authored-by: Max Inden <[email protected]>

Update quinn-udp/src/unix.rs

c69940e

Co-authored-by: Max Inden <[email protected]>

Update quinn-udp/src/unix.rs

04be034

Co-authored-by: Max Inden <[email protected]>

Fixes

155708e

Merge branch 'main' into feat-apple-datapath

15e0a66

Add fast-apple-datapath feature

d191358

mxinden reviewed Oct 8, 2024

View reviewed changes

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

quinn-udp/src/unix.rs Outdated Show resolved Hide resolved

mxinden mentioned this pull request Oct 8, 2024

bench(udp): support all GSO and recvmmsg permutations across platforms larseggert/quinn#1

Closed

larseggert added 5 commits October 9, 2024 09:27

Use cfg_aliases to simplify the code

5d9677d

Nits

25db226

bsd macro

137422d

fmt

c10487a

Merge branch 'main' into feat-apple-datapath

32edcd7

mxinden mentioned this pull request Oct 10, 2024

bench(udp): run GSO, GRO and recvmmsg permutations #2010

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Faster UDP/IO on Apple platforms #1993

feat: Faster UDP/IO on Apple platforms #1993

larseggert commented Sep 20, 2024 •

edited

Loading

Ralith commented Sep 20, 2024

larseggert commented Sep 23, 2024

Ralith commented Sep 23, 2024

larseggert commented Sep 23, 2024

larseggert commented Sep 23, 2024

mxinden commented Sep 23, 2024

larseggert commented Sep 23, 2024 •

edited

Loading

djc left a comment

mxinden left a comment

Ralith left a comment

larseggert commented Sep 24, 2024

larseggert commented Oct 1, 2024

mxinden left a comment

mxinden Oct 3, 2024

larseggert Oct 9, 2024

mxinden Oct 9, 2024

larseggert Oct 9, 2024

mxinden Oct 10, 2024

djc commented Oct 9, 2024

larseggert commented Oct 10, 2024 •

edited

Loading

AndrewDryga commented Oct 10, 2024 •

edited

Loading

larseggert commented Oct 10, 2024

AndrewDryga commented Oct 10, 2024 •

edited

Loading

larseggert commented Oct 10, 2024

thomaseizinger commented Oct 10, 2024

feat: Faster UDP/IO on Apple platforms #1993

Are you sure you want to change the base?

feat: Faster UDP/IO on Apple platforms #1993

Conversation

larseggert commented Sep 20, 2024 • edited Loading

Ralith commented Sep 20, 2024

larseggert commented Sep 23, 2024

Ralith commented Sep 23, 2024

larseggert commented Sep 23, 2024

larseggert commented Sep 23, 2024

mxinden commented Sep 23, 2024

larseggert commented Sep 23, 2024 • edited Loading

djc left a comment

Choose a reason for hiding this comment

mxinden left a comment

Choose a reason for hiding this comment

Ralith left a comment

Choose a reason for hiding this comment

larseggert commented Sep 24, 2024

larseggert commented Oct 1, 2024

mxinden left a comment

Choose a reason for hiding this comment

mxinden Oct 3, 2024

Choose a reason for hiding this comment

larseggert Oct 9, 2024

Choose a reason for hiding this comment

mxinden Oct 9, 2024

Choose a reason for hiding this comment

larseggert Oct 9, 2024

Choose a reason for hiding this comment

mxinden Oct 10, 2024

Choose a reason for hiding this comment

djc commented Oct 9, 2024

larseggert commented Oct 10, 2024 • edited Loading

AndrewDryga commented Oct 10, 2024 • edited Loading

larseggert commented Oct 10, 2024

AndrewDryga commented Oct 10, 2024 • edited Loading

larseggert commented Oct 10, 2024

thomaseizinger commented Oct 10, 2024

larseggert commented Sep 20, 2024 •

edited

Loading

larseggert commented Sep 23, 2024 •

edited

Loading

larseggert commented Oct 10, 2024 •

edited

Loading

AndrewDryga commented Oct 10, 2024 •

edited

Loading

AndrewDryga commented Oct 10, 2024 •

edited

Loading