Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Linux Connection ID based RSS #4551

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Conversation

ami-GS
Copy link
Contributor

@ami-GS ami-GS commented Sep 20, 2024

Description

Conventional NIC RSS does not work well with QUIC as it can continue its connection with the Connection ID after changing IP/port.
This PR makes XDP to route packets to appropriate core using Connection ID.

  • XSKMAP (AF_XDP) based redirection doesn't work at least on my environment (netvsc). Working with netvsc folks about inter-queue redirection feature.
  • CPUMAP based redirection works. This is hybrid implementation with datapath_epoll.c

Testing

There is no functional test yet, but secnetperf based test (sometimes) works.

# server side
$ sudo MSQUIC_XDP_RSS_TYPE=1 MSQUIC_XDP_NORMAL_DATAPATH=1 ./artifacts/bin/linux/x64_Debug_openssl3/secnetperf -exec:maxtput -io:xdp -pollidle:1000

# client side
# multi-connection has bug. (e.g. -conns:10)
$ sudo MSQUIC_XDP_DROP_PATH_CHALLENGE=0  MSQUIC_FAKE_NAT_REBINDING=1 ./artifacts/bin/linux/x64_Debug_openssl3/secnetperf -target:<SERVER_IP> -exec:maxtput -down:10s -ptput:1 -tcp:0 -trimout -watchdog:25000 -pconn:1 -pstream:1 -io:xdp -conns:1

Documentation

Not yet

Server side option
MSQUIC_XDP_RSS_TYPE: 0x01 -> using ConnectionID to cpu/queue map which is confirmed after finishing handshake. Not flexible enough for NEW_CONNECTION_ID.
MSQUIC_XDP_RSS_TYPE: 0x08 -> using Hash(ConnectionID). (not implemented yet)
MSQUIC_XDP_NORMAL_DATAPATH:1 -> this uses normal socket with xdp program (not AF_XDP socket).

Client side option
MSQUIC_FAKE_NAT_REBINDING:0/1 -> client changes (changes back) port to 55555 as if NAT rebinding happens
MSQUIC_XDP_DROP_PATH_CHALLENGE:0/1 -> whether to ignore PATH_CHALLENGE frame after NAT rebinding

XskInfo = Packet->Queue->XskInfo;

CxPlatLockAcquire(&XskInfo->UmemLock);
while (PacketChain) {
Packet =
CXPLAT_CONTAINING_RECORD(PacketChain, XDP_RX_PACKET, RecvData);
PacketChain = PacketChain->Next;
CXPLAT_DBG_ASSERT(Packet != NULL);
CXPLAT_DBG_ASSERT(Packet->Queue != NULL);
CXPLAT_DBG_ASSERT(Packet->Queue->XskInfo != NULL);
Copy link
Contributor Author

@ami-GS ami-GS Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XskInfo is sometimes broken when running with multi connection, like -conns:100. This is common issue on main branch

SourceCid->CID.Length,
SourceCid->CID.Data
);
// CXPLAT_FREE(SourceCid, QUIC_POOL_CIDHASH);

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.
Comment on lines +381 to +382
} else {
}

Check notice

Code scanning / CodeQL

Empty branch of conditional Note

Empty block without comment.
@ami-GS
Copy link
Contributor Author

ami-GS commented Sep 23, 2024

When disabling the RSS.
Using different Partition for NAT rebinded packets
image

When enabling the RSS.
Always use same Partition (once connection is established)
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant