Transparently proxying UDP traffic with pf? #1543

Orum · 2024-05-29T05:35:54Z

This is a similar symptom to my earlier issue (#1473), but as the circumstances have drastically changed I thought I'd open a new issue. To summarize the changes, I am no longer running sslocal on the host (Linux/iptables) machine; it's instead being run on my FreeBSD router which uses pf.

Obviously, this requires different firewall rules to work, but there is little if any documentation I could find for transparently proxying via pf within the shadowsocks-rust project. What I've done here is largely guesswork, but this is the rule I came up with to transparently proxy both TCP & UDP to sslocal (and then on to ssserver):
rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080

sslocal is being run on the router via the following command:
sslocal -b 127.0.0.1:1080 -U --protocol redir -s 192.168.x.y:8388 -m none --tcp-redir pf --udp-redir pf

...and finally, ssserver is run with -U -m none -b 192.168.x.y:8388

This "works" in the sense that both TCP & UDP traffic are being redirected to sslocal (unlike in #1473 where only TCP traffic was ever reaching sslocal), which is then sending the traffic on to sserver. However, once again, only TCP traffic is being fully proxied correctly.

UDP traffic does arrive at sslocal without any doubt, as it shows up when running sslocal with -vvv like so (timestamps removed for brevity, and the host system's address being substituted here with simply <host>):

TRACE  tokio-runtime-worker ThreadId(05) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 127.0.0.1:1080, control: UdpSocketControlData { client_session_id: 7043831170210357480, server_session_id: 0, packet_id: 1, user: None }, payload length 96 bytes, packet length 103 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:307: received UDP packet from <host>:44849, destination 127.0.0.1:1080, length 96 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay <host>:44849 -> 127.0.0.1:1080 (proxied) with 96 bytes

This traffic is then forwarded on to ssserver, which I can verify by watching outgoing traffic on the router via tcpdump. After they arrive at ssserver though, they fail to reach the internet, so clearly something isn't configured correctly.

My best guess as to what is happening here, based on the logs, is that sslocal thinks the destination of the UDP packet is 127.0.0.1 (i.e. the address that sslocal is running on and where incoming UDP traffic on the router is redirected to), and fails to retrieve the original destination address before pf's rdr rule took effect. This wouldn't surprise me as my rdr rule is complete guesswork, so I assume I'm missing something.

If that is indeed the problem, what should the rdr rule look like to transparently proxy UDP traffic with pf? Or am I barking up the wrong tree, and is the problem due to something else entirely?

The text was updated successfully, but these errors were encountered:

zonyitoo · 2024-05-29T06:40:43Z

I have never tested on FreeBSD, so this project supports FreeBSD is still "theoretically".

I have tested pf on macOS, which supports TCP perfectly just like you did on FreeBSD. But on macOS, UDP's destination must be retrieved from ioc_getstates. This API is far different from FreeBSD's definition, so I think FreeBSD doesn't use the same mechanism to get UDP packets' destination when redirected by pf.

In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.

The 127.0.0.1:1080 in your case was exactly the value of msg_name returned from recv_dest_from.

Orum · 2024-05-30T01:03:04Z

In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.

That wasn't me, so you must be thinking of someone else. While I know how to write rules for pf fairly well, I've never programmed anything to interface with it, so I'm unsure of how the API is supposed to be used.

The good news is that you can access the man pages online. There's also a forum, mailing lists, and a Discord server, all of which feature some form of development/programming discussion.

I'm interested in learning as well, but I'm still learning rust, so it may be some time until I am of use.

zonyitoo · 2024-06-01T03:29:14Z

Glad to know that you are familiar with FreeBSD and pf. Could you find any doc / references about how to get the "original destination address" of UDP packets?

On Linux, it is quite straight forward: https://man7.org/linux/man-pages/man7/ip.7.html (IP_RECVORIGDSTADDR).

On macOS, I got it from another post: (?) Deleted.

A famous tools mitmproxy only supports TCP on FreeBSD: https://docs.mitmproxy.org/stable/howto-transparent/ .

So it is quite hard to find a reference about how FreeBSD handles UDP redirects and how to get the original destination address programmatically.

zonyitoo · 2024-06-01T03:47:30Z

And you could also try the ipfw: https://forums.freebsd.org/threads/retrieving-the-destination-port-in-an-ipfwed-udp-packet.37545/

zonyitoo · 2024-06-14T06:23:19Z

@Orum Do you still interest in making this Project working on FreeBSD? What's the current status?

Orum · 2024-06-14T06:31:19Z

I am still very interested in making it work. However, I think the best way to do so at this point is to just read through FreeBSD's source code. As such, I'm familiarizing myself with it, but it's going to take some time as I have other obligations and projects demanding my attention, so it's on the back burner for me right now. This is probably the best, or at least, surest way to figure out how to get the address (assuming it's even possible at present) from pf.

Additionally, I'd need to learn much more about rust before I contribute anything other than information to this project, as my knowledge of it is spotty at best. But, if I do figure out the FreeBSD side of the picture, perhaps someone here can handle that side of things once they have the information. I'm also not only willing, but happy to test things to see if they work once they're implemented.

ge9 · 2024-06-14T07:27:16Z

semigodking/redsocks#200
FYI, I succeeded in running transparent proxy in FreeBSD with redsocks+SOCKS5 (sorry they are completely unrelated to shadowsocks) and ipfw, but not with pf. When I tested with pf, the UDP destination address was obtained as the transparent proxy's listen address, just as @Orum mentioned.
As I mentioned in the link, in OpenBSD, where pf originates from, using divert-to rule of pf worked perfectly. divert rule seems not ported to FreeBSD's pf.

Orum · 2024-06-14T07:44:34Z

FreeBSD and OpenBSD diverged in their rules syntax quite a few years back when the latter made some changes that broke older rule sets, IIRC. I've stuck with FreeBSD, which might have an equivalent of divert-to (or not), but I don't really know what it does as I never bothered to keep up with OBSD's syntax as I don't use it.

I also don't really want to switch to ipfw, or any other firewall, as I really value simplicity and clarity of pf's rules' syntax. So many other firewalls are hard to decipher, which is undesirable to say the least when it comes to security.

In any case I will try and dedicate some time to reading through and understanding the pf/bpf code in the future, but that will be some time from now.

zonyitoo · 2024-06-14T07:54:34Z

@ge9 Have you tried shadowsocks-rust with ipfw? What problem did you see?

ge9 · 2024-06-16T06:13:30Z

I tested it and it worked, but there were a problem.
shadowsocks-rust uses IPv4-mapped IPv6 address to return message to IPv4 clients:

shadowsocks-rust/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs

Line 199 in ac67a08

    
           peer_addr = SocketAddr::new(v4_peer_addr.ip().to_ipv6_mapped().into(), v4_peer_addr.port());

However, FreeBSD disables IPv4-mapped IPv6 address by default (according to scala-native/scala-native#3630). If I set sysctl net.inet6.ip6.v6only=0, then it worked.
Except for this, normal transparent proxy setting will work.

FYI, My settings are following:

Initial setup (the machine already had 10.0.2.15/24)

#!/bin/sh
kldload ipfw
fwcmd=ipfw
ifconfig em0 alias 10.0.2.25 netmask 0xffffff00
$fwcmd add 100 allow all from any to any via lo0
$fwcmd add 500 fwd 127.0.0.1,22222 tcp from 10.0.2.25 to any
$fwcmd add 600 fwd 127.0.0.1,22222 udp from 10.0.2.25 to any
$fwcmd add 700 allow ip from any to any

sslocal -b 127.0.0.1:22222 -U --protocol redir -s 192.168.1.7:1080 --tcp-redir pf --udp-redir pf -m none
- (specifying pf seems fine)
ssserver -U -s 192.168.1.7:1080 -m none (on another PC)
- note that specifying 0.0.0.0:1080 here doesn't work

zonyitoo · 2024-06-16T08:02:06Z

There is a SUPPORT_IPV6_TRANSPARENT flag that indicates whether the current platform supports IPv6, if it is true, then shadowsocks-rust will always convert IPv4 to IPv4-mapped-IPv6 and uses only one socket for sending back.

This is always working for Linux, because dual-stack socket was enabled by default for most distribution of Linux.

Is there a way to test if dual-stack is enabled on FreeBSD? By reading sysctl net.inet6.ip6.v6only=0 should be one possible option.

zonyitoo · 2024-06-16T08:18:26Z

Could you please help to run the test program on FreeBSD and see if any of these syscalls returned errors?
https://stackoverflow.com/questions/30184377/how-to-detect-if-dual-stack-socket-is-supported

zonyitoo · 2024-06-16T08:21:35Z

cc @madeye , FreeBSD doesn't support IPv4-mapped-IPv6 by default. So my previous (years ago) PR that uses IPv6 sockets for sending back UDP packets won't be able to run properly on FreeBSD.

ge9 · 2024-06-16T09:47:18Z

This is the result of the python test program.

Traceback (most recent call last):
  File "/root/test.py", line 4, in <module>
    s.connect(('::ffff:169.254.1.1', 53))
OSError: [Errno 22] Invalid argument

If I set sysctl net.inet6.ip6.v6only=0 and commented out s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1), it worked. Otherwise, it failed.

In the shadowsocks, this is the error.

2024-06-17T03:45:10.334750916+09:00  WARN udp failed to send back 68 bytes to client 10.0.2.25:44550, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

This originates from

shadowsocks-rust/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs

Line 203 in ac67a08

inbound.send_to(data, peer_addr).await.map(|n| {

.

- ref #1543

zonyitoo · 2024-06-16T10:44:42Z

Please test if it is Ok without settingnet.inet6.ip6.v6only=0 manually.

- ref #1543

zonyitoo · 2024-06-16T11:15:35Z

I just found a better solution. Working on it.

- ref #1543 - Reference Implementation: golang src/net/ipsock_posix.go ipStckCapabilities

- ref #1543

zonyitoo · 2024-06-16T12:07:03Z

Tested on OpenWRT. Please help testing it if it is working correctly on FreeBSD. @ge9

ge9 · 2024-06-16T12:34:29Z

5ba8b7d worked but 765c9e5 or cd25d25 (latest) doesn't (same error).

zonyitoo · 2024-06-16T12:36:38Z

Please run again with -v and check these debug logs:

shadowsocks-rust/crates/shadowsocks/src/net/sys/mod.rs

Lines 143 to 181 in cd25d25

    
           static IP_STACK_CAPABILITIES: Lazy<IpStackCapabilities> = Lazy::new(|| { 
        
               // Reference Implementation: https://github.com/golang/go/blob/master/src/net/ipsock_posix.go 
        
               let mut caps = IpStackCapabilities { 
        
                   support_ipv4: false, 
        
                   support_ipv6: false, 
        
                   support_ipv4_mapped_ipv6: false, 
        
               }; 
        
               // Check IPv4 
        
               if let Ok(_) = Socket::new(Domain::IPV4, Type::STREAM, Some(Protocol::TCP)) { 
        
                   caps.support_ipv4 = true; 
        
                   debug!("IpStackCapability support_ipv4=true"); 
        
               } 
        
               // Check IPv6 (::1) 
        
               if let Ok(ipv6_socket) = Socket::new(Domain::IPV6, Type::STREAM, Some(Protocol::TCP)) { 
        
                   if let Ok(..) = ipv6_socket.set_only_v6(true) { 
        
                       let local_host = SockAddr::from(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0)); 
        
                       if let Ok(..) = ipv6_socket.bind(&local_host) { 
        
                           caps.support_ipv6 = true; 
        
                           debug!("IpStackCapability support_ipv6=true"); 
        
                       } 
        
                   } 
        
               } 
        
               // Check IPv4-mapped-IPv6 (127.0.0.1) 
        
               if let Ok(ipv6_socket) = Socket::new(Domain::IPV6, Type::STREAM, Some(Protocol::TCP)) { 
        
                   if let Ok(..) = ipv6_socket.set_only_v6(false) { 
        
                       let local_host = SockAddr::from(SocketAddr::new(Ipv4Addr::LOCALHOST.to_ipv6_mapped().into(), 0)); 
        
                       if let Ok(..) = ipv6_socket.bind(&local_host) { 
        
                           caps.support_ipv4_mapped_ipv6 = true; 
        
                           debug!("IpStackCapability support_ipv4_mapped_ipv6=true"); 
        
                       } 
        
                   } 
        
               } 
        
               caps 
        
           });

Are they all true?

ge9 · 2024-06-16T12:39:45Z

here is the log. seems all true.

2024-06-17T06:38:52.894052461+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:65158, destination 3.132.228.249:3478, length 28 bytes
2024-06-17T06:38:52.894445849+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:65158
2024-06-17T06:38:52.894817377+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:65158 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-17T06:38:52.894896617+09:00 TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555479936), interests=READABLE | WRITABLE
2024-06-17T06:38:52.895050148+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.8:1080 (outbound: 192.168.1.8:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T06:38:52.895129216+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 786820869274048462, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-17T06:38:53.135364801+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-17T06:38:53.136228332+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:65158 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-17T06:38:53.136421625+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-17T06:38:53.136798744+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-17T06:38:53.137187564+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:175: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-17T06:38:53.13739486+09:00 TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555480192), interests=READABLE | WRITABLE
2024-06-17T06:38:53.137685127+09:00  WARN  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:65158, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

zonyitoo · 2024-06-16T12:43:26Z

Well, so FreeBSD allows bind()ing a IPv4-mapped-IPv6 address, but doesn’t allow sendmsg to it. Programs that are written in Go should also fails with the same reason.

- ref #1543

zonyitoo · 2024-06-16T13:15:01Z

Call connect() in this commit. Please help verify it again.

ge9 · 2024-06-16T14:01:45Z

Hmm it seems still not working...

2024-06-16T23:01:19.392825484Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:13870, destination 3.132.228.249:3478, length 28 bytes
2024-06-16T23:01:19.393494564Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:13870
2024-06-16T23:01:19.393869192Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:13870 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-16T23:01:19.394037649Z TRACE  tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799232), interests=READABLE | WRITABLE
2024-06-16T23:01:19.394211694Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-16T23:01:19.394416748Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 8860848107368566370, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-16T23:01:19.566423525Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-16T23:01:19.567810852Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:13870 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-16T23:01:19.568463729Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:156: IpStackCapability support_ipv4=true
2024-06-16T23:01:19.568883894Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:165: IpStackCapability support_ipv6=true
2024-06-16T23:01:19.56980524Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:173: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-16T23:01:19.569994091Z TRACE  tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799488), interests=READABLE | WRITABLE
2024-06-16T23:01:19.578334536Z  WARN  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:13870, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

- ref #1543

zonyitoo · 2024-06-16T14:40:49Z

Replace it with the method just like the one on stackoverflow. Please try again. @ge9

ge9 · 2024-06-17T17:13:23Z

I have one thing to add, actually I'm testing transparent proxy with only one machine, by making the machine send packet to itself through the loopback device. 10.0.2.25 is the source address to which the proxy will be applied. Then we can test proxy by some commands like nc -s 10.0.2.25 192.168.1.1 8888.
FYI, this is my FreeBSD pf (not ipfw) rule for testing.

rdr pass on lo0 proto {tcp, udp} from 10.0.2.25 -> 127.0.0.1 port 22222
pass out quick route-to lo0 from 10.0.2.25
pass

zonyitoo · 2024-06-18T01:49:36Z

BTW, mark-based routing on FreeBSD is also supported, which should be something like Linux's SO_MARK:

shadowsocks-rust/crates/shadowsocks/src/net/sys/unix/bsd/freebsd.rs

Lines 43 to 59 in a27410c

    
           // Set SO_USER_COOKIE for mark-based routing on FreeBSD 
        
           if let Some(user_cookie) = opts.user_cookie { 
        
               let ret = unsafe { 
        
                   libc::setsockopt( 
        
                       socket.as_raw_fd(), 
        
                       libc::SOL_SOCKET, 
        
                       libc::SO_USER_COOKIE, 
        
                       &user_cookie as *const _ as *const _, 
        
                       mem::size_of_val(&user_cookie) as libc::socklen_t, 
        
                   ) 
        
               }; 
        
               if ret != 0 { 
        
                   let err = io::Error::last_os_error(); 
        
                   error!("set SO_USER_COOKIE error: {}", err); 
        
                   return Err(err); 
        
               } 
        
           }

It should allow you to easily route the outbound requests sent from sslocal to the actual outbound interface.

But I really don't know how to set it in pf or ipfw.

  SO_USER_COOKIE can be used to set the uint32_t so_user_cookie field  in
       the  socket.   The  value is an uint32_t, and can be used in the	kernel
       code that manipulates traffic related to	the socket.  The default value
       for the field is	0.  As an example, the value can be used as the	skipto
       target or pipe number in	ipfw/dummynet.

ipfw document about this SO_USER_COOKIE:

	skipto number | tablearg
	       Skip  all  subsequent  rules  numbered  less  than number.  The
	       search continues	with the first rule numbered number or higher.
	       It is possible to use the tablearg keyword with a skipto	for  a
	       computed	 skipto.   Skipto  may	work either in O([log(N)](https://man.freebsd.org/cgi/man.cgi?query=log&sektion=N&apropos=0&manpath=FreeBSD+14.1-RELEASE+and+Ports)) or in
	       [O(1)](https://man.freebsd.org/cgi/man.cgi?query=O&sektion=1&apropos=0&manpath=FreeBSD+14.1-RELEASE+and+Ports) depending on amount	of  memory  and/or  sysctl  variables.
	       See the "SYSCTL VARIABLES" section for more details.

zonyitoo · 2024-06-18T15:32:51Z

My VM has IP 192.168.11.191, and sslocal ran with the following command:

./target/debug/ssservice local -b '127.0.0.1:22222' -s '192.168.11.1:8388' -m none --protocol redir -v -U

I want to test UDP redirects locally on FreeBSD,

On the other hand, /etc/pf.conf was setting like this:

rdr pass on lo0 proto {udp} from 192.168.11.191 -> 127.0.0.1 port 22222
pass out quick route-to lo0 from 192.168.11.191
pass

It doesn't work, which seems to create an infinite loop.

ge9 · 2024-06-18T15:39:52Z

Did you confirm an infinite loop actually happened? The config seems fine to me.
Packets coming out from 192.168.11.191 will be routed on lo0 and then redirected to 127.0.0.1:22222, being received by sslocal.

zonyitoo · 2024-06-18T15:48:05Z

2024-06-18T23:31:54.173984822Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:50143 is closed
2024-06-18T23:31:54.174065558Z DEBUG tokio-runtime-worker ThreadId(09) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:15262 is closed
2024-06-18T23:31:54.174056618Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:55806 is closed
2024-06-18T23:31:54.174088187Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:28037 is closed
2024-06-18T23:31:54.174128695Z DEBUG tokio-runtime-worker ThreadId(06) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:12287 is closed
2024-06-18T23:31:54.174117241Z DEBUG tokio-runtime-worker ThreadId(04) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:51797 is closed
2024-06-18T23:31:54.174116682Z DEBUG tokio-runtime-worker ThreadId(05) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:35430 is closed
2024-06-18T23:31:54.17421446Z DEBUG tokio-runtime-worker ThreadId(09) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:54599 is closed
2024-06-18T23:31:54.174220885Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:15870 is closed
2024-06-18T23:31:54.174257761Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:43392 is closed
2024-06-18T23:31:54.174318663Z DEBUG tokio-runtime-worker ThreadId(06) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:15490 is closed

Yes, it do exist.

ge9 · 2024-06-18T15:50:25Z

Ah, did you assigned an additional IP? I assigned both 10.0.2.15 and 10.0.2.25 on the external interface. 10.0.2.15 was assigned by default (by Virtualbox's DHCP) and 10.0.2.25 is manually assigned, thus can be used for proxying.

zonyitoo · 2024-06-18T15:51:24Z

Nope. 192.168.11.192 was assigned by DHCP. So I should add another IP to em0?

ifconfig em0 192.168.11.20 netmask 255.255.255.0 alias

zonyitoo · 2024-06-18T16:01:10Z

Ok now, here is the problem. Currently sslocal cannot get the original destination IP address from FreeBSD, so

nc -us 192.168.11.20 192.168.11.1 8888 -v

was successfully redirected to 127.0.0.1:22222, but

2024-06-18T23:54:44.357260908Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: created udp association for 192.168.11.20:45818
2024-06-18T23:54:44.363167525Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: 192.168.11.20:45818 -> 127.0.0.1:22222 (proxied) sending 1 bytes failed, error: Connection refused (os error 61)
2024-06-18T23:54:44.36442355Z ERROR tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: udp relay 192.168.11.20:45818 <- ... (proxied) failed, error: Connection refused (os error 61)

Well, the destination address was 127.0.0.1:22222, so it will again generate a loop if a ssserver was actually there.

How can you test for a response UDP packet sent from remote?

ge9 · 2024-06-18T16:04:56Z

Yes, that's the problem on pf, which I don't know how to solve.
I was testing UDP behavior with ipfw, not pf.

zonyitoo · 2024-06-18T16:05:17Z

I just found that pf has a divert-to action, which will preserve the original destination:

https://man.openbsd.org/pf.conf#divert-to

How to configure a rule with divert-to?

ge9 · 2024-06-18T16:10:04Z

No, it's OpenBSD behavior. FreeBSD's divert-to has similar syntax but completely different meaning, as documented in https://man.freebsd.org/cgi/man.cgi?pf.conf(5) . (Also, from my experience on running redsocks in OpenBSD, supporting OpenBSD+divert-to requires a little work on UDP/TCP socket options).

zonyitoo · 2024-06-19T16:55:59Z

2024-06-19T01:50:51.156486787Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks::relay::udprelay::proxy_socket: UDP server client receive from 192.168.11.186:16700, control: None, packet length 11 bytes, payload length 4 bytes
2024-06-19T01:50:51.165007423Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:48221 <- 192.168.11.186:16700 (proxied) received 4 bytes
2024-06-19T01:50:51.16521583Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks::net::sys: IpStackCapability support_ipv4=true
2024-06-19T01:50:51.16532143Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks::net::sys: IpStackCapability support_ipv6=true
2024-06-19T01:50:51.165420045Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks::net::sys: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-19T01:50:51.169426421Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks_service::local::redir::udprelay: udp redir send back data 4 bytes, remote: 192.168.11.186:16700, peer: [::ffff:10.0.2.25]:48221, socket_opts: RedirSocketOpts
2024-06-19T01:50:51.169623373Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:48221 <- 192.168.11.186:16700 (proxied) with 4 bytes

Just tested on FreeBSD with ipfw. UDP works successfully if: sysctl net.inet6.ip6.v6only=0.

If it is 1:

2024-06-19T01:53:32.434992296Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: IpStackCapability support_ipv4=true
2024-06-19T01:53:32.43515321Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: IpStackCapability support_ipv6=true
2024-06-19T01:53:32.435340943Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-19T01:54:33.632720219Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: UDP server client receive from 192.168.11.186:16700, control: None, packet length 12 bytes, payload length 5 bytes
2024-06-19T01:54:33.632885883Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:53661 <- 192.168.11.186:16700 (proxied) received 5 bytes
2024-06-19T01:54:33.633137591Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::redir::udprelay: udp redir send back data 5 bytes, remote: 192.168.11.186:16700, peer: [::ffff:10.0.2.25]:53661, socket_opts: RedirSocketOpts
2024-06-19T01:54:33.633245146Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:53661 <- 192.168.11.186:16700 (proxied) with 5 bytes

No errors was shown. But the client couldn't receive the response.

In the latter case, I think net.inet6.ip6.v6only=1 will disable IPv4-mapped IPv6 IP packets routing to IPv4 clients.

zonyitoo · 2024-06-19T17:08:06Z

With the test just did in Python:

import socket
s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0)
s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
s.connect(('::ffff:169.254.1.1', 53))
print(s.getsockname())

When net.inet6.ip6.v6only=1:

Script will always success without any errors. The line of setsockopt(IPV6_V6ONLY) could be removed without affecting the result.

If IPV6_V6ONLY was set to 1, connect() will fail with EINVAL, which is expected.

When net.inet6.ip6.v6only=0:

Behavior was exactly the same.

So there is no way to detect with socket, we may have to read sysctl in code.

database64128 · 2024-06-20T01:55:14Z

The behavior seems quite wrong. Maybe we should let FreeBSD developers know about it.

Orum · 2024-06-20T16:05:10Z

They offer a convenient way to do that.

In any case, perhaps this issue report on shadowsocks should be split into two separate issues, one for the original issue of using transparent proxying with pf, and a new one for the IPv4/v6 issue.

zonyitoo · 2024-06-20T16:07:33Z

True. But currently the topic about IPv4-mapped IPv6 support was already come to a conclusion. We should focus on how to support UDP with pf.

ge9 · 2024-06-24T11:45:07Z

The IPv4/v6 issue seems to be fixed now?
By the way, I added support for OpenBSD/pf by doing the same thing as I did for redsocks.
3224af8
I'll make PR if it's OK.

zonyitoo · 2024-06-24T13:56:38Z

@ge9 Of course, please make a PR.

zonyitoo · 2024-06-25T05:16:55Z

Here is an issue listed all known ways to make transparent proxy: 2EXP/2exp.github.io#2 .

pfioc_states seems to work on macOS, but some fields are missing on FreeBSD.

zonyitoo · 2024-06-25T05:30:08Z

https://man.freebsd.org/cgi/man.cgi?query=pf&sektion=4&manpath=OpenBSD

DIOCNATLOOK struct pfioc_natlook	*pnl
	       Look up a state table entry by source and destination addresses
	       and ports.

	       struct pfioc_natlook {
		       struct pf_addr	saddr;
		       struct pf_addr	daddr;
		       struct pf_addr	rsaddr;
		       struct pf_addr	rdaddr;
		       u_int16_t	rdomain;
		       u_int16_t	rrdomain;
		       u_int16_t	sport;
		       u_int16_t	dport;
		       u_int16_t	rsport;
		       u_int16_t	rdport;
		       sa_family_t	af;
		       u_int8_t		proto;
		       u_int8_t		direction;
	       };

	       This  was  primarily  used  to support transparent proxies with
	       rdr-to rules.  New proxies should use divert-to rules  instead.
	       These  do  not  require access to the privileged	/dev/pf	device
	       and   preserve	the   original	 destination	address	   for
	       [getsockname(2)](https://man.freebsd.org/cgi/man.cgi?query=getsockname&sektion=2&apropos=0&manpath=OpenBSD+7.5).	 For  SOCK_DGRAM sockets, the [ip(4)](https://man.freebsd.org/cgi/man.cgi?query=ip&sektion=4&apropos=0&manpath=OpenBSD+7.5) socket op-
	       tions IP_RECVDSTADDR and	IP_RECVDSTPORT can be used to retrieve
	       the destination address and port.

In the document it said IP_RECVDSTADDR and IP_RECVDSTPORT could be used to retrive the destination address and port.

shadowsocks-rust/crates/shadowsocks-service/src/local/redir/udprelay/sys/unix/bsd.rs

Lines 296 to 338 in 8b32d85

    
           fn get_destination_addr(msg: &libc::msghdr) -> io::Result<SocketAddr> { 
        
               // https://www.freebsd.org/cgi/man.cgi?ip(4) 
        
               // 
        
               // Called `recvmsg` with `IP_ORIGDSTADDR` set 
        
               unsafe { 
        
                   let (_, addr) = SockAddr::try_init(|dst_addr, dst_addr_len| { 
        
                       let mut cmsg: *mut libc::cmsghdr = libc::CMSG_FIRSTHDR(msg); 
        
                       while !cmsg.is_null() { 
        
                           let rcmsg = &*cmsg; 
        
                           match (rcmsg.cmsg_level, rcmsg.cmsg_type) { 
        
                               (libc::IPPROTO_IP, libc::IP_ORIGDSTADDR) => { 
        
                                   ptr::copy_nonoverlapping( 
        
                                       libc::CMSG_DATA(cmsg), 
        
                                       dst_addr as *mut _, 
        
                                       mem::size_of::<libc::sockaddr_in>(), 
        
                                   ); 
        
                                   *dst_addr_len = mem::size_of::<libc::sockaddr_in>() as libc::socklen_t; 
        
                                   return Ok(()); 
        
                               } 
        
                               (libc::IPPROTO_IPV6, libc::IPV6_ORIGDSTADDR) => { 
        
                                   ptr::copy_nonoverlapping( 
        
                                       libc::CMSG_DATA(cmsg), 
        
                                       dst_addr as *mut _, 
        
                                       mem::size_of::<libc::sockaddr_in6>(), 
        
                                   ); 
        
                                   *dst_addr_len = mem::size_of::<libc::sockaddr_in6>() as libc::socklen_t; 
        
                                   return Ok(()); 
        
                               } 
        
                               _ => {} 
        
                           } 
        
                           cmsg = libc::CMSG_NXTHDR(msg, cmsg); 
        
                       } 
        
                       let err = Error::new(ErrorKind::InvalidData, "missing destination address in msghdr"); 
        
                       Err(err) 
        
                   })?; 
        
                   Ok(addr.as_socket().expect("SocketAddr")) 
        
               } 
        
           }

But it doesn't work, right? @Orum

       If the IP_RECVDSTADDR option is enabled on  a  SOCK_DGRAM  socket,  the
       [recvmsg(2)](https://man.freebsd.org/cgi/man.cgi?query=recvmsg&sektion=2&apropos=0&manpath=OpenBSD+7.5)  call	will return the	destination IP address for a UDP data-
       gram.  The msg_control field in the msghdr structure points to a	buffer
       that contains a cmsghdr structure followed  by  the  IP	address.   The
       cmsghdr fields have the following values:

	     cmsg_len =	CMSG_LEN(sizeof(struct in_addr))
	     cmsg_level	= IPPROTO_IP
	     cmsg_type = IP_RECVDSTADDR

       If  the	IP_RECVDSTPORT	option	is enabled on a	SOCK_DGRAM socket, the
       [recvmsg(2)](https://man.freebsd.org/cgi/man.cgi?query=recvmsg&sektion=2&apropos=0&manpath=OpenBSD+7.5) call will return the destination port	for  a	UDP  datagram.
       The  msg_control	 field in the msghdr structure points to a buffer that
       contains	a cmsghdr structure followed by	the  port  in  16-bit  network
       byte order.  The	cmsghdr	fields have the	following values:

	     cmsg_len =	CMSG_LEN(sizeof(u_int16_t))
	     cmsg_level	= IPPROTO_IP
	     cmsg_type = IP_RECVDSTPORT

Implementation detail: https://reviews.freebsd.org/D9235

According to this implementation, IP_RECVDSTADDR will returns sockaddr_in with address and port, so IP_RECVDSTPORT is not useful in IPv4. IPv6 implementation only has IPV6_RECVDSTADDR and no IPV6_RECVDSTPORT. But it is different from the document (manpage), which said IP_RECVDSTADDR returns in_addr instead of sockaddr_in.

zonyitoo · 2024-06-25T05:50:30Z

https://github.com/freebsd/freebsd-src/blob/bbecd3148abf68918b1aa5fc7750dd8ec17fea72/sys/netinet/udp_usrreq.c#L289-L300

It is sockaddr_in. So the man doc is wrong?

…TADDR NOTE: They have the same value as IP_ORIGDSTADDR, IPV6_ORIGDSTADDR ref #1543

ge9 · 2024-06-25T06:29:19Z

I didn't know FreeBSD's man page describes UDP transparent proxy treatments. That seems much like OpenBSD's behavior. I may try it.

zonyitoo · 2024-06-25T06:32:16Z

All the references are from FreeBSD's manpage.

ge9 · 2024-06-25T09:32:45Z

I tried using IP_RECVDSTADDR in FreeBSD but obtained IP was still 127.0.0.1. No difference from IP_ORIGDSTADDR. We can enable both at once and only gain 127.0.0.1 from both...🤔
Also, IP_RECVDSTPORT seems not exist in FreeBSD.

zonyitoo · 2024-06-25T14:13:22Z

IP_RECVORIGDSTADDR = IP_ORIGDSTADDR, they have the same value.

but obtained IP was still 127.0.0.1

I think the next step is to findout where the value of udp_in[1] is from.

ge9 · 2024-06-25T14:52:14Z

IP_RECVORIGDSTADDR equals IP_ORIGDSTADDR, but IP_RECVDSTADDR is different. Indeed, messages received by recvmsg() are slightly different, but both include 127.0.0.1.`

zonyitoo · 2024-06-25T15:03:51Z

https://github.com/freebsd/freebsd-src/blob/bbecd3148abf68918b1aa5fc7750dd8ec17fea72/sys/netinet/udp_usrreq.c#L493-L505

udp_in[1] is IP packet's destination address. So I think the next step is: how to let pf send packets to this port without modifying its original destinaion address.

zonyitoo · 2024-06-25T15:07:25Z

Why we cannot obtain the original destination address from DIOCNATLOOK? If we set a NAT rule for UDP, can we make it work?

zonyitoo added a commit that referenced this issue Jun 16, 2024

fix(local-redir): EINVAL if platform doesn't support IPv4-mapped-IPv6

eddd439

- ref #1543

zonyitoo added a commit that referenced this issue Jun 16, 2024

fix(server): check EINVAL if platform doesn't support dual-stack

56cab5a

- ref #1543

zonyitoo added a commit that referenced this issue Jun 16, 2024

feat(local-redir, server): Check IP stack capability globally

5ba8b7d

- ref #1543 - Reference Implementation: golang src/net/ipsock_posix.go ipStckCapabilities

zonyitoo added a commit that referenced this issue Jun 16, 2024

fix(shadowsocks): IpStackCapability check IPv4-mapped-IPv6 properly

765c9e5

- ref #1543

zonyitoo added a commit that referenced this issue Jun 16, 2024

feat(shadowsocks): IpStackCapability check Ipv4-mapped-Ipv6 by connect()

2f2d3cd

- ref #1543

zonyitoo added a commit that referenced this issue Jun 16, 2024

fix(shadowsocks): check IPv4-mapped-IPv6 compatiblity with UDP socket

5c59fba

- ref #1543

ge9 mentioned this issue Jun 24, 2024

add support for OpenBSD/pf #1567

Merged

zonyitoo added a commit that referenced this issue Jun 25, 2024

fix(local-redir): FreeBSD UDP set IP_RECVORIGDSTADDR, IPV6_RECVORIGDS…

6d7bc35

…TADDR NOTE: They have the same value as IP_ORIGDSTADDR, IPV6_ORIGDSTADDR ref #1543

Transparently proxying UDP traffic with pf? #1543

Transparently proxying UDP traffic with pf? #1543

Comments

Orum commented May 29, 2024 • edited Loading

zonyitoo commented May 29, 2024

Orum commented May 30, 2024

zonyitoo commented Jun 1, 2024

zonyitoo commented Jun 1, 2024

zonyitoo commented Jun 14, 2024

Orum commented Jun 14, 2024 • edited Loading

ge9 commented Jun 14, 2024

Orum commented Jun 14, 2024

zonyitoo commented Jun 14, 2024 • edited Loading

ge9 commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

ge9 commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

ge9 commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

ge9 commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

ge9 commented Jun 16, 2024

zonyitoo commented Jun 16, 2024

ge9 commented Jun 17, 2024

zonyitoo commented Jun 18, 2024 • edited Loading

zonyitoo commented Jun 18, 2024

ge9 commented Jun 18, 2024

zonyitoo commented Jun 18, 2024

ge9 commented Jun 18, 2024

zonyitoo commented Jun 18, 2024 • edited Loading

zonyitoo commented Jun 18, 2024

ge9 commented Jun 18, 2024

zonyitoo commented Jun 18, 2024

ge9 commented Jun 18, 2024

zonyitoo commented Jun 19, 2024

zonyitoo commented Jun 19, 2024

database64128 commented Jun 20, 2024

Orum commented Jun 20, 2024

zonyitoo commented Jun 20, 2024

ge9 commented Jun 24, 2024

zonyitoo commented Jun 24, 2024

zonyitoo commented Jun 25, 2024 • edited Loading

zonyitoo commented Jun 25, 2024 • edited Loading

zonyitoo commented Jun 25, 2024

ge9 commented Jun 25, 2024

zonyitoo commented Jun 25, 2024

ge9 commented Jun 25, 2024

zonyitoo commented Jun 25, 2024 • edited Loading

ge9 commented Jun 25, 2024

zonyitoo commented Jun 25, 2024 • edited Loading

zonyitoo commented Jun 25, 2024

Orum commented May 29, 2024 •

edited

Loading

Orum commented Jun 14, 2024 •

edited

Loading

zonyitoo commented Jun 14, 2024 •

edited

Loading

zonyitoo commented Jun 18, 2024 •

edited

Loading

zonyitoo commented Jun 18, 2024 •

edited

Loading

zonyitoo commented Jun 25, 2024 •

edited

Loading

zonyitoo commented Jun 25, 2024 •

edited

Loading

zonyitoo commented Jun 25, 2024 •

edited

Loading

zonyitoo commented Jun 25, 2024 •

edited

Loading