Would use_system_route_table: true be coming to FreeBSD any time soon?

[yaroslav-gwit]

I have a use case for Nebula on FreeBSD, where it is used as an overlay network between the virtualisation hosts. NAT and firewall aspects are covered by the pf, and I am using unsafe_routes to populate any NAT enabled VM networks present on 1 host to all the other hosts in the cluster. So far, so good, and everything works as expected.

Yesterday I've tried to integrate VxLAN into the setup, to make the VM failover more robust, and to create new endpoints on the fly without the need to sync them every time there is a change. VxLAN in the multicast mode didn't work (have not tried Unicast, because it's outside of my use case). Nebula blocks the connection due to the multicast address not being present in the CIDR, or the unsafe_routes.

So finally the question is: are there any plans to integrate use_system_route_table: true with FreeBSD? If not, is there any way to disable the integrated route/IP security checks, in order to get the VxLAN working?

Thanks in advance for your answers.

[johnmaguire]

So finally the question is: are there any plans to integrate use_system_route_table: true with FreeBSD?

AFAIK, there are no current plans to support FreeBSD. You are welcome to file a feature request. This feature relies on netlink. It does look like FreeBSD recently added support for netlink, however the Go library we use for netlink does not seem to support FreeBSD.

If not, is there any way to disable the integrated route/IP security checks, in order to get the VxLAN working?

Can you share logs of the security checks you're referring to? The only other knob I'm aware of for configuring unsafe_routes is the install flag which can be set false if you are planning to manage your routes by hand. Nebula needs to learn the subnet gateway (via) somehow. You may also want to look at tun.drop_multicast.

[yaroslav-gwit]

I am referring to this log line:

time="2023-10-05T23:10:57+01:00" level=debug msg="dropping outbound packet, vpnIp not in our CIDR or in unsafe routes" fwPacket="&{192.168.100.13 225.0.0.1 59303 4789 17 false}" vpnIp=225.0.0.1

tun.drop_multicast is already set to false in my config file, so it should not be an issue, right?

tun.unsafe_routes.[route].install = false could really be an answer here, because it will add the route to Nebula's ACLs (so we can pass the security checks, and not drop the packet(s)), but at same time Nebula will not attempt adding it to the system's routing table.

I'll check if it works shortly. Thank you for pointing me to the right direction.

UPDATE: it didn't work. Still gives me the error above. Tried both options (v1 including 225.0.0.0/24 for every machine, v2 including 225.0.0.0/24 for a single machine):

  unsafe_routes:
  - route: 10.0.101.254/24
    via: 192.168.100.2
  - route: 225.0.0.0/24
    via: 192.168.100.2
    install: false
  - route: 10.0.5.0/24
    via: 192.168.100.3
  - route: 225.0.0.0/24
    via: 192.168.100.3
    install: false

  unsafe_routes:
  - route: 10.0.101.254/24
    via: 192.168.100.2
  - route: 225.0.0.0/24
    via: 192.168.100.2
    install: false
  - route: 10.0.5.0/24
    via: 192.168.100.3