Can LinuxHost and Lighthouse be the same in Extend network access beyond overlay hosts ?

[kuolemaaa]

I'm trying to follow this article to access my local network via nebula.

I have my linux host that is also my lighthouse, behind a nat with the correct port forwarded.

LinuxHost: (my nebula lighthouse, that also hosts some web panel)

• 192.168.1.54/24 real local network ip
• 10.192.168.54/24 nebula ip

LocalServer: (my target)

• 192.168.1.20/24 the server that i want to access thru nebula
• no nebula ip

My laptop: (i want to read LocalServer via nebula with it)

• 192.168.66.123/24 it's own local network and ip
• 10.192.168.22/24 nebula ip

I've followed https://nebula.defined.net/docs/guides/unsafe_routes/ this article step by step and indeed I can get into the web panels of the LinuxHost from my laptop (connected to another network) via nebula and I can also ping 192.168.1.54 from the laptop. Unfortunately I cannot reach the LocalServer host at 192.168.1.20.

A peculiar debug log message says: Error while validating outbound packet: packet is not ipv4, type: 6 but I am pretty sure none of my device has ipv6

So, the question is in the title:

Can LinuxHost and Lighthouse be the same in Extend network access beyond overlay hosts ?

And I'm asking because in the example of the article lighthouse and linux host have different nebula ips and different nebula rule (i guess, lighthouse has am_lighthouse:true and linux host has am_lighthose:false ?)

[johnmaguire]

IIRC, Error while validating outbound packet: packet is not ipv4, type: 6 shows up when the Lighthouse has an IPv6 address for a host and another host tries to talk to it, but does not have IPv4 connectivity. This is likely unrelated to your issue.

Yes, Lighthouse hosts can act as unsafe_routes routers. This configuration should work.

indeed I can get into the web panels of the LinuxHost from my laptop (connected to another network) via nebula and I can also ping 192.168.1.54 from the laptop. Unfortunately I cannot reach the LocalServer host at 192.168.1.20.

My first guess in this instance would be a firewall rule preventing access. Are you able to share the Nebula configs from your hosts (with pki.key removed)? Logs would also be helpful to see if anything stands out. It is necessary to add inbound firewall rules permitting access to the unsafe_routes hosts using the local_cidr config option. The guide may be outdated and missing this bit.

Another common issue is not enabling forwarding on the router host. Can you verify the output of the sysctl command here? https://nebula.defined.net/docs/guides/unsafe_routes/#step-4-enable-ip-forwarding-on-linux-host-lan

If you still have issues, the next step would be to start some packet captures and determine where the packets are getting hung up. Something like tcpdump 'host 192.168.1.20' on the router host and tcpdump 'host 192.168.1.54' on LocalServer, then sending some traffic, might be a good start.

[kuolemaaa]

Giving up. Tired. It is not clear to me how nebula works and what nebula is doing with anything outside of the nebula IP addresses. Here a (vanilla-ish) config: managed to make laptop pings lighthouse's nebula ip and lighthouse local ip.

Commands:

• nebula-cert ca -name "remotenat"
• nebula-cert sign -name 'lighthouse0' -ip '10.192.168.254/2'4 -subnets '10.192.168.0/24,192.168.1.0/24'
• nebula-cert sign -name 'user_laptop' -ip '10.192.168.22/24' -subnets '10.192.168.0/24,192.168.1.0/24'

A laptop:

• local ip: 192.168.xxx.yyy (changes often)
• nebula 10.192.168.22

Lighthouse:

• local ip: 192.168.1.254
• nebula ip: 10.192.168.254
  sudo sysctl net.ipv4.ip_forward -> net.ipv4.ip_forward = 1
  EDIT2: actually lightohuse is under NAT, it's listen port, tcp and udp, is forwarded

Target:

• anything in the 192.168.1.0/24 network

Configs:

• laptop:

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    CjsKCXJlbW90ZW5hdCiOx5a3BjCOrpvGBjogmL/UzhpVsNDdOqz508vWIPcJDmN6
    vKdOWFmLHnxEksdAARJAMIHZylJUZW9tyDhmYZf41EcxXdGMB5bsZ61kXeCAvOLI
    i763mSQjq0Vxw423Y84H85waiLBmelMkmNTQU4p/CQ==
    -----END NEBULA CERTIFICATE-----

  cert: |
    -----BEGIN NEBULA CERTIFICATE-----
    Cn0KC3VzZXJfbGFwdG9wEgmW0IJWgP7//w8aE4DQglaA/v//D4CCoIUMgP7//w8o
    o8iWtwYwja6bxgY6INhK5mXzUYuS0OgMGV4UMROVksMybVn6sU+WGYGrVx4uSiDi
    ifLUXsi63mWBbEsjnqqWhHKLO1Wm250rNyXK6N8ASBJAybJKztrdoE9SVU3yWwL/
    iJ/OxFMFH5x/1+vpytMPB4xx3GofMl6ntQgKKS0rwYfoOdAMXGB2brbR31DX4qZw
    Dg==
    -----END NEBULA CERTIFICATE-----

  key: |
    ..........


static_host_map:
  "10.192.168.254": [........]


lighthouse:
  am_lighthouse: false
  hosts:
  - "10.192.168.254"

punchy:
  punch: true

tun:
  dev: nbl-remotenat
  unsafe_routes:
  - route: 192.168.1.0/24
    via: 10.192.168.254
    mtu: 1300
    metric: 100
    install: true

firewall:
  outbound:
    - port: any
      proto: any
      host: any

  inbound:
    - port: any
      proto: any
      host: any

• lighthouse

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    CjsKCXJlbW90ZW5hdCiOx5a3BjCOrpvGBjogmL/UzhpVsNDdOqz508vWIPcJDmN6
    vKdOWFmLHnxEksdAARJAMIHZylJUZW9tyDhmYZf41EcxXdGMB5bsZ61kXeCAvOLI
    i763mSQjq0Vxw423Y84H85waiLBmelMkmNTQU4p/CQ==
    -----END NEBULA CERTIFICATE-----

  cert: |
    -----BEGIN NEBULA CERTIFICATE-----
    Cn0KC2xpZ2h0aG91c2UwEgn+0YJWgP7//w8aE4DQglaA/v//D4CCoIUMgP7//w8o
    3seWtwYwja6bxgY6IHkTnh4VrW5CcL48WWmh55UZTu+vRxBpftVD5tqVKlljSiDi
    ifLUXsi63mWBbEsjnqqWhHKLO1Wm250rNyXK6N8ASBJAQHZS9YARHmS5yn6TJMlX
    GNg+YpJh7gsJuX2WuM+pcfDCaavDoM/1dvPc9HKuCzFk01YgY236ka+I7yWJquso
    Cw==
    -----END NEBULA CERTIFICATE-----

  key: |
    ..................

lighthouse:
  am_lighthouse: true

tun:
  dev: nbl-remotenat


listen:
  port: 4242

firewall:
  outbound:
    - port: any
      proto: any
      host: any

  inbound:
    - port: any
      proto: any
      host: any
    
    # added this entry after. restarted nebula on lighthouse, still no success
    - port: any
      proto: any
      host: any
      local_cidr: 192.168.1.0/24

EDIT: added another rule entry on lighthouse

[kuolemaaa]

Do you have any other VPNs or routing software installed? I'd be curious what the rest of your nft tables look like.

Stuff that Docker installed via iptables-nft

[kuolemaaa]

So, I think I placed a "breakpoint" on the nftable via meta nftrace set 1

table ip nebula_remotenat {
        chain postrouting {  [...]  }

        chain forward {
                type filter hook forward priority filter; policy accept;
                ct state established,related counter packets 0 bytes 0 accept
                iifname "nbl-remotenat" oifname "enp3s0" ip saddr 10.192.168.0/24 ip daddr 192.168.1.0/24 meta nftrace set 1 accept
        }
}

Here the output:

trace id 84a8fbed ip nebula_remotenat forward packet: iif "nbl-remotenat" oif "enp3s0" ip saddr 10.192.168.22 ip daddr 192.168.1.250 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 48948 ip length 84 icmp type echo-request icmp code 0 icmp id 58 icmp sequence 1 
trace id 84a8fbed ip nebula_remotenat forward rule iifname "nbl-remotenat" oifname "enp3s0" ip saddr 10.192.168.0/24 ip daddr 192.168.1.0/24 meta nftrace set 1 accept (verdict accept)
trace id 84a8fbed ip filter FORWARD packet: iif "nbl-remotenat" oif "enp3s0" ip saddr 10.192.168.22 ip daddr 192.168.1.250 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 48948 ip length 84 icmp type echo-request icmp code 0 icmp id 58 icmp sequence 1 
trace id 84a8fbed ip filter FORWARD rule counter packets 1509 bytes 883241 jump DOCKER-USER (verdict jump DOCKER-USER)
trace id 84a8fbed ip filter DOCKER-USER verdict return 
trace id 84a8fbed ip filter FORWARD rule counter packets 1509 bytes 883241 jump DOCKER-ISOLATION-STAGE-1 (verdict jump DOCKER-ISOLATION-STAGE-1)
trace id 84a8fbed ip filter DOCKER-ISOLATION-STAGE-1 verdict return 
trace id 84a8fbed ip filter FORWARD verdict continue    <---- i dont understand from what this 'continue' came from
trace id 84a8fbed ip filter FORWARD policy drop 

I think the MOTHERF***ING Docker rule drops my packets:

table ip filter{
         [...]
        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                iifname "br-c974e23b9945" oifname != "br-c974e23b9945" counter packets 480 bytes 743137 jump DOCKER-ISOLATION-STAGE-2
                iifname "br-493974da630d" oifname != "br-493974da630d" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                counter packets 1512 bytes 883493 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "docker0" counter packets 0 bytes 0 drop
                oifname "br-c974e23b9945" counter packets 0 bytes 0 drop
                oifname "br-493974da630d" counter packets 0 bytes 0 drop
                counter packets 480 bytes 743137 return
        }

        chain FORWARD { <<----------------- i think this is the one
                type filter hook forward priority filter; policy drop;
                counter packets 1512 bytes 883493 jump DOCKER-USER
                counter packets 1512 bytes 883493 jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" xt match "conntrack" counter packets 0 bytes 0 accept
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
                iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
                oifname "br-c974e23b9945" xt match "conntrack" counter packets 633 bytes 107992 accept
                oifname "br-c974e23b9945" counter packets 9 bytes 540 jump DOCKER
                iifname "br-c974e23b9945" oifname != "br-c974e23b9945" counter packets 480 bytes 743137 accept
                iifname "br-c974e23b9945" oifname "br-c974e23b9945" counter packets 0 bytes 0 accept
                oifname "br-493974da630d" xt match "conntrack" counter packets 0 bytes 0 accept
                oifname "br-493974da630d" counter packets 0 bytes 0 jump DOCKER
                iifname "br-493974da630d" oifname != "br-493974da630d" counter packets 0 bytes 0 accept
                iifname "br-493974da630d" oifname "br-493974da630d" counter packets 0 bytes 0 accept
        }

        chain DOCKER-USER {
                counter packets 1512 bytes 883493 return
        }
        [...]
}

[kuolemaaa]

I managed to reach the real local network.

Even though I use nft I needed to use the iptables version of the commands in the article.

That's because, I guess (I'm not an expert) Docker placed rules on the netfiler using iptables(-nft) creating tables managed by THAT userspace program. The guess comes from a command I found on the output of ntf list rulest

`ntf list rulest` output

# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain DOCKER {
                [....]
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr [......] oifname != "docker0" counter packets 0 bytes 0 xt target "MASQUERADE"
                [......]
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                xt match "addrtype" counter packets 1062 bytes 73285 jump DOCKER
        }

        chain OUTPUT {
                type nat hook output priority dstnat; policy accept;
                ip daddr != 127.0.0.0/8 xt match "addrtype" counter packets 2 bytes 120 jump DOCKER
        }
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain DOCKER {[......]}
        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                [......]
        }

        chain DOCKER-ISOLATION-STAGE-2 {[......]}

        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                [......]
        }
        chain DOCKER-USER {
                counter packets 2197 bytes 1328499 return
        }
}

Now the only things that are not clear to me are:

• how exactly works the fact that my laptop sends a packets for an unsafe_route trough the tunnel, how is it possible? how the operating system knows that a packet for 192.168.1.xxx needs to go through the nebula interface?
• What is the reason to use of the local_cidr option in the linux-nax/lighthouse nebula firewall? My guess is that the nebula interface on the 'linux-nat/lighthouse' sees an incoming packet from laptop's nebulaIP with destination 192.168.1.xxx and decides to drop it ?
• It seems that I don't need the same firewall.inbound rule on the laptop but only on the lighthouse. Cannot think about a proper reason. Does it being enforced on the laptop automatically by the tun.unsafe_route[] entry?

[johnmaguire]

how the operating system knows that a packet for 192.168.1.xxx needs to go through the nebula interface?

nebula installs these rules to your routing table when it starts up. you can use ip route to see them. note that the via is set to the router node.

What is the reason to use of the local_cidr option in the linux-nax/lighthouse nebula firewall? My guess is that the nebula interface on the 'linux-nat/lighthouse' sees an incoming packet from laptop's nebulaIP with destination 192.168.1.xxx and decides to drop it ?

yes, local_cidr exists so that, for example, you can give users access to to 192.168.1.1 while admins get access to192.168.1.0/24. or lock it down to specific ports and services (e.g. a CUPS printing server.)

It seems that I don't need the same firewall.inbound rule on the laptop but only on the lighthouse. Cannot think about a proper reason. Does it being enforced on the laptop automatically by the tun.unsafe_route[] entry?

are you making connections from your local network TO your laptop, through the lighthouse? if not, the inbound rules don't come into play. you're making outbound connections to the network, through the lighthouse.