error="Listener is IPv4, but writing to IPv6 remote"

[danthegoodman1]

I have a lighthouse on hetzner cloud (ipv4), another server on hetzner cloud in another region, and my laptop connected to the same service mesh. When running nebula, everything seems to connect fine.

Lighthouse logs of laptop join:

INFO[0424] Handshake message sent                        certName=laptop fingerprint=a80bd016ada2b197ce0bcec3bbe4e2013a79bc863e8244db581ef284552acbd1 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=1447448180 issuer=15830ae9f98361eafe2f03a09a0b4ec004d7436d30dedb6c9afee94aed5a0946 remoteIndex=0 responderIndex=808009430 udpAddr="107.193.54.107:4242" vpnIp=192.168.100.5

Other server join:

INFO[0435] Handshake message sent                        certName=mobile fingerprint=9df47d6b87e63f6968653a439910b75615d06eff87ce058ddaaaa91d010239d8 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=937558351 issuer=15830ae9f98361eafe2f03a09a0b4ec004d7436d30dedb6c9afee94aed5a0946 remoteIndex=0 responderIndex=2886553244 udpAddr="5.161.85.181:4242" vpnIp=192.168.100.9

(these ips are ephemeral for me testing, so I'm not worried about redacting)

Both hosts have the following config:

pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  key: /etc/nebula/host.key

static_host_map:
  "192.168.100.1": ["5.78.70.99:4242"]

lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - "192.168.100.1"

listen:
  host: 0.0.0.0
  port: 4242

punchy:
  punch: true

relay:
  am_relay: false
  use_relays: true

tun:
  disabled: false
  dev: nebula1
  drop_local_broadcast: false
  drop_multicast: false
  tx_queue: 500
  mtu: 1300
logging:
  level: info
  # json or text formats currently available. Default is text
  format: text
  # Disable timestamp logging. useful when output is redirected to logging system that already adds timestamps. Default is false
  #disable_timestamp: true
  # timestamp format is specified in Go time format, see:
  #     https://golang.org/pkg/time/#pkg-constants
  # default when `format: json`: "2006-01-02T15:04:05Z07:00" (RFC3339)
  # default when `format: text`:
  #     when TTY attached: seconds since beginning of execution
  #     otherwise: "2006-01-02T15:04:05Z07:00" (RFC3339)
  # As an example, to log as RFC3339 with millisecond precision, set to:
  #timestamp_format: "2006-01-02T15:04:05.000Z07:00"

#stats:
  #type: graphite
  #prefix: nebula
  #protocol: tcp
  #host: 127.0.0.1:9999
  #interval: 10s

  #type: prometheus
  #listen: 127.0.0.1:8080
  #path: /metrics
  #namespace: prometheusns
  #subsystem: nebula
  #interval: 10s

  # enables counter metrics for meta packets
  #   e.g.: `messages.tx.handshake`
  # NOTE: `message.{tx,rx}.recv_error` is always emitted
  #message_metrics: false

  # enables detailed counter metrics for lighthouse packets
  #   e.g.: `lighthouse.rx.HostQuery`
  #lighthouse_metrics: false

# Handshake Manager Settings
#handshakes:
  # Handshakes are sent to all known addresses at each interval with a linear backoff,
  # Wait try_interval after the 1st attempt, 2 * try_interval after the 2nd, etc, until the handshake is older than timeout
  # A 100ms interval with the default 10 retries will give a handshake 5.5 seconds to resolve before timing out
  #try_interval: 100ms
  #retries: 20

  # query_buffer is the size of the buffer channel for querying lighthouses
  #query_buffer: 64

  # trigger_buffer is the size of the buffer channel for quickly sending handshakes
  # after receiving the response for lighthouse queries
  #trigger_buffer: 64


# Nebula security group configuration
firewall:
  # Action to take when a packet is not allowed by the firewall rules.
  # Can be one of:
  #   `drop` (default): silently drop the packet.
  #   `reject`: send a reject reply.
  #     - For TCP, this will be a RST "Connection Reset" packet.
  #     - For other protocols, this will be an ICMP port unreachable packet.
  outbound_action: drop
  inbound_action: drop

  # Controls the default value for local_cidr. Default is true, will be deprecated after v1.9 and defaulted to false.
  # This setting only affects nebula hosts with subnets encoded in their certificate. A nebula host acting as an
  # unsafe router with `default_local_cidr_any: true` will expose their unsafe routes to every inbound rule regardless
  # of the actual destination for the packet. Setting this to false requires each inbound rule to contain a `local_cidr`
  # if the intention is to allow traffic to flow to an unsafe route.
  #default_local_cidr_any: false

  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m

  # The firewall is default deny. There is no way to write a deny rule.
  # Rules are comprised of a protocol, port, and one or more of host, group, or CIDR
  # Logical evaluation is roughly: port AND proto AND (ca_sha OR ca_name) AND (host OR group OR groups OR cidr) AND (local cidr)
  # - port: Takes `0` or `any` as any, a single number `80`, a range `200-901`, or `fragment` to match second and further fragments of fragmented packets (since there is no port available).
  #   code: same as port but makes more sense when talking about ICMP, TODO: this is not currently implemented in a way that works, use `any`
  #   proto: `any`, `tcp`, `udp`, or `icmp`
  #   host: `any` or a literal hostname, ie `test-host`
  #   group: `any` or a literal group name, ie `default-group`
  #   groups: Same as group but accepts a list of values. Multiple values are AND'd together and a certificate would have to contain all groups to pass
  #   cidr: a remote CIDR, `0.0.0.0/0` is any.
  #   local_cidr: a local CIDR, `0.0.0.0/0` is any. This could be used to filter destinations when using unsafe_routes.
  #      Default is `any` unless the certificate contains subnets and then the default is the ip issued in the certificate
  #      if `default_local_cidr_any` is false, otherwise its `any`.
  #   ca_name: An issuing CA name
  #   ca_sha: An issuing CA shasum

  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: icmp
      host: any

    - port: 8080
      proto: tcp
      groups:
        - laptop
        - mobile

Currently the laptop is laptop and the other server is mobile, created like:

./nebula-cert sign -name "lighthouse1" -ip "192.168.100.1/24"
./nebula-cert sign -name "laptop" -ip "192.168.100.5/24" -groups "laptop,ssh" # laptop
./nebula-cert sign -name "mobile" -ip "192.168.100.9/24" -groups "mobile,ssh # other server

When I try to curl http://192.168.100.5:8080 from the server, I get the following errors on the server nebula process:

ERRO[0125] Failed to send handshake message              error="Listener is IPv4, but writing to IPv6 remote" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2453346562 localIndex=2453346562 remoteIndex=0 udpAddr="[2600:1700:5892:9280::25]:4242" vpnIp=192.168.100.5
ERRO[0125] Failed to send handshake message              error="Listener is IPv4, but writing to IPv6 remote" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2453346562 localIndex=2453346562 remoteIndex=0 udpAddr="[2600:1700:5892:9280:68:e3c1:cf29:a766]:4242" vpnIp=192.168.100.5
ERRO[0125] Failed to send handshake message              error="Listener is IPv4, but writing to IPv6 remote" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2453346562 localIndex=2453346562 remoteIndex=0 udpAddr="[2600:1700:5892:9280:f8af:776b:204c:850b]:4242" vpnIp=192.168.100.5
INFO[0125] Handshake message sent                        handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2453346562 localIndex=2453346562 remoteIndex=0 udpAddrs="[107.193.54.107:4242 192.168.1.246:4242]" vpnIp=192.168.100.5
INFO[0125] Handshake message received                    certName=laptop durationNs=155439847 fingerprint=a80bd016ada2b197ce0bcec3bbe4e2013a79bc863e8244db581ef284552acbd1 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2453346562 issuer=15830ae9f98361eafe2f03a09a0b4ec004d7436d30dedb6c9afee94aed5a0946 remoteIndex=2453346562 responderIndex=1510725437 sentCachedPackets=1 udpAddr="107.193.54.107:4242" vpnIp=192.168.100.5

Both the servers and the laptop have both an ipv4 and ipv6. I tried adding:

static_map:
  network: ip4

I also tried changing the laptop host to '::' to test, but no luck there either.

to the laptop and server configs, but no change.

What am I doing wrong? Is there a way to force ipv4? As far as I can tell from join logs, it's connecting with ipv4, and I'm only advertising with ipv4

[nbrownus]

You aren't doing anything wrong it's just a really verbose error message that needs to be rethought.

Basically it looks like the host at 192.168.100.5 has 5 underlay network addresses, 2 ipv4 and 3 ipv6, and whenever nebula tries to send a handshake its whining really loudly about the v6 addresses. The end result is that you get a tunnel and the curl should return just fine, but you have annoying error logs.

If you want to enable ipv4 and ipv6 then listen.host: "::" should be all you need to do.

If you want to disable ipv6 then listen.host: 0.0.0.0 is the correct first step but you also want to block advertising ipv6 address or consuming ipv6 addresses from the lighthouse:

remote_allow_list:
  "0.0.0.0/0": true # dont try to handshake with anything other than ipv4
local_allow_list:
  "0.0.0.0/0": true # dont announce anything other than an ipv4 address to the lighthouses

Also, static_map.network: ip4 just controls which addresses are used from the result of a DNS lookup and I don't see any DNS names in your config.

[danthegoodman1]

Thanks for the swift response, to clarify my issue, the curl hangs indefinitely. With -vvv it’s just stuck on “trying (ip)…”

I’ll test changing those allow lists when I get back to my laptop!

[nbrownus]

I see, I was focused on the error message being the issue.

The logs you posted show a successful tunnel being formed so the issue appears to be the firewall config.

    - port: 8080
      proto: tcp
      groups:
        - laptop
        - mobile

this firewall rule will only allow a host with both laptop and mobile to pass. Based on the cert details you provided it looks like what you want is actually:

    - port: 8080
      proto: tcp
      groups:
        - laptop
        
    - port: 8080
      proto: tcp
      groups:
        - mobile

[danthegoodman1]

will try that, thanks!

[danthegoodman1]

worked great, it seems the connection is a bit slow to establish at first, but I'm guessing it's lazily UDP hole punching only when I make the request

[danthegoodman1]

FWIW for future readers, the docs are clear about this, I just didn't read them carefully enough 🫠

groups: Same as group but accepts a list of values. Multiple values are AND'd together and a certificate must contain all groups to pass.

[nbrownus]

Thank you for your detailed post!