Supported multi network deployment for distributed install

[cbspidi]

Version

2.4.111

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

32

Storage for /

250

Storage for /nsm

500

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi Guys,
Are you able to confirm if my setup is a valid deployment scenario?

I'm in a network setup where a manager node and several search nodes (1, 2, and 3) are configured with dual connections.

The manager node has one connection to the internet (with a public IP) and another connection to an isolated LAN. The search nodes also have similar dual connections, with one link providing internet access and the other facilitating inter-node communication.
The primary issue observed is the failure of the installer to download the necessary certificate and repository files from the manager node, leading to a timeout and failure to join the manager.

Pings and DNS works across all the nodes in the setup.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cbspidi]

Setup diagram:

[cbspidi]

Hi Chris,
The command output is below; comment on my side on your point about the bond interface.

In my opinion, creating a bond will not change much as it seems the manager node picks an interface where a default route is attached; creating a bond will not change the entries in the routing table.

If what I say is true the only way to get this to work would be to join all the nodes into an overlay network like zerotier and pick mgmt interface to be the one facing the overlay.

The issue with my approach would be the complexity of the setup and the additional cost of overlay and routing in/out of private and public traffic of the overlay.

sec-onion-m1-fqdn_manager:
    ens3
sec-onion-s1_searchnode:
    ens3
sec-onion-s2_searchnode:
    ens3
sec-onion-s3_searchnode:
    ens3

local:
    ----------
    sec-onion-m1-fqdn_manager:
        - 117.225.198.148
    sec-onion-s1_searchnode:
        - 172.16.16.217
    sec-onion-s2_searchnode:
        - 172.16.16.250
    sec-onion-s3_searchnode:
        - 172.16.16.152

sec-onion-m1-fqdn_manager:
    ----------
    docker0:
        - 172.17.0.1
    ens3:
        - 117.225.198.148
        - fe80::f816:3eff:fe84:ea63
    ens4:
        - 172.16.16.208
        - fe80::f8b3:78b4:baf6:927e
    lo:
        - 127.0.0.1
    sobridge:
        - 192.168.255.1
    veth0ab8e2c:
    veth2249adc:
    veth2651d1d:
    veth303d80e:
    veth326e1a9:
    veth4b34528:
    veth97f6ab3:
    veth9a7cb61:
    vethd593300:
    vethe4fe397:
    vethfe0abfe:
    zt2lruwhx6:
        - 10.147.17.10
sec-onion-s2_searchnode:
    ----------
    docker0:
        - 172.17.0.1
    ens3:
        - 172.16.16.250
        - fe80::f816:3eff:fee4:2c28
    lo:
        - 127.0.0.1
    sobridge:
        - 192.168.255.1
    vethb59e3e3:
    vethd414e8e:
sec-onion-s1_searchnode:
    ----------
    docker0:
        - 172.17.0.1
    ens3:
        - 172.16.16.217
        - fe80::f816:3eff:feab:b8c6
    lo:
        - 127.0.0.1
    sobridge:
        - 192.168.255.1
    veth7627691:
    vethb03f0b5:
sec-onion-s3_searchnode:
    ----------
    docker0:
        - 172.17.0.1
    ens3:
        - 172.16.16.152
        - fe80::f816:3eff:fee6:a16a
    lo:
        - 127.0.0.1
    sobridge:
        - 192.168.255.1
    veth36ca620:
    vethabafa0c:

[cm-ops]

In the past I have seen zerotier used once. From what I observed there were mixed results.

[cbspidi]

Hey Chris,

Maybe, the answer to the questions is:
Security Onion does not support multi-interface/multi-network deployments.

[cm-ops]

From the docs:

You’ll need at least one wired network interface dedicated to management (preferably connected to a dedicated management network). We recommend using static IP addresses where possible.

https://docs.securityonion.net/en/2.4/hardware.html#nic

[cbspidi]

Maybe, ... an update is needed in the documentation to mention that multiple networks are not supported in the distributed type of deployments.
In communication between the manager and other nodes, the manager will try to use the egress interface where the default gateway is assigned.