Understanding Log Storage and Alerts in Security Onion Distributed Setup

[SankaGamage]

Hi everyone,

I have set up a distributed Security Onion deployment with search, forward, receiver, and manager nodes, all connected to the manager node. I have a few questions regarding how logs are handled in this architecture.

1. Log Storage & Retention:

• When using this setup, where are the logs permanently stored?
• Can I change the storage location separately for hot and cold logs?

2. Alerts Not Appearing:

• I can’t see any alerts in the Alerts section. Do I need to do any additional configuration to get alerts properly?

3.Elastic Agent Logs Visibility:

• I added my PC to Security Onion using the Elastic Agent. While I can see logs in Dashboards and Kibana, I am unsure if they are reaching the right places.
• Are there specific steps to ensure logs are fully integrated into the system?

I would appreciate any guidance on these issues. Thanks in advance!

[dougburks]

When using this setup, where are the logs permanently stored?

Logs are stored on search nodes:
https://docs.securityonion.net/en/2.4/architecture.html#search-node

Can I change the storage location separately for hot and cold logs?

Yes, take a look at the ILM documentation:
https://docs.securityonion.net/en/2.4/elasticsearch.html#index-management
https://docs.securityonion.net/en/2.4/elasticsearch.html#ilm
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html

I can’t see any alerts in the Alerts section. Do I need to do any additional configuration to get alerts properly?

If you have a forward node that is consuming network traffic from a tap or span port, then you can go through the Troubleshooting Alerts section of the documentation:
https://docs.securityonion.net/en/2.4/suricata.html#troubleshooting-alerts

Are there specific steps to ensure logs are fully integrated into the system?

Go to SOC Dashboards, click the query dropdown, select the Elastic Agent Overview dashboard, and check to see if you have logs from your PC's Elastic Agent.

For more information, please see:
https://docs.securityonion.net/en/2.4/elastic-agent.html

[dougburks]

Logs are stored on search nodes – This is clear, but can you provide a specific path where the logs are saved? Alternatively, is there a specific method to find the exact log storage location?

Logs are stored on search nodes in Elasticsearch:
https://docs.securityonion.net/en/2.4/elasticsearch.html

If I use a VPN to collect logs (where endpoints connect as part of the same network), I notice that disconnecting the VPN causes the agent connection to be lost. If an issue occurs during this disconnected period, will logs or alerts for that time be available once the VPN reconnects, or are they lost?

Another way to approach this is to place a Fleet node on the DMZ so that agents can always be connected to the Fleet node even when they are not on the VPN:
https://docs.securityonion.net/en/2.4/architecture.html#elastic-fleet-standalone-node

[SankaGamage]

I've also had issues with testing alerts to confirm that the Elastic Agent is working. What I ended up doing was going into the Detections section and Enabling all the Sigma rules. They're almost all Disabled by default. Once those are Enabled and you wait 15 minutes to be sure things are synced, go to an Endpoint that has the Elastic Agent running and perform some security related action that should trigger an alert. I use Pentesting Malware tests for my testing. You can do something simple though like running a Powershell command as the System user, which should trigger an Alert. You could also write your own Sigma Detections that you can then trigger for your testing.

I enabled all the rules, and now I see an 'ElastAlert: Rule mismatch' issue. Can anyone guide me on how to fix this?

[dougburks]

I enabled all the rules, and now I see an 'ElastAlert: Rule mismatch' issue. Can anyone guide me on how to fix this?

From https://docs.securityonion.net/en/2.4/detections.html#rule-engine-status:
Rule Mismatch: An integrity check process detected a mismatch between the deployed rules and the enabled rules. The SOC log will note the specific mismatched rules.

[SankaGamage]

I enabled all the rules, and now I see an 'ElastAlert: Rule mismatch' issue. Can anyone guide me on how to fix this?

From https://docs.securityonion.net/en/2.4/detections.html#rule-engine-status: Rule Mismatch: An integrity check process detected a mismatch between the deployed rules and the enabled rules. The SOC log will note the specific mismatched rules.

I can see like this. THen What do I need to do next?

[dougburks]

Have you tried changing the time range in the upper right to see if you get any results?

Did you enable Sigma rule 24e3e58a-646b-4b50-adef-02ef935b9fc8 - Hacktool Execution - Imphash? If so, disable it and see if that helps.

You don't mention what version you're on. If you're not on the latest version (2.4.120), please try updating:
https://blog.securityonion.net/2025/02/security-onion-24120-now-available.html

If you continue to have issues, please start a new discussion with an appropriate title since we are straying from the subject of this discussion.