Skip to content

Filter Anomaly thread

Jump to bottom
frikilax edited this page Sep 17, 2020 · 13 revisions

Anomaly thread

Description

It's a filter that receives connection data, stocks them in a redis. Every 5 minutes, it fetches redis' data and applies an algorithm on it, in order to detect anomalies.

How it work :

---data---> |anomaly filter| ---add---> (redis) <---query every 5min--- |anomaly thread|

If enough data when anomaly thread query redis:

(redis) <---trim used data--- |anomaly thread| ---write results---> |log file|

Filter Code

0x544D4C59

Dependencies

  • armadillo 9.400 or above, with:
    • lapack (preferably lapacke)
    • blas (preferably openblas)
  • mlpack 3.0.1 or above

Darwin configuration

Example of darwin configuration for this filter :

{
    "tanomaly_1": {
        "exec_path": "/path/to/darwin/build/darwin_anomaly",
        "config_file":"/path/to/darwin/conf/ftanomaly/ftanomaly.conf",
        "output": "",
        "next_filter": "",
        "nb_thread": 1,
        "log_level": "DEBUG",
        "cache_size": 0
    }
}

Config file

  • redis_socket_path (optional) : the Redis in which the filter will store the data
  • alert_redis_list_name (optional) : the key under which the filter will store the raised alerts in Redis
  • alert_redis_channel_name (optional) : the Redis channel in which the raised alerts will be publish
  • log_file_path (optional) : the path in which the filter will write it's results
  • start_detection_thread (optional) : force the start of the detection thread. By default, the thread is started if the local Redis given in config is a master thread. This can be useful when using Darwin on several nodes in a cluster, with a master Redis.
  • detection_frequency (optional) [default 300] : set the delay between two detection triggers by the detection thread, default is 300s (5min)

Example :

{
    "redis_socket_path": "/var/sockets/redis/redis.sock",
    "alert_redis_list_name": "darwin_alerts",
    "alert_redis_channel_name": "darwin.alerts",
    "log_file_path": "/var/log/darwin/alerts.log"
}

Example (legacy) :

{
    "redis_socket_path": "/var/sockets/redis/redis.sock",
    "redis_list_name": "anomalyFilter",
    "log_file_path": "/var/log/darwin/alerts.log"
}

Body

[ 
      ["<net_src>","<net_dst>","<net_dst_port>","<ip_proto>"], 
      [...] 
]
  • Where ip_proto need to represent either udp (17), tcp (6), icmp(1). If it's icmp, net_dst_port needs a value, but it won't be taken into account.

Example

If you just want to send some data to the filter :

[
    ["73.90.76.52","99.184.81.66","1017","17"],
    ["250.230.92.234","54.220.65.198","2922","6"],
    ["171.104.231.132","0.127.226.192","467","17"],
    ["42.214.30.108","246.163.54.146","2979","1"],
    ["79.187.169.202","46.126.241.248","2677","1"],
    ["57.126.101.247","255.171.17.199","2468","17"],
    ["102.27.128.38","75.125.227.149","2249","1"],
    ["116.145.214.73","182.20.121.254","1687","1"],
    ["248.78.140.91","112.67.123.34","1119","1"],
    ["47.159.155.135","117.9.1.88","1740","6"],
    ["114.20.51.158","39.139.219.223","2435","1"],
    ["228.206.220.156","136.226.212.47","400","17"],
    ["233.18.134.13","66.32.21.214","684","1"],
    ["77.81.84.70","46.192.216.12","1347","17"],
    ["116.254.19.59","31.113.218.152","2498","6"],
    ["209.157.143.89","48.78.238.19","659","6"],
    ["64.183.225.150","111.166.166.114","526","6"],
    ["128.72.180.29","253.81.68.206","1195","17"],
    ["143.92.16.229","142.228.238.233","705","17"]
]

Results

Always sends a 0 certitude when the line is correct, 101 if not. A thread will asynchronously fetch all the data aggregated and execute the detection, so alerts will be generated periodically (if present). Example of log:

{ 
    "time": "<ISO8601>",
    "anomaly": {
        "ip": "10.5.22.144",
        "udp_nb_host": 16.000000,
        "udp_nb_port": 2.000000,
        "tcp_nb_host": 520.000000,
        "tcp_nb_port": 4.000000,
        "icmp_nb_host": 0.000000,
        "distance": 624.351380
    }
}

Alert Details

"details": {
    "udp_nb_host": <float, number of unique host connected via udp>,
    "udp_nb_port": <float, number of unique port connected via udp>,
    "tcp_nb_host": <float, number of unique host connected via tcp>,
    "tcp_nb_port": <float, number of unique port connected via tcp>,
    "distance": <float, distance to the closest normal asset>
}

Configuration with Rsyslog

Modules necessary

To work alongside this filter, Rsyslog must be used with those compiled modules:

  • impcap
  • mmdarwin

Impcap

The Impcap module is a network sniffing module, it will recover information and data from packets on the network. Please go to its corresponding wiki to get full documentation on how to configure it.

Mmdarwin

The Mmdarwin module comes after Impcap, and handles the transmission of data to darwin, the configuration of Mmdarwin in the rsyslog configuration file should be:

action(type="mmdarwin" socketpath="/path/to/darwin/sockets/tanomaly_1.sock" fields=["!impcap!net_src_ip", "!impcap!net_dst_ip", "!ipmcap!net_dst_port", "!impcap!IP_proto"] key="TANOMALY" filtercode="0x544D4C59")

Clone this wiki locally