Match content on normalized fields

[jonschipp]

I have an idea for a useful feature. I think being able to match content, pcre, etc. on specific fields that are normalized would be really useful. In variable length logs such as Bro logs snort like modifiers such as within, depth, and offset are too specific. Maybe you can use a numerical index to accomplish this by counting the number of normalize fields in a message based on the rulebase.

#Field:             1                2                3                      4
rule=: Accepted %-:word% for %username:word% from %src-ip:ipv4% port %src-port:number% ssh2

then you could do something like content: "jschipp"; field: 2; in a rule where content jschipp would only match if it's in the 2nd field which is the username field. Also, utilizing the string identifiers from a rulebase file as the field names would be dope e.g. username: "jschipp";. I suspect this data can be easily pulled from liblognorm.

[CyberTaoFlow]

bump

[CyberTaoFlow]

I see in 1.0.1 there is support for additional lognorm fieldnames (url,hash) presumably to support bluedot. Not sure if bro intel values are also checked against those fieldnames..

[beave]

Looking at this for version 1.1.2

[jonschipp]

Awesome, I think this would allow a lot of flexibility for all kinds of different logs. It would be similar to writing a decoder in OSSEC but with far more flexibility since they have a limited number of fields to decode:

location - where the log came from (only on FTS)
srcuser - extracts the source username
dstuser - extracts the destination (target) username
user - an alias to dstuser (only one of the two can be used)
srcip - source ip
dstip - dst ip
srcport - source port
dstport - destination port
protocol - protocol
id - event id
url - url of the event
action - event action (deny, drop, accept, etc)
status - event status (success, failure, etc)
extra_data - Any extra data

It's quite painful if you want to match something in a log that doesn't have a descriptive name that fits with the list above. For example, I often use url to search for a file path in sudo logs where sudo: USER=root COMMAND=<url> when there really should be another field called filepath or something. Being able to label them and have as many fields as the user defines with liblognorm would be quite exceptional I believe in a log analysis tool

[beave]

Interesting idea. Let me chat with our team. The one nice thing is that liblognorm doesn't define "fields". you as the user get to do that. There is no limitations there. Let me wrap my head around it and see what we can come up with. We are likely doing a 1.1.1 release this week, so it will obviously be after that release :)

[beave]

Thinking about this a little more.

The "field" idea in your example (content: "jschipp"; field: 2;) wouldn't work. Liblognorm doesn't return a field number. What if we did something like:

normalize_content: "jschipp", username;

or

normalize_content: "10.7.5.111", src_ip;

While we don't have the field number, we could extract and the inquire if the extracted field is what we want? Let me know your thoughts.

[beave]

I'm thinking the last field would be a bit dynamic. For example, if you liblognorm rulebase is:

rule=: Accepted %-:word% for %funkyjoe:word% from %src-ip:ipv4% port %src-port:number% ssh2

you could do:

normaiize_content: "frank", funkyjoe; normalize_nocase;

See what I mean?

[beave]

???

[beave]

What do you think about this concept?

[jonschipp]

Hey Champ, sorry for not responding. I kept forgetting to. Perfect, I like it and it's better than my original suggestion of counting fields. I agree, the user defined fields makes liblognorm really nice. This will make Sagan more flexible than other similar tools in my opinion. Thanks for looking at this

[lometur]

I see this is a few years old, but has there been headway on this at all? We see value for this feature the way that @beave suggested.

[nosmircss]

track by normalized field would be amazing too.