s01-parse/crowdsecurity /dovecot-logs.yaml - Not matching brute force log entries

[randomcop]

Describe the bug
dovecot collection does not catch any brut-force IPs inside the dovecot.log

To Reproduce
Steps to reproduce the behavior

cscli collections install crowdsecurity/dovecot
cscli scenarios install crowdsecurity/dovecot-spam
cscli-explain-out.txt

Expected behavior
A clear and concise description of what you expected to happen.

match IP addresses inside the dovecot.log bruteforcing the imap and pop3 service

Screenshots
If applicable, add screenshots to help explain your problem.

crowdsecurity/dovecot ✔️ enabled 0.1 /usr/local/etc/crowdsec/collections/dovecot.yaml
crowdsecurity/freebsd ✔️ enabled 0.3 /usr/local/etc/crowdsec/collections/freebsd.yaml
crowdsecurity/sshd ✔️ enabled 0.5 /usr/local/etc/crowdsec/collections/sshd.yaml

Additional context
cscli-explain output attached
log-sample attached
https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/dovecot-logs.yaml
cscli-explain-out.txt
dovecot-logs-sample.txt

FreeBSD 14.1-RELEASE-p5 FreeBSD 14.1-RELEASE-p5 GENERIC amd64
crowdsec-1.6.3_2 CrowdSec lightweight and collaborative security engine
crowdsec-firewall-bouncer-0.0.29 CrowdSec bouncer written in golang for firewalls

[LaurenceJJones]

Hey 👋🏻

Thank you for opening a detailed issue!

So currently dovecot parser does not support the LDAP authentication logs, however, it does support the "<protocol>-login" lines EG:

line: Oct 14 00:01:02 pop dovecot[1559]: pop3-login: Disconnected: Connection closed (auth failed, 4 attempts in 31 secs): user=<[email protected]>, method=PLAIN, rip=115.66.228.194, lip=91.204.6.100, TLS: Connection closed, session=<tWlI2WIkhLlzQuTC>
        ├ s00-raw
        |       └ 🟢 crowdsecurity/syslog-logs (+12 ~9)
        ├ s01-parse
        |       └ 🟢 crowdsecurity/dovecot-logs (+8 ~1)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |       ├ 🔴 crowdsecurity/http-logs
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                └ 🟢 crowdsecurity/dovecot-spam

So these would be caught but I guess over the same connection there can be multiple calls the LDAP? (I dont use LDAP so I dont know how it works) cause I see from the IP address multiple calls to LDAP.

Let me know if this is the case and we cannot just have this log line since there can be multiple authentication calls to LDAP in a single connection window.

[randomcop]

Hi. Thank you for picking this up so fast.

So these would be caught but I guess over the same connection there can be multiple calls the LDAP? (I dont use LDAP so I dont know how it works) cause I see from the IP address multiple calls to LDAP.

Yes. It seems to be the case that several auth requests are aggregated inside one call to LDAP.
I looked through the dovecot documentation and found an option to change this behaviour, by letting separately spawned auth worker processes connect individually to ldap.

dovecot/conf.d/10-logging.conf
; Log unsuccessful authentication attempts and the reasons why they failed.
auth_verbose = yes

dovecot/dovecot-ldap.conf.ext
; By default all LDAP lookups are performed by the auth master process.
; If blocking=yes, auth worker processes are used to perform the lookups.
; Each auth worker process creates its own LDAP connection so this can
; increase parallelism. With blocking=no the auth master process can
; keep 8 requests pipelined for the LDAP connection, while with blocking=yes
; each connection has a maximum of 1 request running. For small systems the
; blocking=no is sufficient and uses less resources.
; blocking = no
blocking = yes

This changes the logging more in a direction suitable for counting "unknown user" auth attempts (I see around 20-50k "unknown user" dovecot login attemps (almost all brute force) per day on my mailserver. Fail2ban is only a small remedy, because the people behind those huge brute-force IP swarms take great care to not have one single IP not do too many auth attemps inside a time frame of several hours, to not trigger a block.

So now inside the dovecot.log there are many log lines from dovecot's auth-worker (sample attached)
dovecot-log-bruteforce-auth-worker.txt

Unfortunately those lines are nor grok'ed by s01-parse/crowdsecurity/dovecot-logs.yaml (explain output attached)
cscli-explain-dovecot-log-bruteforce-auth-worker.txt

Problem seems to be with the following two grok regex blocks:

• grok:
  pattern: "auth-worker\(%{INT}\): %{WORD:dovecot_user_backend}\(%{DATA:dovecot_user},%{IP:dovecot_remote_ip},?%{DATA}\): (%{DATA}: )?%{DATA:dovecot_login_message}$"
  apply_on: message

• grok:
  pattern: "auth-worker\(%{INT}\): (Info: )?conn unix:auth-worker \(pid=%{INT},uid=%{INT}\): auth-worker<%{INT}>: %{WORD:dovecot_user_backend}\(%{DATA:dovecot_user},%{IP:dovecot_remote_ip},?%{DATA}\): (%{DATA}: )?%{DATA:dovecot_login_message}$"
  apply_on: message

I have tried to make those pattern match, but failed. Also there seem to be no tools available to help with constructing a matching crowdsec pattern for a given log sample. I also tried my luck with a tool like https://grokdebugger.com/ but crowdsec patterns are unknown to this tool.

[randomcop]

I think I finally succeeded in coming up with a working grok pattern for dovecot-ldap using the auth-worker option for connections to a LDAP backend..

The following pattern works for me, detecting brute force "authentication failed" attempts.

• grok:
  pattern: "auth-worker\(%{INT}\): (Info: )? ?conn unix:auth-worker \(uid=%{INT}\): auth-worker<%{INT}>: %{WORD:dovecot_user_backend}\(%{DATA:dovecot_user},%{IP:dovecot_remote_ip},?%{DATA}\): (%{DATA}: )?%{DATA:dovecot_login_message}$"
  apply_on: message

(Hinz: If testing out crowdsec grok patterns on grokdebugger.com you have to remove one backquote from double backquotes)

I have attached a diff file. Can this be commited to dovecot-logs.yaml ?

dovecot-logs.yaml.diff.txt

[LaurenceJJones]

I think I finally succeeded in coming up with a working grok pattern for dovecot-ldap using the auth-worker option for connections to a LDAP backend..

The following pattern works for me, detecting brute force "authentication failed" attempts.

* grok:
  pattern: "auth-worker\(%{INT}\): (Info: )? ?conn unix:auth-worker \(uid=%{INT}\): auth-worker<%{INT}>: %{WORD:dovecot_user_backend}\(%{DATA:dovecot_user},%{IP:dovecot_remote_ip},?%{DATA}\): (%{DATA}: )?%{DATA:dovecot_login_message}$"
  apply_on: message

(Hinz: If testing out crowdsec grok patterns on grokdebugger.com you have to remove one backquote from double backquotes)

I have attached a diff file. Can this be commited to dovecot-logs.yaml ?

dovecot-logs.yaml.diff.txt

For "efficiency" within dovecot I suppose the best way would be to revert your dovecot change and parse each LDAP request OR parse out the number in the log that states "failed auth 4 times" and create a new bucket that can count those as we would need a conditional bucket.

Since it seems dovecot is having a bunch of auth attempts I think changing dovecot to spawn a worker per auth request is not bad, but it not optimized. I have this issue in my todos, just not at the top. btw, this is not to take away from your epic work, I just think that having this "only works if dovecot is configured like this" would cause somebody to come back and say it doesnt work until they configure this way.