crowdsec-firewall-bouncer.service v0.0.23-debian-pragmatic : hang netlink server when starting

[lelanta]

Hello,
Servers are proxmox v7.2 with severals lxc containers as web and mail servers on debian v11.
On containers crowdsec seems to run well.
I also test your wordpress plugins as we mainly use wordpress and joomla.

Your solution seems to be really a big progress in battle against virtual criminals but...

We use only nftables for all servers with specifics tables, chains and rules and not the integrated iptables firewall from proxmox.

PROBLEM IS :

I installed crowdsec with crowdsec-firewall-bouncer on a proxmox host : no problems.
I change configuration of the bouncer to work with our netfilter rules regarding your last release and instructions.
Big problem : netlink of the server become down and we have only access to it with kvm and not ssh.
As soon as i stop the bouncer from kvm, netlink of server become again on line.
I attach to this couriel logs and configuration files for expertise.
Now i purged the bouncer and reinstall it with default parameters : it's not hang and service run.
In waiting,
Best regards.

###########################
I/ : crowdsec config file :
###########################
cscli config show
Global:

• Configuration Folder : /etc/crowdsec
• Data Folder : /var/lib/crowdsec/data
• Hub Folder : /etc/crowdsec/hub
• Simulation File : /etc/crowdsec/simulation.yaml
• Log Folder : /var/log/
• Log level : info
• Log Media : file
  Crowdsec:
• Acquisition File : /etc/crowdsec/acquis.yaml
• Parsers routines : 1
  cscli:
• Output : human
• Hub Branch :
• Hub Folder : /etc/crowdsec/hub
  Local API Server:
• Listen URL : 127.0.0.1:8080
• Profile File : /etc/crowdsec/profiles.yaml
• Trusted IPs:
  • 127.0.0.1
  • ::1
• Database:
  • Type : sqlite
  • Path : /var/lib/crowdsec/data/crowdsec.db
  • Flush age : 7d
  • Flush size : 5000

###########################
II/ : bouncer config file :
###########################
cat crowdsec-firewall-bouncer.yaml
mode: nftables
pid_dir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
log_compression: true
log_max_size: 100
log_max_backups: 3
log_max_age: 30
api_url: http://127.0.0.1:8080/
api_key: 30eba9a169bdd4e854db8e50f3840162
insecure_skip_verify: false
disable_ipv6: true
deny_action: DROP
deny_log: false
supported_decisions_types:

• ban
  #to change log prefix
  #deny_log_prefix: "CS: "
  #to change the blacklists name
  blacklists_ipv4: ip_banned
  blacklists_ipv6: crowdsec6-blacklists
  #if present, insert rule in those chains
  iptables_chains:
• INPUT

- FORWARD

- DOCKER-USER

nftables

nftables:
ipv4:
enabled: true
set-only: true
table: blacklist
chain: input
ipv6:
enabled: false
set-only: false
table: crowdsec6
chain: crowdsec6-chain

packet filter

pf:

an empty string disables the anchor

anchor_name: ""

###########################
III/ : crowdsec bouncer log
###########################
time="16-05-2022 17:14:59" level=info msg="backend type : nftables"
time="16-05-2022 17:14:59" level=info msg="IPV6 is disabled"
time="16-05-2022 17:14:59" level=info msg="nftables initiated"
time="16-05-2022 17:14:59" level=info msg="Processing new and deleted decisions . . ."
time="16-05-2022 17:14:59" level=info msg="416 decisions deleted"
time="16-05-2022 17:14:59" level=error msg="unable to commit add decisions Receive: netlink receive: invalid argument"

NETLINK RECEIVE PROBLEM. - CONSOLE SERVER AND SSH CONNECTION OUT

time="16-05-2022 17:14:59" level=info msg="14313 decisions added"
time="16-05-2022 17:15:39" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="16-05-2022 17:15:39" level=error msg="Get ["http://127.0.0.1:8080/v1/decisions/stream?startup=false\"](http://127.0.0.1:8080/v1/decisions/stream?startup=false\): dial tcp 127.0.0.1:8080: i/o timeout"
time="16-05-2022 17:16:09" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="16-05-2022 17:20:18" level=info msg="flushing 'ip_banned' set in 'blacklist' table"
time="16-05-2022 17:20:18" level=info msg="Shutting down firewall-bouncer service"

###########################
III/ : crowdsec.log
###########################
time="16-05-2022 18:16:33" level=info msg="Crowdsec v1.3.4-debian-pragmatic-linux-ddfe95e45d98d1e7a6496d2499e2e44a023135be"
time="16-05-2022 18:16:33" level=info msg="Loading prometheus collectors"
time="16-05-2022 18:16:33" level=info msg="Loading CAPI pusher"
time="16-05-2022 18:16:33" level=info msg="Loading grok library /etc/crowdsec/patterns"
time="16-05-2022 18:16:33" level=info msg="Loading enrich plugins"
time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'IpToRange'"
time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'reverse_dns'"
time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'ParseDate'"
time="16-05-2022 18:16:33" level=info msg="Loading parsers 6 stages"
time="16-05-2022 18:16:33" level=info msg="Loaded 2 parser nodes" file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/proxmox-logs.yaml
time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
time="16-05-2022 18:16:33" level=info msg="Loaded 7 nodes, 3 stages"
time="16-05-2022 18:16:33" level=info msg="Loading postoverflow Parsers"
time="16-05-2022 18:16:33" level=info msg="Loaded 0 nodes, 0 stages"
time="16-05-2022 18:16:33" level=info msg="Loading 26 scenario files"
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=wispy-frog file=/etc/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml name=crowdsecurity/vmware-vcenter-vmsa-2021-0027
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=quiet-violet file=/etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml name=crowdsecurity/apache_log4j2_cve-2021-44228
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=billowing-sound file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=small-shape file=/etc/crowdsec/scenarios/jira_cve-2021-26086.yaml name=crowdsecurity/jira_cve-2021-26086
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=red-rain file=/etc/crowdsec/scenarios/grafana-cve-2021-43798.yaml name=crowdsecurity/grafana-cve-2021-43798
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=cold-haze file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=hidden-waterfall file=/etc/crowdsec/scenarios/fortinet-cve-2018-13379.yaml name=crowdsecurity/fortinet-cve-2018-13379
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=young-night file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=winter-wave file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=LePresidente/http-generic-401-bf
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=rough-meadow file=/etc/crowdsec/scenarios/http-probing.yaml name=crowdsecurity/http-probing
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=dawn-meadow file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=blue-sky file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=purple-morning file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=shy-silence file=/etc/crowdsec/scenarios/http-path-traversal-probing.yaml name=crowdsecurity/http-path-traversal-probing
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=dawn-meadow file=/etc/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml name=crowdsecurity/pulse-secure-sslvpn-cve-2019-11510
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=floral-wood file=/etc/crowdsec/scenarios/proxmox-bf.yaml name=fulljackz/proxmox-bf
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=ancient-flower file=/etc/crowdsec/scenarios/proxmox-bf.yaml name=fulljackz/proxmox-bf-user-enum
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=red-rain file=/etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml name=crowdsecurity/f5-big-ip-cve-2020-5902
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=morning-fire file=/etc/crowdsec/scenarios/http-cve-2021-42013.yaml name=crowdsecurity/http-cve-2021-42013
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=white-dawn file=/etc/crowdsec/scenarios/vmware-cve-2022-22954.yaml name=crowdsecurity/vmware-cve-2022-22954
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=crimson-feather file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=still-field file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=small-silence file=/etc/crowdsec/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=floral-voice file=/etc/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml name=crowdsecurity/thinkphp-cve-2018-20062
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=silent-rain file=/etc/crowdsec/scenarios/http-open-proxy.yaml name=crowdsecurity/http-open-proxy
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=late-cherry file=/etc/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml name=crowdsecurity/spring4shell_cve-2022-22965
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=delicate-snow file=/etc/crowdsec/scenarios/http-cve-2021-41773.yaml name=crowdsecurity/http-cve-2021-41773
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=aged-violet file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=still-waterfall file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=divine-shape file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t
time="16-05-2022 18:16:33" level=warning msg="Loaded 30 scenarios"
time="16-05-2022 18:16:33" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="16-05-2022 18:16:33" level=warning msg="No matching files for pattern /var/log/nginx/.log" type=file
time="16-05-2022 18:16:33" level=warning msg="No matching files for pattern ./tests/nginx/nginx.log" type=file
time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/auth.log to datasources" type=file
time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/syslog to datasources" type=file
time="16-05-2022 18:16:33" level=warning msg="No matching files for pattern /var/log/apache2/.log" type=file
time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/syslog to datasources" type=file
time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/kern.log to datasources" type=file
time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/messages to datasources" type=file
time="16-05-2022 18:16:33" level=info msg="test done"

END of LOGS

[buixor]

Hello,

Sorry for the delay.
To be sure to understand your issue :

• you had a previous version of the bouncer, but after changing its configuration, it was blocking traffic ?
• you then purged and reinstall bouncer and it worked again ?

Please :

• Join your configuration files and log files using markdown code blocks otherwise it's impossible to read because of formatting issues.
• Can you tell me more about how it the firewall configuration without the bouncer ? It might be a rule priority issue

[lelanta]

Hello,
For firewall that cannot be a rule priority issue.
nftables manage with several tables and chain the host and all VMs.
host with filter table chain input and VMs with filter table chain forward.

The advantage of this configuration is your work only with a centralized firewall on host for all.
severals tables and chains have specifics rules (ipv6 disabled on all hosts)
tables: filter, table ip filter, ip blacklist, ip ip_france, ip fail2ban, ip crowd, etc...
chains: host.rules, vms.rules, geo-block.rules, blacklisted.rules, etc...

This structure is made to not flush or fully delete tables and rules with a nft flush ruleset but only specific table and rules when restart a specific service like fail2ban or crowdsec.

We use proxmox since v3 on all servers since 9 years with previously iptables firewall.
since proxmox v6 we use nftables which is a real advance in firewall managment and let us have only half of lines for a better service with advantage of netdev use.

To solve problem, i reinstall only bouncer with standart parameters and all is ok.
problem come if you change configuration and particulary for parameter set-only defined to true.
I tried to only change name of table and chain and it's ok

for me bouncer entered in a infinite loop ?? because possibility to unloading it with KVM.

I will do some other tests on a local testing proxmox host ASAP.

Now i saw that jarppiko tried to make some modifications on bouncer concerning the use of non standart configuration. i think his modifications have not been merged on last release and is for me incomplete to use it on a proxmox or similar system.

I began to fork bouncer code to analyse structure and code.
I think it's necessary to fully implement netfilter api inside to have fully possible configuration.
possibility to use severals different type, hooks, priority, policy, etc... at same time to control host and VMs
As soon as i'll have finish to analyse and well understand philosophy of code for crowdsec and bouncer. i'll submit to the team a proposal of modifications and if OK, i'll begin to code a new bouncer as soon as finished to explore capabilities of golang. (i developped netware NLMs systems in C during 10 years.)

On a other side , what is this error ?
level=error msg="unable to commit add decisions Receive: netlink receive: invalid argument"
i think that is when a local decision occur and is not apply because only CAPIs decisions are applied.

To conclude, as i told in last msg, philosophy of crowd is well thinked to share security updates in real-time and as it's a french team ... like me.

Best Regards