protobuf tag not enough fields in MetricFamily.state

[sawyersteven]

What happened?

When running cscli metrics the following was printed:

ERRO[02-03-2024 00:22:59] crowdsec - goroutine crowdsec/ShowPrometheus crashed : protobuf tag not enough fields in MetricFamily.state:
ERRO[02-03-2024 00:22:59] please report this error to https://github.com/crowdsecurity/crowdsec/
ERRO[02-03-2024 00:22:59] stacktrace/report is written to /tmp/crowdsec-crash.128475427.txt : please join it to your issue
FATA[02-03-2024 00:22:59] crowdsec stopped

What did you expect to happen?

Metrics to print

How can we reproduce it (as minimally and precisely as possible)?

Ubuntu 24.04, the following commands:

curl https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | os=ubuntu dist=jammy bash
apt install crowdsec
apt install crowdsec-firewall-bouncer-iptables
cscli metrics

No configs have been modified.

Anything else we need to know?

Crash log: https://pastebin.com/JgPEaZtp

Crowdsec version

$ cscli version
2024/03/02 00:27:13 version: v1.4.6-6build1-debian
2024/03/02 00:27:13 Codename: alphaga
2024/03/02 00:27:13 BuildDate: 2023-08-24_05:13:32
2024/03/02 00:27:13 GoVersion: 1.21.0
2024/03/02 00:27:13 Platform: linux
2024/03/02 00:27:13 Constraint_parser: >= 1.0, <= 2.0
2024/03/02 00:27:13 Constraint_scenario: >= 1.0, < 3.0
2024/03/02 00:27:13 Constraint_api: v1
2024/03/02 00:27:13 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

$ cscli hub list -o raw
# paste output here

Acquisition config

``` $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- source: journalctl journalctl_filter: - "_SYSTEMD_UNIT=ssh.service" labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2 ```

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /var/lib/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
  - Hub Folder              : /var/lib/crowdsec/hub
Local API Server:
  - Listen URL              : 127.0.0.1:8080
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

$ cscli metrics
# See above

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions]

@sawyersteven: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

Seems you may have two crowdsec binaries on the system as the version is quite outdated

which -a crowdsec

[sawyersteven]

$ which -a crowdsec
/usr/bin/crowdsec
/bin/crowdsec

I'm not sure how more than one binary got installed... This is a relatively fresh os install and everything else I run is containerized.

edit:

Running apt purge crowdsec results in both binaries being removed, but apt install crowdsec puts them both back.

[LaurenceJJones]

$ which -a crowdsec
/usr/bin/crowdsec
/bin/crowdsec

I'm not sure how more than one binary got installed... This is a relatively fresh os install and everything else I run is containerized.

edit:

Running apt purge crowdsec results in both binaries being removed, but apt install crowdsec puts them both back.

or maybe you have two cscli ? cause the version output you provided says 1.4.6

[sawyersteven]

$ which -a cscli
/usr/bin/cscli
/bin/cscli

But after an apt purge of everything and a reboot the cscli metrics command works as expected. But it still shows two paths (bin and usr/bin) for both cscli and crowdsec.

So if it runs I'll just have to do some good pen tests to make sure it is working correctly and actually blocks bad actors.

[mmetc]

In many distributions /bin is a link to /usr/bin so seeing two binaries is not a surprise.

[Asrael00md]

Check your /etc/apt/sources.list.d for the content of the relevant repository.
I do use Debian testing and the script.deb.sh created trixie instead of newer versions (like bookworm) as link to the repository.
You probably currently do see an error when running apt update for the URL.
After changing trixie -> bookworm (or relevant latest supported Ubuntu version), run
'apt update & apt upgrade'
You should see an upgrade to the latest Version of crowdsec (1.6.1*)

This fixed the cscli metrics issue for me.