Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Telemetry Errors Ignore Log Format #29188

Open
michaelkosir opened this issue Dec 13, 2024 · 1 comment
Open

Telemetry Errors Ignore Log Format #29188

michaelkosir opened this issue Dec 13, 2024 · 1 comment
Labels
bug Used to indicate a potential bug core/telemetry

Comments

@michaelkosir
Copy link
Contributor

michaelkosir commented Dec 13, 2024

Describe the bug
Telemetry errors do not follow the log format set in the Vault configuration file or environment variable.

To Reproduce

$ vault server -config vault.hcl

WARNING: storage configured to use "inmem" which should NOT be used in
production

==> Vault server configuration:

Administrative Namespace: 
                     Cgo: disabled
   Environment Variables: DBUS_SESSION_BUS_ADDRESS, GOTRACEBACK, HOME, LANG, LESSCLOSE, LESSOPEN, LOGNAME, LS_COLORS, OLDPWD, PATH, PWD, SHELL, SHLVL, SSH_CLIENT, SSH_CONNECTION, SSH_TTY, TERM, USER, VAULT_ADDR, VAULT_TOKEN, XDG_DATA_DIRS, XDG_RUNTIME_DIR, XDG_SESSION_CLASS, XDG_SESSION_ID, XDG_SESSION_TYPE, _
              Go Version: go1.22.8
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", disable_request_limiter: "false", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.18.2, built 2024-11-20T11:24:56Z
             Version Sha: e36bac59ddb8e10e8912c0ddb44416c850939855

==> Vault server started! Log data will stream in below:

{"@level":"info","@message":"proxy environment","@timestamp":"2024-12-13T10:58:12.889236-06:00","http_proxy":"","https_proxy":"","no_proxy":""}
{"@level":"info","@message":"incrementing seal generation","@timestamp":"2024-12-13T10:58:12.889389-06:00","generation":1}
{"@level":"warn","@message":"no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set","@timestamp":"2024-12-13T10:58:12.889403-06:00"}
{"@level":"info","@message":"Initializing version history cache for core","@module":"core","@timestamp":"2024-12-13T10:58:12.910885-06:00"}
{"@level":"info","@message":"Starting event system","@module":"events","@timestamp":"2024-12-13T10:58:12.910898-06:00"}
2024-12-13 10:58:14.021923 I | [ERR] Error flushing to statsd! Err: write udp 127.0.0.1:51053->127.0.0.1:8125: write: connection refused
2024-12-13 10:58:19.118196 I | [ERR] Error flushing to statsd! Err: write udp 127.0.0.1:55100->127.0.0.1:8125: write: connection refused
2024-12-13 10:58:24.323977 I | [ERR] Error flushing to statsd! Err: write udp 127.0.0.1:45287->127.0.0.1:8125: write: connection refused
2024-12-13 10:58:31.311091 I | [ERR] Error flushing to statsd! Err: write udp 127.0.0.1:48204->127.0.0.1:8125: write: connection refused

Expected behavior
Telemetry errors should follow the log format defined in the VAULT_LOG_FORMAT environment variable or log_format set in the Vault configuration file.

Environment:

Description:   Ubuntu 24.04.1 LTS
Architecture:  aarch64

Version: Vault v1.18.2, built 2024-11-20T11:24:56Z
Version Sha: e36bac59ddb8e10e8912c0ddb44416c850939855

Vault server configuration file(s):

log_format = "json"
log_level  = "info"

storage "inmem" {}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}

telemetry {
  disable_hostname      = true
  enable_hostname_label = true
  statsd_address        = "127.0.0.1:8125"
}
@heatherezell heatherezell added core/telemetry bug Used to indicate a potential bug labels Dec 13, 2024
@davidadeleon
Copy link
Contributor

Hi @michaelkosir! Thanks for submitting this issue! Yes it does appear as this is the case. It looks like this is due to the underlying go-metrics library performing a log.Printf directly which bypasses the formatting in place by Vault and its hclogger. I'm taking a look to see how best to handle this situation here as these particular logs are from a longer running goroutine to routinely flush these metrics so its not something we can just bubble any underlying errors up to the caller in this case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Used to indicate a potential bug core/telemetry
Projects
None yet
Development

No branches or pull requests

3 participants