Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

higress开启o11y配置后higress-console-loki、higress-console-prometheus pod启动不起来 #1668

Open
heartbeat2013 opened this issue Jan 13, 2025 · 3 comments

Comments

@heartbeat2013
Copy link

higress开启可观测配置o11y后pod启动不起来,higress-console-loki报错如下:
mkdir /var/loki/rules: permission denied
error initialising module: ruler-storage
github.com/grafana/dskit/modules.(*Manager).initModule
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/pkg/loki.(*Loki).Run
/src/loki/pkg/loki/loki.go:461
main.main
/src/loki/cmd/loki/main.go:110
runtime.main
/usr/local/go/src/runtime/proc.go:267
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
level=warn ts=2025-01-13T11:46:13.803585436Z caller=loki.go:288 msg="global timeout not configured, using default engine timeout ("5m0s"). This behavior will change in the next major to always use the default global timeout ("5m")."
level=info ts=2025-01-13T11:46:13.804124299Z caller=main.go:108 msg="Starting Loki" version="(version=2.9.4, branch=HEAD, revision=f599ebc535)"
level=info ts=2025-01-13T11:46:13.806002814Z caller=server.go:322 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=error ts=2025-01-13T11:46:13.807895043Z caller=log.go:230 msg="error running loki" err="mkdir /var/loki/rules: permission denied\nerror initialising module: ruler-storage\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

higress-console-prometheus报错如下:
ts=2025-01-13T11:45:41.627Z caller=main.go:501 level=warn deprecation_notice="'storage.tsdb.retention' flag is deprecated use 'storage.tsdb.retention.time' instead."
ts=2025-01-13T11:45:41.627Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.40.7, branch=HEAD, revision=ab239ac5d43f6c1068f0d05283a0544576aaecf8)"
ts=2025-01-13T11:45:41.628Z caller=main.go:561 level=info build_context="(go=go1.19.4, user=root@afba4a8bd7cc, date=20221214-08:49:43)"
ts=2025-01-13T11:45:41.628Z caller=main.go:562 level=info host_details="(Linux 5.4.119-19.0009.44 #1 SMP Tue May 7 20:08:55 CST 2024 x86_64 higress-console-prometheus-84fd87cffd-929kd (none))"
ts=2025-01-13T11:45:41.628Z caller=main.go:563 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2025-01-13T11:45:41.628Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2025-01-13T11:45:41.630Z caller=query_logger.go:91 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7ffc5a04cd72, 0xb}, 0x14, {0x3b9fa20, 0xc0006a6690})
/app/promql/query_logger.go:121 +0x3cd
main.main()
/app/cmd/prometheus/main.go:618 +0x6973

higress-console values.yaml配置如下:

global:
  local: false # When deploying to a local cluster (e.g.: kind cluster), set this to true.
  kind: false # Deprecated. Please use "global.local" instead. Will be removed later.
  ingressClass: "higress"
  # Observability (o11y) configurations
  o11y:
    enabled: true
    grafana:
      replicas: 1
      image:
        repository: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/grafana
        tag: 9.3.6
      port: 3000
      storage: 10Gi
      pvc:
        storageClassName: "cbs"
      securityContext: {}
    prometheus:
      replicas: 1
      image:
        repository: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/prometheus
        tag: v2.40.7
      port: 9090
      storage: 10Gi
      pvc:
        storageClassName: "cbs"
      resources:
        limits:
          cpu: 500m
          memory: 2Gi
      securityContext: {}
    loki:
      replicas: 1
      image:
        repository: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/loki
        tag: 2.9.4
      ports:
        http: 3100
        grpc: 9095
      storage: 10Gi
      pvc:
        storageClassName: "cbs"
      securityContext: {}
  pvc:
    rwxSupported: false

replicaCount: 1
@heartbeat2013
Copy link
Author

heartbeat2013 commented Jan 13, 2025

pod下securityContext添加如下配置,配置运行用户为root,已解决

securityContext:
    runAsUser: 0  # 设置为 root 用户(UID 0)
    runAsGroup: 0  # 可选,设置为 root 组(GID 0)

@johnlanni
Copy link
Collaborator

cc @CH3CHO loki和prometheus是否默认用root启动

@CH3CHO
Copy link
Collaborator

CH3CHO commented Jan 14, 2025

cc @CH3CHO loki和prometheus是否默认用root启动

这个我觉得问题倒不大。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants