help request: Loki instance external to cluster returning 404

[Martyn10]

Describe the issue

Hi,
I am trying to forward logs to a Loki instance outside of the Kubernetes cluster. I get the below 404 error in the fluent-bit container logs.
[error] [output:loki:loki.0] logging-dev.example.com:443, HTTP status=404
404 page not found

A default Loki install returns a 404 at logging-dev.example.com

I have used curl to access the Loki API from the fluent-bit debug container successfully.

I noticed the uri parameter doesn't exist when i export the fluent-bit-config secret and it isn't defined in fluentbit-output-loki.yaml . Could this be part of the issue?

How did you install fluent operator?

helm install -f custom-values.yml fluent-operator --create-namespace -n fluent-operator charts/fluent-operator/

custom-values.yml

containerRuntime: containerd

operator:

initcontainer:

    repository: "docker"

    tag: "27.2.1"

fluentbit:

crdsEnable: true

enable: true

image:

    repository: "ghcr.io/fluent/fluent-operator/fluent-bit"

    tag: "3.1.7-debug"

tls:

    verify: false

input:

    tail:

        enable: true

        refreshIntervalSeconds: 10

        memBufLimit: 100MB

        bufferMaxSize: ""

        path: "/var/log/containers/*.log"

        skipLongLines: true

        readFromHead: false

        storageType: memory

        pauseOnChunksOverlimit: "off"

    systemd:

        enable: true

        systemdFilter:

            enable: true

            filters: []

        path: "/var/log/journal"

        includeKubelet: true

        stripUnderscores: "off"

        storageType: memory

        pauseOnChunksOverlimit: "off"

output:

    loki:

        enable: true

        host: logging-dev.example.com

        port: 443

        uri: /loki/api/v1/push

        httpUser: user

        httpPassword: pass

---

Additional context

Output of fluent-bit-config secret:
[Service]

Http_Server    true

Parsers_File    /fluent-bit/etc/parsers.conf

Parsers_File    /fluent-bit/config/parsers_multiline.conf

[Input]

Name    systemd

Path    /var/log/journal

DB    /fluent-bit/tail/systemd.db

DB.Sync    Normal

Tag    service.*

Systemd_Filter    _SYSTEMD_UNIT=containerd.service

Systemd_Filter    _SYSTEMD_UNIT=kubelet.service

Strip_Underscores    off

storage.type    memory

[Input]

Name    tail

Path    /var/log/containers/*.log

Read_from_Head    false

Refresh_Interval    10

Skip_Long_Lines    true

DB    /fluent-bit/tail/pos.db

DB.Sync    Normal

Mem_Buf_Limit    100MB

Parser    cri

Tag    kube.*

storage.type    memory

[Filter]

Name    lua

Match    kube.*

script    /fluent-bit/config/containerd.lua

call    containerd

time_as_table    true

[Filter]

Name    kubernetes

Match    kube.*

Kube_URL    https://kubernetes.default.svc:443

Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token

Labels    false

Annotations    false

[Filter]

Name    nest

Match    kube.*

Operation    lift

Nested_under    kubernetes

Add_prefix    kubernetes_

[Filter]

Name    modify

Match    kube.*

Remove    stream

Remove    kubernetes_pod_id

Remove    kubernetes_host

Remove    kubernetes_container_hash

[Filter]

Name    nest

Match    kube.*

Operation    nest

Wildcard    kubernetes_*

Nest_under    kubernetes

Remove_prefix    kubernetes_

[Filter]

Name    lua

Match    service.*

script    /fluent-bit/config/systemd.lua

call    add_time

time_as_table    true

[Output]

Name    loki

Match_Regex    (?:kube|service)\.(.*)

host    logging-dev.example.com

port    443

http_user    user

http_passwd    pass