Fluentbit healthcheck and kafka downtime

[serut]

When fluentbit does not resolve the output domain anymore, I would expect the fluentbit healthiness to be mark as unhealthy.
I have sometimes network issues on my cluster and would expect fluentbit to shutdown as it cannot connect to my output anymore.

But it does not, it still returns ok.

So here is my setup : I've just removed the my-kafka service so fluentbit tries to connect to a service that does not exist.

Here is my configuration:

# See https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
# Main service configuration
[SERVICE]
    Log_Level       info
    Parsers_File    parsers.conf
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_PORT    2020
    Health_Check On 
    HC_Errors_Count 10
    HC_Retry_Failure_Count 10
    HC_Period 10

[INPUT]
    Name Forward
    Port 24224
    mem_buf_limit   20M

# [FILTER]
#     Name  stdout
#     Match *

# Filter all logs to define the time key which is used by Loki as log datetime
# Transform the timestamp provided by Docker into a ISO8601 datetime
# See override_time.lua for implementation
[FILTER]
    Name lua
    Match *
    script build_time.lua
    call build_time
    time_as_table true

# Parser is defined in parsers.conf
# Permitted to transform all logs in json format
# if logs is not under json format, we will have
# a 'log' and 'time' keys by default
[FILTER]
    Name            parser
    Parser          docker
    Match           *
    Key_Name        log
    Reserve_Data    True


# Add hostname key to json log
[FILTER]
    Name            record_modifier
    Match           *
    Record hostname ${NODE_HOSTNAME}

# Transmits logs to message queue kafka
[OUTPUT]
    Name                        kafka
    Match                       *
    Brokers                     my-kafka:9092
    Topics                      stdout-containers
    storage.total_limit_size    20M

    net.dns.resolver            LEGACY

Here is the log :

Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/12/18 11:47:45] [ info] [fluent bit] version=2.2.0, commit=db8487d123, pid=1
[2023/12/18 11:47:45] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/12/18 11:47:45] [ info] [cmetrics] version=0.6.4
[2023/12/18 11:47:45] [ info] [ctraces ] version=0.3.1
[2023/12/18 11:47:45] [ info] [input:forward:forward.0] initializing
[2023/12/18 11:47:45] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
[2023/12/18 11:47:45] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2023/12/18 11:47:45] [ info] [output:kafka:kafka.0] brokers='my-kafka:9092' topics='stdout-containers'
[2023/12/18 11:47:45] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2023/12/18 11:47:45] [ info] [sp] stream processor started
[2023/12/18 11:49:04] [ info] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Disconnected (after 41913ms in state UP)
[2023/12/18 11:49:04] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: 2/2 brokers are down
[2023/12/18 11:49:04] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Disconnected (after 41913ms in state UP)
[2023/12/18 11:49:04] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 136ms in state CONNECT)
[2023/12/18 11:49:04] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 4ms in state CONNECT, 1 identical error(s) suppressed)
[2023/12/18 11:49:05] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Connect to ipv4#10.11.1.216:9092 failed: Connection refused (after 1004ms in state CONNECT)
[2023/12/18 11:49:05] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 136ms in state CONNECT)
[2023/12/18 11:49:05] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 4ms in state CONNECT, 1 identical error(s) suppressed)
[2023/12/18 11:49:05] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Connect to ipv4#10.11.1.216:9092 failed: Connection refused (after 1004ms in state CONNECT)
[2023/12/18 11:49:05] [ warn] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: my-kafka-1:9092/1: Metadata request failed: broker down: Local: Host resolution failure (0ms): Permanent
[2023/12/18 11:49:05] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 3ms in state CONNECT)
[2023/12/18 11:49:06] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 3ms in state CONNECT)
[2023/12/18 11:49:07] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 3ms in state CONNECT, 1 identical error(s) suppressed)
[2023/12/18 11:49:08] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 3ms in state CONNECT, 1 identical error(s) suppressed)
[2023/12/18 11:49:37] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 5ms in state CONNECT, 8 identical error(s) suppressed)
[2023/12/18 11:49:37] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:49:38] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 5ms in state CONNECT, 8 identical error(s) suppressed)
[2023/12/18 11:49:38] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:50:08] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 16 identical error(s) suppressed)
[2023/12/18 11:50:09] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 16 identical error(s) suppressed)
[2023/12/18 11:50:15] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 6ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:50:15] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 6ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:50:38] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:50:40] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:50:55] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 3ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:50:55] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 3ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:51:10] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 6ms in state CONNECT, 16 identical error(s) suppressed)
[2023/12/18 11:51:11] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 6ms in state CONNECT, 16 identical error(s) suppressed)
[2023/12/18 11:51:32] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 3ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:51:36] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 3ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:51:40] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:51:41] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:52:02] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 6ms in state CONNECT, 3 identical error(s) suppressed)
[2023/12/18 11:52:06] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 6ms in state CONNECT, 3 identical error(s) suppressed)
[2023/12/18 11:52:10] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:52:11] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:52:41] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:52:41] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:52:42] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 11ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:52:44] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 11ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:53:11] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:53:11] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 4ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:53:18] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 5ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:53:19] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 5ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:53:41] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka:9092/bootstrap]: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:53:41] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka:9092/bootstrap: Failed to resolve 'my-kafka:9092': Name or service not known (after 5ms in state CONNECT, 15 identical error(s) suppressed)
[2023/12/18 11:53:58] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:my-kafka-1:9092/1]: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 5ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:54:00] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: my-kafka-1:9092/1: Failed to resolve 'my-kafka-1:9092': Name or service not known (after 5ms in state CONNECT, 4 identical error(s) suppressed)
[2023/12/18 11:54:10] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out
[2023/12/18 11:54:10] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out
[2023/12/18 11:54:10] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out

When I check the result of the /api/v2/metrics endpoint, I can see the fluentbit_output_ looks like everything working well, explaining why the health endpoint returns "ok":

2023-12-18T12:27:28.576795784Z fluentbit_uptime{hostname="rs-fluentbit"} = 961
2023-12-18T12:27:28.544344914Z fluentbit_input_bytes_total{name="forward.0"} = 1311701
2023-12-18T12:27:28.544344914Z fluentbit_input_records_total{name="forward.0"} = 2506
2023-12-18T12:27:28.544350927Z fluentbit_filter_records_total{name="lua.0"} = 2506
2023-12-18T12:27:28.544350927Z fluentbit_filter_bytes_total{name="lua.0"} = 15841842
2023-12-18T12:11:27.881291467Z fluentbit_filter_add_records_total{name="lua.0"} = 0
2023-12-18T12:11:27.881291467Z fluentbit_filter_drop_records_total{name="lua.0"} = 0
2023-12-18T12:27:28.544350927Z fluentbit_filter_records_total{name="parser.1"} = 2506
2023-12-18T12:27:28.544350927Z fluentbit_filter_bytes_total{name="parser.1"} = 15906752
2023-12-18T12:11:27.881877981Z fluentbit_filter_add_records_total{name="parser.1"} = 0
2023-12-18T12:11:27.881877981Z fluentbit_filter_drop_records_total{name="parser.1"} = 0
2023-12-18T12:27:28.544350927Z fluentbit_filter_records_total{name="record_modifier.2"} = 2506
2023-12-18T12:27:28.544350927Z fluentbit_filter_bytes_total{name="record_modifier.2"} = 15830544
2023-12-18T12:11:27.881921989Z fluentbit_filter_add_records_total{name="record_modifier.2"} = 0
2023-12-18T12:11:27.881921989Z fluentbit_filter_drop_records_total{name="record_modifier.2"} = 0
2023-12-18T12:27:27.577670051Z fluentbit_output_proc_records_total{name="kafka.0"} = 2501
2023-12-18T12:27:27.577670051Z fluentbit_output_proc_bytes_total{name="kafka.0"} = 1442346
2023-12-18T12:11:27.881961665Z fluentbit_output_errors_total{name="kafka.0"} = 0
2023-12-18T12:11:27.881961665Z fluentbit_output_retries_total{name="kafka.0"} = 0
2023-12-18T12:11:27.881961665Z fluentbit_output_retries_failed_total{name="kafka.0"} = 0
2023-12-18T12:11:27.881961665Z fluentbit_output_dropped_records_total{name="kafka.0"} = 0
2023-12-18T12:11:27.881961665Z fluentbit_output_retried_records_total{name="kafka.0"} = 0
2023-12-18T12:27:28.576795784Z fluentbit_process_start_time_seconds{hostname="rs-fluentbit"} = 1702901487
2023-12-18T12:27:28.576795784Z fluentbit_build_info{hostname="rs-fluentbit",version="2.2.0",os="linux"} = 1702901487
2023-12-18T12:27:28.576795784Z fluentbit_hot_reloaded_times{hostname="rs-fluentbit"} = 0
2023-12-18T12:27:28.576815977Z fluentbit_storage_chunks = 1
2023-12-18T12:27:28.576815977Z fluentbit_storage_mem_chunks = 1
2023-12-18T12:27:28.576815977Z fluentbit_storage_fs_chunks = 0
2023-12-18T12:27:28.576815977Z fluentbit_storage_fs_chunks_up = 0
2023-12-18T12:27:28.576815977Z fluentbit_storage_fs_chunks_down = 0
2023-12-18T12:27:27.576822564Z fluentbit_input_storage_overlimit{name="forward.0"} = 0
2023-12-18T12:27:27.576822564Z fluentbit_input_storage_memory_bytes{name="forward.0"} = 1432
2023-12-18T12:27:27.576822564Z fluentbit_input_storage_chunks{name="forward.0"} = 1
2023-12-18T12:27:27.576822564Z fluentbit_input_storage_chunks_up{name="forward.0"} = 1
2023-12-18T12:27:27.576822564Z fluentbit_input_storage_chunks_down{name="forward.0"} = 0
2023-12-18T12:27:27.576822564Z fluentbit_input_storage_chunks_busy{name="forward.0"} = 0
2023-12-18T12:27:27.576822564Z fluentbit_input_storage_chunks_busy_bytes{name="forward.0"} = 0
2023-12-18T12:11:27.881961665Z fluentbit_output_upstream_total_connections{name="kafka.0"} = 0
2023-12-18T12:11:27.881961665Z fluentbit_output_upstream_busy_connections{name="kafka.0"} = 0

Env:

• Fluentbit 2.2.0
• running inside docker contianer