FluentBit Performance for persistent TCP connection

[brunodeoliveiramacedo]

I'm not able to reach the full CPU/MEM capacity of my fluent bit.

I'm using Fluentbit to forward some messages to Kafka. I can see the performance degradation when I start to forward more messages to Fluentbit.

This is my current configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [INPUT]
        Name         tcp
        Listen       0.0.0.0
        Port         24224
        Format       none

    [INPUT]
        Name         tcp
        Listen       0.0.0.0
        Port         514
        Format       none
        Buffer_Size  1024KB   
        Chunk_Size   512KB

    [INPUT]
        Name         udp
        Listen       0.0.0.0
        Port         514
        Format       none
        Buffer_Size  1024KB   
        Chunk_Size   512KB

    [SERVICE]
        Flush        1 
        Daemon       Off
        Log_Level    info
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Health_Check On
        Mem_Buf_Limit 2GB 
    
        
    [OUTPUT]
        Name         kafka
        Match        *
        Brokers      kafka-broker:9200
        Topics       syslogs
        Retry_Limit  False
        Format       json
        rdkafka.queue.buffering.max.ms 100  
        rdkafka.request.required.acks 1     
        rdkafka.message.send.max.retries 5  
        rdkafka.retry.backoff.ms 100

      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2.1-debug
          securityContext:
            runAsUser: 0
          command: ["/fluent-bit/bin/fluent-bit"]
          args: ["-c", "/fluent-bit/etc/fluent-bit.conf"]
          resources:
            requests:
              memory: "524Mi"
              cpu: "1000m"
            limits:
              memory: "1Gi"
              cpu: "2000m"

I also have persistent TCP connections. So I would have for example two connections forwarding all the messages for a single FluentBit pod.

I would need to handle:
~ 70MB per sec
~ 700 bytes per message
110.000 messages per sec

Would you have any configuration ideas that could improve my performance for these persistent TCP connections?

[patrick-stephens]

You could try workers on the output side and threaded on on the input side to provide dedicated threads potentially?

[brunodeoliveiramacedo]

hey @patrick-stephens thanks for the reply :D

I believe fluent bit doesn't support these options. At least I couldn't find into TCP input and Kafka output

https://docs.fluentbit.io/manual/pipeline/inputs/tcp
https://docs.fluentbit.io/manual/pipeline/outputs/kafka

[patrick-stephens]

They may not be in the docs but settings that are common to all plugins are typically not documented either, e.g. TLS.

PRs are always welcome for docs, it's hard to always keep them correct.

Not all outputs support workers >1 due to race conditions.

[patrick-stephens]

At the moment I think all your input (and filtering/etc.) is running on the main thread so CPU is restricted to that.

[brunodeoliveiramacedo]

that worked, omg, thanks a lot of @patrick-stephens

I'll work on the PR for the doc, for sure :D

[patrick-stephens]

Sometimes I'm an expert lol

[brunooliveiramac]

fluent/fluent-bit-docs#1312

[brunooliveiramac]

Hi @patrick-stephens
I'm still not being able to handle more throughput. The worker improved the performance, but I needed to handle more traffic.

     [INPUT]
        Name         tcp
        Listen       0.0.0.0
        Port         514
        Format       none
        Buffer_Size  1024KB   
        Chunk_Size   512KB
        Threaded      on
        Tag          syslogs

I receive around 80MB/sec

[brunooliveiramac]

If you would know any other way to get this better :D