Configuring rewrite_tag buffering for high performance and reliability

[zhenyami]

Hey!

I'm curious how others configure their logging pipelines that make use of the rewrite_tag filter. Documentation recommends using the filesystem buffering to help with backpressure from the output destination, usually a web-service. How does one configure filesystem storage when the logging pipeline includes both tail and rewrite_tag plugins?

We have two setups:

tail + output (with unbuffered filters in between)

[INPUT]
    Name                                tail
    storage.type                        filesystem
    storage.pause_on_chunks_overlimit   On
    Mem_Buf_Limit                       50MB

We found that if the output destination is not reachable, e.g. there's an outage, then Fluent Bit starts excessively writing to disk once the disk buffer limits are reached, especially with high volume of logs. This results in high CPU iowait. This problem is more apparent in our setups that place Fluent Bit buffers on AWS EBS storage. Therefore, we have to enable storage.pause_on_chunks_overlimit.

tail + rewrite_tag + output (with unbuffered filters in between)

[INPUT]
    Name                                tail
    storage.type                        memory
    storage.pause_on_chunks_overlimit   On
    Mem_Buf_Limit                       50MB

[FILTER]
    Name                                rewrite_tag
    Emitter_Storage.type                filesystem
    Emitter_Mem_Buf_Limit               50M

As I understand, when rewrite_tag is present, the tail input doesn't require filesystem buffering. It might be detrimental even – I saw that Fluent Bit wrote dozens of tail buffer files each second (resulting in high disk IO), which were immediately removed (because no backpressure?). Therefore, in this case we only enable filesystem buffering for the rewrite_tag emitter. However, it looks like we have a similar problem when the output destination service is down and the filesystem storage max limit is reached – disk IO and CPU iowait are increased by a lot.

Are my assumptions correct or did others have a different experience? Any advice on the best way to configure Fluent Bit in such scenario? Should we add a config parameter similar to storage.pause_on_chunks_overlimit to rewrite_tag plugin to combat this issue?

Thanks!

CC: @agup006