buffer params explanation

[mirisu2]

Hi,
From documentation the chunk size is approximately 2MB.
The Tail plugin supports the following configuration parameters:
Buffer_Chunk_Size and Buffer_Max_Size
I'm trying to understand what this params are? What for? How these parameters correlate with mem_buf_limit and how many files the plugin reads?

mem_buf_limit is a limit of memory for all files the input plugin reads.
But what about Buffer_Chunk_Size and Buffer_Max_Size ?

storage.max_chunks_pause - Specifies if file storage is to be paused when reaching the chunk limit. reaching the chunk limit - what does it mean?

From documentation:
So, if we impose a limit of 10MB for the input plugins and consider the worse case scenario of the output plugin consuming 20MB extra, as a minimum we need (30MB x 1.2) = 36MB.
1.2 - what this multiplier is?
Why to sum a limit of input plugin with output plugin? Mem_Buf_Limit is limit for input plugin.

Can anyone to explain?

[hsingli20]

https://docs.fluentbit.io/manual/pipeline/inputs/tail.
The document is not clear to set the parameters, like Buffer_Chunk_Size and Buffer_Max_Size.
Issues:
log missings about 60% when Buffer_Chunk_Size is 32K, the size of one line log is about 4K. After change Buffer_Chunk_Size to 1M, the log missing is 0%.
Could you give more indicative methods to set these parameters?
when set it too small, what's the impact? once set it too large, what's the impact?

[emmacz]

Any update on this pls.?

[zi1611061206]

I also faced headaches when considering setting these parameters; the issue is that the documentation doesn’t clearly explain the correlation between them.