Merge files based on size

[farazoman]

From my understanding of our the Output filter works is that it will move files every x minutes or when the size exceeds a certian amount. But I was wondering if the follow use-case exists that is solved by fluent-bit.

For every x minutes, if the file size is not exceeded appended to the existing file (this is for s3 ingestion). But still update every x mins (to get quick updates)

The issue is we end up having a large amount of files which is hard to go through manually. Is it possible to do what I'm referring to or would this be a feature request?

[agup006]

Hi @farazoman thanks for submitting this post, I'm a bit confused on the ask here.

Fluent Bit chunks incoming data in sizes of 2MB each then routes that data to specified outputs. In the case of S3 there is a custom store_dir parameter where the data is further queued

[agup006]

ah I see, this may be part of the upload_chunk_size parameter in the S3 plugin. I'm adding @PettitWesley who probably knows better

[PettitWesley]

Fluent Bit S3 does what you are asking for. Please read its docs. The total_file_size parameter is a target for the total file size in S3, its what the plugin will try to achieve. But if the upload_timeout expires first, then that overrides the file size and you will get whatever size of data accumulated in that time. The upload_chunk_size parameter is for multipart uploads and defines the size of each part in the multipart upload. Again the docs cover this.

Basically, Fluent Bit S3 uses a special buffering mechanism that is on top of the Fluent Bit internal buffering mechanisms. S3 buffers are stored in the store_dir.

[farazoman]

@PettitWesley thanks for the response, let me look into the total_file_size parameter, perhaps I misread somewhere. Appreciate it!

[farazoman]

@PettitWesley as I was re-reading the docs (https://docs.fluentbit.io/manual/pipeline/outputs/s3), the limitation of that approach you're describing is that if we want our files to update frequently (in my case every 5 minutes), we're opening 12 files per hour of logs we're looking for.

What I was wondering if there exists the functionality to append to s3 files every (in our case 5 minutes / file_size_limit ) until a merge_file_size_limit, i.e it will keep appending the chunks to an s3 file until that s3 file reaches a certain size.

fyi my config:

  outputs: |
    [OUTPUT]
        Name s3
        Match *
        bucket {{ logging[aws_region]['s3_logging_bucket'] }}
        region {{ aws_region }}
        total_file_size 250M
        s3_key_format <some_path>/%Y-%m-%d/$TAG[4]/%H-%M-%S
        s3_key_format_tag_delimiters ._
        upload_timeout 5m
        store_dir /home/ec2-user/buffer
        role_arn {{ logging[aws_region]['s3_log_role_arn'] }}

I'm wondering if I remove the seconds part from the name will it change anything? Because Fluent will auto append a unique string to the end of the file name in s3 to avoid overwriting files.

[PettitWesley]

I see. This is a feature request. Fluent Bit s3 does not have any ability to append to files inside of the S3 bucket. Right, if you want logs in the bucket within 5 minutes you must accept small files one created every 5 minutes.

I'm wondering if I remove the seconds part from the name will it change anything? Because Fluent will auto append a unique string to the end of the file name in s3 to avoid overwriting files.

That'd just remove the seconds part. Fluent Bit will append a random UUID at the end unless you specify $UUID somewhere inside the s3_key_format.