Looking for suggestions on how we can improve lua filters

[tarruda]

We are looking for ways to improve user experience of lua filters and would like to ask the community for suggestions on how this can be done. One idea we currently have is to expose a standard library with commonly used utilities that will be available for all lua scripts. As an example, we could add a few functions that would simplify timestamp/table manipulation.

If any lua filter users would like to contribute to this discussion, please feel free to do so. Also, if you post here scripts that you wrote, we might have other ideas on how we can make things easier for you.

[edsiper]

cc: @pranavmarla

[edsiper]

From Slack

https://fluent-all.slack.com/archives/C0CTQGHKJ/p1642553528164700

[pwhelan]

I have two ideas:

• an emit function for emitting arbitrary records
• persistent (between invocations) memory inside the lua filters

Emitting records could help cases where an HTTP API or some other mechanism sends multiple metrics or logs inside a single record. If we combine that with the ability to store records between invocations lua filters could also perform a form of map/reduce on records to emit a single record from multiple.

[pwhelan]

Apparently this is supported:

https://docs.fluentbit.io/manual/pipeline/filters/lua:

The Lua filter allows you to modify the incoming records (even split one record into multiple records) using custom ...

Also, nothing stops someone from using a global variable to hold records between invocations. The only corner case I can think of is records being left behind since there is no way to tell if the lua filter is being invoked during the final flush during exit.

[ksa-real]

It's actually a good idea. Can a Lua script be invoked based on set conditions with empty payload? E.g. if there is no input for certain time and pre-shutdown?

[kurtismullins]

Suggestion:

A simple function to create a new timestamp with nanosecond precision.

Background:

I have encountered a situation where the timestamps provided via-syslog are unreliable. An example where this might occur in the wild is when using syslog with machines which experience time-drift and aren't corrected quickly enough.

I know the best way to solve this problem is to fix the root cause, but it would be very useful to have the ability to simply tack on our own (accurate) timestamp. Unfortunately using os.date() or os.time() is not precise enough, especially when dealing with syslog.

Example:

function append_time(tag, timestamp, record)
    new_record = record
    new_record["fluentbit_timestamp"] = fb_date() -- (this does not actually exist AFAIK)
    return 1, timestamp, new_record
end

[ralphw]

I'd love to see a way to retrieve a custom LUA/filtering configuration from a central source, so we can include or exclude certain windows Events by Event ID and Event Type.

Caching the configuration in a file on startup would be helpful.

[agup006]

Ah I like this example, we have a central repo here. Maybe we can add that in the following?: https://github.com/calyptia/fluent-bit-lua-samples

Also do you want to completely remove the message or are you looking to remove certain fields?

[tarruda]

Ah I like this example, we have a central repo here. Maybe we can add that in the following?: https://github.com/calyptia/fluent-bit-lua-samples

Eventually I'd like to integrate https://github.com/calyptia/fluent-bit-lua-samples with https://calyptia.github.io/sandbox/ so that examples can be quickly tested on the browser. Another cool feature would be if the user could edit the filter in the playground and click to export a complete yaml config file (with the lua code embedded)

[kishorv10]

Suggestion:
Support for Making HTTP Calls Using LUA Scripts for Fluent Bit Filters

Background:
We need to filter sensitive data from the logs, and an in-house service offers this capability. We're trying to create a new fluent-bit filter that can invoke this service over HTTP and modify the incoming records. However, with the current support of LUA scripts, there is no way to make HTTP calls. I would love to hear about any alternatives to accomplish this.

[tarruda]

Suggestion: Support for Making HTTP Calls Using LUA Scripts for Fluent Bit Filters

Background: We need to filter sensitive data from the logs, and an in-house service offers this capability. We're trying to create a new fluent-bit filter that can invoke this service over HTTP and modify the incoming records. However, with the current support of LUA scripts, there is no way to make HTTP calls. I would love to hear about any alternatives to accomplish this.

One simple way to do this is invoke curl command from Lua

[kishorv10]

Thank you for the suggestion. I am unsure if curl is part of the fluent bit distroless image. I believe you propose using LUA OS commands to invoke curl. This way, the curl output can't be captured, which is needed in our case.

[tarruda]

You can use local p = io.popen to start a process, and p:read('*a') to read all of its output. If you need to send data, first write it to a file and then pass to curl as an argument (cannot read and write from a process at the same time with popen).

Not sure if curl is present in the distroless image (probably not). In that case you would need to build your own image based on fluent-bit offical image.

[kishorv10]

Are any Lua json libraries available in Fluent Bit's out-of-the-box that can be used in Lua filter scripts?

[tarruda]

@kishorv10 no, but you can easily install with luarocks (lua package manager) when building a custom image.

Check out this example I made a couple of years ago. It doesn't contain any json examples, but you can use a similar setup to install json libraries available in luarocks registry: https://github.com/chronosphereio/calyptia-fluent-bit-lua-samples/blob/main/Dockerfile

[arunim2405]

I have seen many use cases for installing packages, and I was wondering if we can solve this by supporting compiled lua files (luac.out)
@tarruda

[tchrono]

I have seen many use cases for installing packages, and I was wondering if we can solve this by supporting compiled lua files (luac.out) @tarruda

You can probably add compiled lua modules to a docker image, IIRC Lua can import compiled files

[patrick-stephens]

There was a related discussion on this somewhere...

EDIT: #7749 (reply in thread)