Implement flexible filtering

[colmsnowplow]

There's a PR to implement a basic and quick solution to filter enriched events, as a means to provide something basic in a short timeline.

This issue concerns what the design of a flexible and extensible filtering functionality should look like - since we may wish to go ahead with that instrumentation, but in the long term we should put more consideration into the design of filtering.

Considerations already identified are as follows:

In order to facilitate combining filter conditions (who depend on different fields) with disjunction (condition A OR condition B), some redesign of configuration and how the filters are instrumented is likely required.

---

It seems unintuitive to treat filters as transformations, since acking within a transformation is odd, and also they seem qualitatively different.

So, we may wish to instrument a new pkg pattern for filtering specifically, to run before transformations, which would probably follow a roughly similar pattern to how transformations are instrumented.

The only potential drawback I see to this is that should a condition depend on the output of a transformation, this design wouldn't work (eg. we need to decode some data first or perform some computation first).

---

Alex has suggested embedding Lua for filtering. @alexanderdean I'm unfamiliar, what's the benefit of this approach?

[alexanderdean]

what's the benefit of this approach?

Greenspun's tenth rule of programming 🤖. Before you know it you are supporting multiple conditions, or an event name that has to start with 'foo', and you need some funky nested array handling, and you might as well have started with something that is Turing complete...

[colmsnowplow]

That makes sense. Interesting suggestion - for sure that exact issue is on my mind already so worth looking into whether Lua solves that issue. Thanks!

[colmsnowplow]

Closing as we're implementing regex based filtering on any field - once that's in place we can see what further requirements arise in future.