Shared state across instances

[deangalvin-cb]

👋 Migrating from Discord here:
https://discord.com/channels/742820443487993987/865294205655973928/1336449652177764402

I was wondering if the concept of shared state has ever come up & if Vector has a stance on it.

Rationale: I would love to do rate-limiting on noisy services, however I can't reliably do that as I don't know which container is going to pickup that service's logs. I was thinking it would be lovely to utilize something like redis to share ratelimiting state across instances, however I wasn't sure if that had ever been discussed (if it has happy to move there instead as well!)

It would be really ideal if we could make VRL calls that had the ability to utilize external services like KV stores to share state across instances. This would allow for much more impactful transforms, sampling, and rate limiting.

[From Discord]

Pavlos Rontidis — Today at 4:58 PM
It's a challenging topic since we don't maintain any servers. We have been focusing on exclusively local components. But I can imagine designing some interfaces and allow the users to maintain their own infra. E.g. for a global quota you would need to maintain a key value store.

[deangalvin-cb]

Thats precisely what I was thinking, some interface or even predefined external sources for state storage.

Id imagine it would be useful to store state "conditionally" ie. storing every log that comes through is wildly expensive, but storing 1/100 would be much cheaper & still give valid results.

[pront]

Hi @deangalvin-cb, this is an exciting new area. It seems we are aligned on the very high level idea of an interface to abstract away infra/storage owned by users. A short RFC (so that we can review and commit to the repo) would work best here. One idea I would explore is an enrichment table that abstracts away a key/value store. But there might be more elegant/flexible ways to go about this.

[deangalvin-cb]

Yes! Thats pretty in line with what im thinking. Im not a rust dev by any stretch, but Im happy to tackle a first draft!

Also sorry for the silence was away for a bit!

[pront]

Im not a rust dev by any stretch, but Im happy to tackle a first draft!

This is great. The interface ideas are language agnostic but it can get tricky to understand Vector core internals. Happy to help if needed. We can keep using this thread.