You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What is your feature about ?
the buffer filter currently has a "flaw" in data management:
if buffer sources are modified after some time, the old sources (aka redis lists) will probably not be empty, this create a "memory leak" of sorts, as those lists won't probably ever be deleted from Redis, stacking up and using unnecessary memory
If your request is about an existing filter/functionality
This Feature Request is about the buffer filter
Describe what you'd like
To prevent that from happening, a minimal change needs to be done:
when the buffer starts or triggers to get all cached data from Redis, it should set/reset an expiry on the key
Thanks to that, 2 problems are solved:
if sources names change inside the buffer filter, old sources will eventually be cleared out of Redis -> no "memory leak"
if Darwin happens to have some downtime, this technique can remove obsoleted data from Redis
Additional context
The expiry value should be coherent with data longevity AND the frequency at which the buffer needs to fetch cached data, a sensible option could be to take the maximum value between twice the fetch interval, and a fixed value corresponding to data "freshness" (time before considering data is obsolete).
In all cases, the expiry value should be strictly more than the configured fetch interval, to avoid losing data before fetching it
The text was updated successfully, but these errors were encountered:
What is your feature about ?
the buffer filter currently has a "flaw" in data management:
if buffer sources are modified after some time, the old sources (aka redis lists) will probably not be empty, this create a "memory leak" of sorts, as those lists won't probably ever be deleted from Redis, stacking up and using unnecessary memory
If your request is about an existing filter/functionality
This Feature Request is about the buffer filter
Describe what you'd like
To prevent that from happening, a minimal change needs to be done:
when the buffer starts or triggers to get all cached data from Redis, it should set/reset an expiry on the key
Thanks to that, 2 problems are solved:
Additional context
The expiry value should be coherent with data longevity AND the frequency at which the buffer needs to fetch cached data, a sensible option could be to take the maximum value between twice the fetch interval, and a fixed value corresponding to data "freshness" (time before considering data is obsolete).
In all cases, the expiry value should be strictly more than the configured fetch interval, to avoid losing data before fetching it
The text was updated successfully, but these errors were encountered: