feat(storage): use keep config for trim() method in database storage

[fabpl]

Like Redis Ingest, use keep config key in database storage trim() functionality.

[timacdonald]

@fabpl, any particular reason why you want to keep less than a weeks worth of data in storage?

[timacdonald]

We probably need to introduce a custom storage key here: pulse.storage.keep

Then we need to decide if we fallback to the ingest value if the storage value is not specified.

[DamienToscano]

Hi @timacdonald, we thought about that because the pulse.ingest.trim.lottery key is already used in the case of DatabaseStorage, so why not use the pulse.ingest.trim.keep too, to customize if we want to keep the data longer ? Or as you said, create a new key in the storage config part.

pulse/src/Pulse.php

Line 310 in 419a9bf

$odds = $this->app->make('config')->get('pulse.ingest.trim.lottery') ?? $this->app->make('config')->get('pulse.ingest.trim_lottery');

[fabpl]

@fabpl, any particular reason why you want to keep less than a weeks worth of data in storage?

Hi @timacdonald,
Perhaps I didn't quite understand the difference between redis ingest and database storage, but I think that if the duration is configured on one, it should be configured on the other?

[timacdonald]

@fabpl, not necessarily. On Forge, we want to limit the Redis ingest to 24 hours while we want to keep 30 days in the database. This ensures that if the pulse:work command goes down, we don't attempt to store 30 days worth of data in Redis and have it fall over causing issues with our incoming requests.

Can you share what specific problem you are experiencing that this PR is hoping to solve for your application?

[fabpl]

I've no specific problem.
I'm trying to understand why ingest (keep delay) are configurable while storage is not.

[timacdonald]

Gotcha. Because your ingest is often an in-memory system, like Redis, it can often fall over if the queue:work command was to go down, simply because it is restricted to memory limits.

Where as the final storage destination, a database, is designed to hold much more information - so we don't trim the storage data until it is outside the window

[DamienToscano]

Gotcha. Because your ingest is often an in-memory system, like Redis, it can often fall over if the queue:work command was to go down, simply because it is restricted to memory limits.

Where as the final storage destination, a database, is designed to hold much more information - so we don't trim the storage data until it is outside the window

That makes sense.

Thanks @timacdonald for the explanation