Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt fetch entry from Redis on update_from_dict (#151) #281

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kchamz
Copy link

@kchamz kchamz commented Nov 20, 2024

As mentioned in this issue , upon restart, redbeat is not able to pick up the persisted last_run_at (in Redis), but instead is defaulting to the current datetime for last_run_at.

As a result, tasks end up being scheduled inconsistently between restarts. In larger systems where restarts could be more frequent, this behaviour be impactful.

This PR adds logic to that attempts to fetch an entry from Redis first, using its generated Redis key.
If the key is not found in Redis, only then we can proceed to create a new entry.

Before fix (overdue task):

LocalTime -> 2024-11-20 15:05:26
Configuration ->
    . broker -> amqp://guest:**@localhost:5672//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> redbeat.schedulers.RedBeatScheduler
       . redis -> redis://localhost/
       . lock -> `st:redbeat:lock` 30.00 seconds (30s)
    . logfile -> [stderr]@%WARNING
    . maxinterval -> 5.00 seconds (5s)
{"logger":"celery.beat","level":"info","timestamp":"2024-11-20T20:05:26.209759Z","message":"beat: Starting...","dd.env":"","dd.service":"","dd.version":""}
{"logger":"celery.beat","level":"info","timestamp":"2024-11-20T20:05:26.227691Z","message":"beat: Acquired lock","dd.env":"","dd.service":"","dd.version":""}
{"logger":"celery.beat","level":"info","timestamp":"2024-11-20T20:05:51.297721Z","message":"Scheduler: Sending due task do_something (src.tasks.do_something)","dd.env":"","dd.service":"","dd.version":""}

We can observe the task was sent only 25 seconds after the restart of redbeat although it was long overdue.
(The delta is 25 seconds)

After fix (overdue task):

LocalTime -> 2024-11-20 15:07:48
Configuration ->
    . broker -> amqp://guest:**@localhost:5672//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> redbeat.schedulers.RedBeatScheduler
       . redis -> redis://localhost/
       . lock -> `st:redbeat:lock` 30.00 seconds (30s)
    . logfile -> [stderr]@%WARNING
    . maxinterval -> 5.00 seconds (5s)
{"logger":"celery.beat","level":"info","timestamp":"2024-11-20T20:07:48.612525Z","message":"beat: Acquired lock","dd.env":"","dd.service":"","dd.version":""}
{"logger":"celery.beat","level":"info","timestamp":"2024-11-20T20:07:48.620173Z","message":"Scheduler: Sending due task do_something (src.tasks.do_something)","dd.env":"","dd.service":"","dd.version":""}

We can observe that the task was sent as soon as redbeat was restarted since it was long overdue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant