How does the job cache and polling relate to each other?

[aried3r]

Hey!

We're currently reading through the documentation again, the code and also some closed issues to better understand how GoodJob works and what to expect.

For us, the documentation doesn't make it perfectly clear how the job cache and polling work together. Thanks to the super interesting discussion and this comment here: #90 (comment) we think that the following happens:

• Once a job is scheduled using .perform_later without a delay, it's being saved to the database and added to the job cache and ultimately executed. Or does .perform_later without a delay circumvent the cache?
• Up to 10,000 jobs (default setting) can remain in the cache.
  • We assume that every poll_interval seconds the cache is being "filled up". Does this mean the cache is the sole queue or, if 20,000 jobs are waiting when the poller queries the DB, what happens to the 20_000 - cache.size jobs?
  • Since we have a very low workload of fewer than 1,000 jobs per day, can we safely increase the polling interval to a minute, 10 minutes, an hour,... without potential pitfalls, assuming we only use .perform_later without delay? We'd assume in case of a restart, and thus a cold start, the cache is being filled again since jobs are being persisted in the DB and will GoodJob will happily keep working in :async mode without polling often, since we stay under the maximum cache size.
  • If there is a delay (future scheduled jobs), we would need regular polling, as per Why poll when using LISTEN/NOTIFY? #90 (comment). Is that correct? Maybe this also relates to Cron-like jobs not always executed, possible reasons? #359?

Thank you in advance!

[bensheldon]

Fantastic question! Let me try to explain this cohesively, and then please poke holes in it.

Every perform_later job gets written to the database, regardless of whether it's now or wait/wait_until. (fyi, perform_now always sidesteps the Adapter and runs directly by ActiveJob, that's an ActiveJob thing and commonly trips people up). Only jobs that have await/wait_until will go into the job cache.

The job cache lives at the thread pool executor level, which is a GoodJob::Scheduler. If you're using multiple pools with a semicolon (GOOD_JOB_QUEUES="mice:2;elephants:3"), your process has multiple Schedulers, and thus multiple job caches.

The job cache isn't really a cache of specific jobs, but is a pile of Concurrent::ScheduledTasks (e.g. futures) that are created when a scheduled job is enqueued (either directly in the process if running Async, or via the Notifier listen/notify, or initially when a Scheduler boots up and queries for all the future scheduled jobs in its queues) that only track the scheduled_at time. When those futures run in the future at their scheduled time, they simply poke the Scheduler to say "there might exist a runnable job right now, you should check". The Scheduler will then check for a runnable job, if it finds one, the Scheduler will run it, and continue running jobs until there are no more runnable jobs.

Polling does not fill the job cache. Polling just pokes the scheduler(s) with the same "there might exist a runnable job right now, you should check". The Poller does not have knowledge of the database, it simply pokes the Scheduler.

Cron is isolated from Schedulers, and enqueues unscheduled jobs at regular intervals, which (as part of enqueue) pokes the Scheduler(s) with the same "there might exist a runnable job right now, you should check".

Which I think leads to the important question: how to size the job cache and how frequently to poll?

• If the maximum number of scheduled jobs (e.g. with wait/wait_until) you will ever have enqueued (in the database) is less than the size of the job cache, you (technically/perfectly) do not need to poll.
• if you have more scheduled jobs than the job cache, again, it depends:
  • if the application is pretty regularly enqueuing jobs jobs to run immediately, your schedulers will run through future scheduled jobs too without much delay, and you do not need to poll.
  • if the application is not regularly active (e.g. it's only used for some backend batch process that runs on Sunday night and queues up a million jobs scheduled regularly throughout the week), you should poll at an interval that is the maximum execution latency you will accept.

Practically: I think the Production environment should always poll as a belt-and-suspenders at an interval that is the maximum scheduled latency you will accept.

BTW, if you are using rescue_from MyError in your jobs (or you're using GoodJob::ActiveJobExtensions::Concurrency which uses rescue_from), you're likely having ActiveJob retry jobs with an incremental backoff, which uses wait to retry those jobs in the future. For bullet-proof production environment, I think it's risky to assume that jobs won't ever be scheduled in the future.

[aried3r]

Thank you for the extensive answer! I do have a few questions that I'll try to answer some myself here.

(fyi, perform_now always sidesteps the Adapter and runs directly by ActiveJob, that's an ActiveJob thing and commonly trips people up)

I agree, the documentation around .perform_now seems lacking. The guide mentions it once (when talking about enqueuing) and only in the API docs it is said, that it does not go into the queue.

We – currently – don't have a single job that uses wait/wait_until, only regular .perform_later jobs. So these will never go into the cache if I understood correctly.

Cron is isolated from Schedulers, and enqueues unscheduled jobs at regular intervals, which (as part of enqueue) pokes the Scheduler(s) with the same "there might exist a runnable job right now, you should check".

Is that regular interval the poll interval?

Enqueuing here seems to mean, that a Concurrent::ScheduledTask with a delay is created, not a job is enqueued with a wait/wait_until. Meaning cron schedules are also bypassing the cache, correct?

[bensheldon]

We – currently – don't have a single job that uses wait/wait_until, only regular .perform_later jobs. So these will never go into the cache if I understood correctly.

Correct. The Scheduler(s') threads will be woken immediately and will pick up those jobs from the database.

I'm trying to work out how to explain the current (as of v2.1) GoodJob thread strategy here, so sorry if it's confusing..

When I write "pokes the Scheduler", I mean that the object (Poller, Notifier, Adapter after enqueue in :async mode) is calling Scheduler#create_thread(state). To explain...

When a job is enqueued, the full set of job data always gets written to the database. There is a separate set of metadata that is also generated: state = {queue_name, scheduled_at}. This piece of metadata is then blasted out to all schedulers locally (if running :async) or remotely via the Notifier (LISTEN/NOTIFY).

. The Scheduler then determines what to do based on the state metadata.

• If the queue_name doesn't match the Scheduler's configuration, it does nothing.
• If the scheduled_at is nil or in the past, the Scheduler will create a Concurrent::ScheduledTask on its threadpool with a delay of 0 (e.g. run immediately) to fetch the next runnable job from the database (JobPerformer#next). The job that is fetched may not be the exact same job that was enqueued, or it may not be any job because other schedulers already fetched/locked all the runnable jobs. If the threadpool is already full of active threads, the ScheduledTask is discarded (this is fine because when one of these ScheduledTask finishes, it will automatically create another ScheduledTask if the previous scheduledTask found a runnable job... i.e. it will always check for exactly one more job.)
• if scheduled_at is in the future, the Scheduler will create that same Concurrent::ScheduledTask, this time with a delay, and put it in its "cache" which is a Concurrent::TimerSet (a bounded collection of ScheduledTasks). If the cache gets full, subsequent ScheduledTasks are discarded.

This is all basically a set of hackery around how stateless Concurrent::ScheduledTasks interact with Concurrent::ThreadPoolExecutors and Concurrent::TimerSets.

Is that regular interval the poll interval?

No, the CronManager uses its own Concurrent::ScheduledTask to create a scheduled interval (e.g. it schedules a task for the future, and when that task runs the task schedules itself again in the future)

Meaning cron schedules are also bypassing the cache, correct?

Correct. My usage of the word "Cache" only refers to the Concurrent::TimerSet that is managed by the GoodJob::Scheduler. (Aside: CronManager does use a Concurrent::ScheduledTask which implicitly uses Concurrent-Ruby's global TimerSet and global ThreadPoolExecutor, but those globals are not managed by GoodJob and are unbounded afaik).

I recognize this is confusing (I get confused myself) because GoodJob's high-level objects (Scheduler, Poller, Notifier, CronManager) all use the same kinds of ConcurrentRuby objects but in different configurations to produce different behavior)