Fix limiter token tracking (again)

[DrJosh9000]

What

Fixes a regression in limiter job counting.

Use k8s Informer more effectively to count jobs-in-flight accurately.

Related to #302 (but probably not a fix for it).

Why

In the pre ~v0.16 paradigm, the limiter had two roles: deduplicate jobs, and enforce the max-in-flight limit. This made sense because to deduplicate jobs required tracking the running jobs in a map, and the map could count running jobs. Later, I changed the mechanism for waking goroutines waiting for jobs to finish from a sync.Cond to a channel ("token bucket"), so that waiting for capacity could be cancelled via context.

In v0.19 I split the old limiter into the new limiter and deduper to make it easier to understand. But I didn't entirely finish the job: the deduping map had the effect of preventing multiple-counting tokens, and because the new limiter didn't have a deduping map, it can easily double-count starting and finishing jobs.

To count jobs correctly we need to:

• take a token only when a job is started
• return a token only when a job is finished

Taking tokens in OnAdd after the cache sync is finished is double-counting, because we're already taking tokens in Handle. Similarly, returning tokens in OnDelete is probably wrong except in the case where the Informer has missed some updates from not-finished to finished. Finally, taking or returning tokens in OnUpdate should only happen when a job changes state between not-finished and finished, not whenever it gets called.

Testing

I manually tested locally on Orbstack with a pipeline that generates 100 trivial jobs, configured with checkout skipped and default MaxInFlight (25).

Time spent running jobs (not including waiting for agent):

• v0.18.0: 46s
• v0.19.2: 4m57s
• v0.19.2-4-g878ddf3: 52s

[wolfeidau]

👍🏻 🚀