Nomad UI shows failed for jobs that are scaled to 0

[caiodelgadonew]

Hello,

We have this behavior on nomad when we run nomad job scale <job> <group> 0 that the job shows as failed.

It would be nice to have a better message for this state instead of Failed, I know the job is not healthy since the service checks cant pass cuz its scaled to zero, but maybe a flag "Downscaled" would be a good thing to have.

Is this on track? are there any plans for it?

Example situation:

Expected:

or

[tgross]

@caiodelgadonew in recent versions of the UI we're presenting a sort of "aggregate state" based on the job status and the allocation status. Can you verify that the API is returning Running for the job status, and not Dead? (That is, what does nomad job status $myjob show for the job status?)

[caiodelgadonew]

@tgross no, its not returning running, its returning dead.
I've deployed an example to test

To reproduce, deploy any job, and then scale the groups to 0
In my example:

$ nomad job scale testing-scheduler app 0
$ nomad job scale testing-scheduler app2 0

[tgross]

Ok, thanks. I'd expect a job with all complete allocations to be "dead". We made a lot of changes to provide reasonable "aggregate" statuses in 1.8, but there are an awful lot of corner cases 😁 I'll mark this for roadmapping.

[caiodelgadonew]

thanks @tgross , you mean Running right?

[tgross]

Actually, no. 😀 The job status is a pretty coarse view of the world and so "dead" just means all allocations are terminal. I could definitely see an argument that this view of the world isn't ideal because it makes the UI design harder, but it's definitely the intended behavior right now. But the UI does want to present a richer "aggregate state" that would account for this case where the job is scaled down.

[philrenaud]

I've got a note to make this experience better in the UI. The common way this shows itself is when allocations complete and are garbage-collected, but garbage-collected allocations don't leave the UI with a lot of clues as to whether their absence is a bad thing (Failed) or an expected thing (Dormant or something).

In this case, a deliberate scale to 0 seems more detectable. I'll see what I can do to give a status less off-putting than Failed here.

[caiodelgadonew]

Any updates on this @philrenaud ? :)

[sevensolutions]

I'am also playing around with job scaling to implement a scale-to-zero-strategy and came accross this issue.
One important thing for me is, that a job, scaled down to zero, must never be removed by the garbage collector so i can re-scale it up at every time.

[philrenaud]

Hi @caiodelgadonew, I've started thinking about a solution to this problem. The current logic for status has something like

    if (failedOrLostAllocs.length >= expectedRunningAllocCount) {
      return { label: 'Failed', state: 'critical' };

Which generally works pretty well, except here, where 0 failed allocs is greater than or equal to 0 expected allocs, triggering the Failed state.

As such, in #23829 , I've started putting in a safety valve for exactly this case:

    if (expectedRunningAllocCount === 0) {
      return { label: 'Scaled Down', state: 'neutral' };
    }

This'll show something like this (3rd job shown):

Is this about what you had in mind?

========================

A secondary concern (cc @sevensolutions as well) is that these jobs are nevertheless considered Dead as far as the scheduling and garbage collection processes go (they are terminal until manual change is made to increase allocation count).

This means that these statuses are at most only temporary: "Scaled Down" is what you'd see until garbage collection takes place / a user runs nomad system gc / etc.

We have been exploring some concepts that would create garbage-collection-avoidance permanence (see Golden Job Versions) for example) that might mitigate this in the future, but I wanted to open a dialogue here to indicate the temporary nature of this status as implemented in #23829

[caiodelgadonew]

@philrenaud That "Scaled Down" looks amazing!
Imho if it could be in a different color (maybe yellow/warning?) would be 10/10

[sevensolutions]

@caiodelgadonew i also thought about a different color first, but i think "Scaled down" doesn't neccesarily need to be a warning. It may be intended.

[caiodelgadonew]

Yeah, you're correct, I like your idea. :)

Any plans for shipping it?

[philrenaud]

Yep, let me test some edge cases and get it tagged for the next minor release. Thanks for your patience with this!

[caiodelgadonew]

Amazing @philrenaud many thanks!!!