scheduler: fixed a bug where resource calculation did not account correctly for poststart tasks

[mvegter]

Fixes a bug in the AllocatedResources.Comparable method, which resulted in
reporting less required resources than actually expected. This could result in
overscheduling of allocations on a single node and overlapping cgroup cpusets.

Split to #24304

---

job "redis2nd" {
  type = "service"
  group "cache" {
    count = 1

    task "redis-prestart" {
      lifecycle {
        hook    = "prestart"
        sidecar = false
      }
      driver = "docker"
      config {
        image = "hello-world:latest"
      }
      resources {
        cpu = 1000
      }
    }

    task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"
      }
      resources {
        cpu = 1000
      }
    }

    task "redis-start-side" {
      lifecycle {
        hook    = "poststart"
        sidecar = true
      }
      driver = "docker"
      config {
        image = "redis:3.2"
      }
      resources {
        cpu = 1000
      }
    }

    task "redis-poststop" {
      lifecycle {
        hook    = "poststop"
        sidecar = false
      }
      driver = "docker"
      config {
        image = "hello-world:latest"
      }
      resources {
        cpu = 1000
      }
    }
  }
}

Before

[sandbox@nomad-dev nomad]$ curl -s http://localhost:4646/v1/metrics | jq '.Gauges[] | select(.Name | contains("allocated.cpu")) | .Name, .Value'
"nomad.client.allocated.cpu"
1000.0
"nomad.client.unallocated.cpu"
277380.0

After

[sandbox@nomad-dev nomad]$ curl -s http://localhost:4646/v1/metrics | jq '.Gauges[] | select(.Name | contains("allocated.cpu")) | .Name, .Value'
"nomad.client.allocated.cpu"
2000.0
"nomad.client.unallocated.cpu"
276380.0

[jrasell]

Hi @mvegter and thanks for raising this PR. I've made a minor suggested change to the changelog wording, but otherwise this LGTM and once the changelog item has been resolved I'll get this merged. Thanks again!