feat: expose nodeclaim disruption through new disruption condition, improves pod eviction event message

[cnmcavoy]

Fixes #N/A

Description

Add's a new nodeclaim condition DisruptionCandidate which is set when a nodeclaim is being disrupted, and is applied after the disruption taint is set. The DisruptionCandidate nodeclaim condition contains the reason why the nodeclaim is being terminated (e.g node worker-mgn6n/ip-10-115-200-242.us-east-2.compute.internal was single node consolidated).

The motivation for this new nodeclaim condition is so that when evicting pods, we can look up this condition and use the condition's message in the pod event.

Example of what the pod events look like now from testing in our clusters:

keda                                          85s         Normal    Evicted                           pod/keda-admission-webhooks-5bd6b554ff-tn25h                                                Evicted pod: node worker-qa-czn28/ip-10-115-195-50.us-east-2.compute.internal drifted
keda                                          86s         Normal    Evicted                           pod/keda-operator-688bc9b887-7t4gd                                                          Evicted pod: node worker-qa-czn28/ip-10-115-195-50.us-east-2.compute.internal drifted

How was this change tested?

Built Karpenter with this change locally and tested in our clusters. Also make presubmit

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cnmcavoy
Once this PR has been reviewed and has the lgtm label, please assign njtran for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 9718717778

Details

• 32 of 54 (59.26%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.1%) to 78.704%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/operator/operator.go	0	3	0.0%
pkg/controllers/disruption/controller.go	14	23	60.87%
pkg/controllers/node/termination/terminator/eviction.go	16	26	61.54%

Totals
Change from base Build 9718332603:	-0.1%
Covered Lines:	8622
Relevant Lines:	10955

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9750492374

Details

• 32 of 54 (59.26%) changed or added relevant lines in 4 files are covered.
• 4 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.1%) to 78.684%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/operator/operator.go	0	3	0.0%
pkg/controllers/disruption/controller.go	14	23	60.87%
pkg/controllers/node/termination/terminator/eviction.go	16	26	61.54%

Files with Coverage Reduction	New Missed Lines	%
pkg/test/expectations/expectations.go	2	93.69%
pkg/utils/atomic/lazy.go	2	87.88%

Totals
Change from base Build 9748897421:	-0.1%
Covered Lines:	8619
Relevant Lines:	10954

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 9781364600

Details

• 39 of 57 (68.42%) changed or added relevant lines in 9 files are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.05%) to 78.695%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/operator/operator.go	0	3	0.0%
pkg/controllers/disruption/controller.go	6	11	54.55%
pkg/controllers/node/termination/terminator/eviction.go	16	26	61.54%

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/provisioning/scheduling/nodeclaim.go	2	89.13%

Totals
Change from base Build 9772878439:	-0.05%
Covered Lines:	8636
Relevant Lines:	10974

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 11635491539

Details

• 60 of 76 (78.95%) changed or added relevant lines in 6 files are covered.
• 3 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.02%) to 80.926%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/node/termination/terminator/eviction.go	19	21	90.48%
pkg/controllers/state/statenode.go	14	18	77.78%
pkg/controllers/disruption/controller.go	23	33	69.7%

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/disruption/controller.go	1	71.86%
pkg/controllers/provisioning/scheduling/nodeclaim.go	2	89.13%

Totals
Change from base Build 11619650738:	-0.02%
Covered Lines:	8549
Relevant Lines:	10564

---

💛 - Coveralls

[cnmcavoy]

/remove-needs-rebase

[github-actions]

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

[github-actions]

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

[njtran]

Nice work!

[njtran]

Nice work, think we're getting close!

[njtran]

We probably want to check that the node doesn't have deletion timestamp set, or else we may be fighting with another controller that adds this status condition.

[njtran]

initialized should capture if the nodeclaim isn't set

Suggested change

	if !s.Initialized() || s.NodeClaim == nil {
	if !s.Initialized() {

https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/controllers/state/statenode.go#L320

[cnmcavoy]

Removing this check results in test panics - I don't think your assertions is correct. Here is Initialized():

func (in *StateNode) Initialized() bool {
	// Node is managed by Karpenter, so we can check for the Initialized label
	if in.Managed() {
		return in.Node != nil && in.Node.Labels[v1.NodeInitializedLabelKey] == "true"
	}
	// Nodes not managed by Karpenter are always considered Initialized
	return true
}

in.Managed() return false because in.NodeClaim is nil. So initialized actually returns true when the nodeclaim is nil.

[njtran]

I personally think we should just silently fail here. I'm worried about super noisy logs in the case where a nodeclaim accidentally goes away, or if there are other issues with the apiserver

[cnmcavoy]

I dislike silently failing, bc when debugging I often end up having to re-add all the logs for suppressed errors. So I compromised and moved it to V(1).