feat(al2023): initialize critical binaries from EBS filesystem

[cartermckinnon]

Issue #, if available:

Improves #1751.

Description of changes:

This PR adds a systemd "template unit" ([email protected]) that is used to "initialize" various critical binaries which are stored on the root EBS filesystem (nodeadm, kubelet, containerd).

EBS blocks are lazily loaded, meaning the first read of one of these binaries usually involves a delay of a handful of seconds. By accessing these files as early in the boot process as possible, this delay is not felt in hot path of nodeadm-config and nodeadm-run.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[cartermckinnon]

/ci

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.24 / al2	success ✅	success ✅
1.24 / al2023	success ✅	success ✅
1.25 / al2	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2	success ✅	failure ❌
1.26 / al2023	success ✅	success ✅
1.27 / al2	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2	success ✅	success ✅
1.30 / al2023	success ✅	success ✅
1.31 / al2	success ✅	success ✅
1.31 / al2023	success ✅	success ✅

[cartermckinnon]

Looks like we got a comparable speedup in nodeadm-run as we did in #1991 for nodeadm-config. 🥳

I've opened #2012 to get some better numbers in the log bundle.

[cartermckinnon]

Gathering systemd-analyze output...

/ci

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.24 / al2023	skipped ⏭️	skipped ⏭️
1.26 / al2023	skipped ⏭️	skipped ⏭️
1.28 / al2023	skipped ⏭️	skipped ⏭️

[cartermckinnon]

Scoping down to AL2023 only

/ci
+workflow:os_distros al2023

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.24 / al2023	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅
1.31 / al2023	success ✅	success ✅

[cartermckinnon]

Before

Expand

Expand

Expand

After

Expand

Expand

Expand

[cartermckinnon]

The new units are having the desired effect, the units execute alongside nodeadm-config and cloud-init.

containerd benefits most from this change. Before, it took 2.5-5 seconds to become active. After, it takes about 500ms, with much less variation in timing.

This allows kubelet to become active 2-4 seconds earlier after this change. I expect #2000 to shave another 1-2 seconds off this, as well as make the timing more consistent.

There is no noticeable benefit of ebs-initialize@nodeadm, because nodeadm-config becomes runnable at the same point in the lifecycle. I'll remove that one.

Similarly, the optimization added in #1991 is still useful, because nodeadm-config and ebs-initialize@kubelet run simultaneously.

[cartermckinnon]

Getting some more samples...

/ci
+workflow:os_distros al2023

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.24 / al2023	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅
1.31 / al2023	success ✅	success ✅