Add encrypted scratch space for container sandbox

[mkulke]

fixes #1921

This is a PoC to address the problem of secure scratch space in which we can download and unpack images to the sandbox, write files at runtime etc.

Currently this is either backed by storage that is not protected by the TEE (packer) or as tmpfs mounts that will consume ram during runtime (mkosi). For larger images like pytorch w/ cuda this implies in theory that we have to set aside 8GB of ram just to be able to download and unpack the image. tmpfs mounts are by default limited to a ratio of the ram, so we might end up having to overprovision the machine to be able to run a workload.

The proposes change introduces a CAA-wide parameter that allows to specify the disk size. A cloud provider can use this as an indicator of the size of the to-be-provisioned podvm disk. A zero value means that we stick to the default size of the image.

if a size has been given (the mkosi image is ~0.5gb) it will extend the image to that size. At launch the podvm will claim that space and create an ad-hoc encrypted volume on it. Depending on the presence of a marker file in user-data the agent unit will mount that to /run/kata-containers prior to launching kata-agent.

I'm not super happy about the API (I would prefer to specify it per pod) and the implementation (the marker file) but I wanted to push early to get feedback. (also, it's only implemented for azure at the moment)

FWIW it does what it's supposed to do, see some numbers on #1958

[bpradipt]

if a size has been given (the mkosi image is ~0.5gb) it will extend the image to that size. At launch the podvm will claim that space and create an ad-hoc encrypted volume on it. Depending on the presence of a marker file in user-data the agent unit will mount that to /run/kata-containers prior to launching kata-agent.

Would it also make sense to extend this approach to use add-on disk like available with some instance types ? Not needed for initial iteration, but is this something we should consider and lay the foundation now?

I'm not super happy about the API (I would prefer to specify it per pod) and the implementation (the marker file) but I wanted to push early to get feedback. (also, it's only implemented for azure at the moment)

I think it would be better to also add an annotation (eg io.katacontainers.config.hypervisor.root_disk_size) in kata for remote hypervisor to use for per-pod scratch space. Similar to what we use for other such annotations.

[mkulke]

Would it also make sense to extend this approach to use add-on disk like available with some instance types ? Not needed for initial iteration, but is this something we should consider and lay the foundation now?

for future extensibility, I think we can freely iterate. We don't currently have a version contract between PodVM and CAA. We would have to look at the user-facing api maybe to make this possible.

I think it would be better to also add an annotation (eg io.katacontainers.config.hypervisor.root_disk_size) in kata for remote hypervisor to use for per-pod scratch space. Similar to what we use for other such annotations.

I agree, I didn't spot any existing fitting annotation in kata at first glance, so we might have to add it.

[bpradipt]

I think it would be better to also add an annotation (eg io.katacontainers.config.hypervisor.root_disk_size) in kata for remote hypervisor to use for per-pod scratch space. Similar to what we use for other such annotations.

I agree, I didn't spot any existing fitting annotation in kata at first glance, so we might have to add it.

Yes, a new annotation will need to be added.

[mkulke]

Yes, a new annotation will need to be added.

We discussed it in the last community call and concluded to start with a caa-wide option. once this is merged, we can start add something to kata-runtime to make it tweakable per-pod

[stevenhorsman]

I think commits 1 & 2 need some tidying as there is new code added in 1 that is changed in 2 which was a bit confusing. Additionally it would have been nicer (for me to review) to split the resource refactor out from commit 1, into a previous commit. It's not a blocker, maybe adding a sentence to explain that in the commit message #1 would be helpful for future people looking back at this?

[stevenhorsman]

Is there any significance to +100M, or is it just a starter amount of space?

[mkulke]

it needs to have a bit of size, so repart will not stumble over it at launch, attempting to create a partition without space. not sure if needs to be +100m, could just also work with +10m, have to test.

[stevenhorsman]

Ok, I don't have an issue with the 100M, just wanted to check if there was special logic behind it.

[stevenhorsman]

This is only being done on the x86 path here as only Azure is supporting this scratch space at the moment I assume? Do you see any obstacles in the code for s390x support if/when needed?

[mkulke]

I don't know much about partitioning schemes on s390x. if systemd-repart works in the same way, it should work ootb. The cloud-provider code would need to consider the .storage property when provisioning a VM and then repart would grab whatever it available to allocate it to the scratch space.

[stevenhorsman]

This looks like a good starting point to me. Thanks @mkulke!

[bpradipt]

/lgtm