[flytekit] Polish - Resources

[wild-endeavor]

Resources

This is a series of tickets to improve the flytekit authoring experience. If any changes are not possible to make in a backwards-compatible way, split it out into a separate ticket.

Rename argument

The argument ephemeral_storage should be renamed to disk. disk is succinct and gets the point across. Make it super clear if not already in user facing notes that the disk is ephemeral.

SHM

Should we add shm as an option to Resources? (i think the same issue that kept us from putting accelerator - the fact that resources are specified as both requests and limits - may be relevant here. Even if k8s doesn't allow it today, does it make sense to for an shm to have two different numbers here? Discuss more and probably cut a separate ticket for this (cc @granthamtaylor)

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[thomasjpfan]

I agree with changing ephemeral_storage to disk.

When adding shared memory with a pod template, it think it just sets the limit and it is not a request. I think most users just leaves it as default, which sets the limit as the total number of ram. Reference

A size limit can be specified for the default medium, which limits the capacity of the emptyDir volume. The storage is allocated from node ephemeral storage. If that is filled up from another source (for example, log files or image overlays), the emptyDir may run out of capacity before this limit. If no size is specified, memory backed volumes are sized to node allocatable memory.

This is what we do with the pytorch plugin: https://github.com/flyteorg/flytekit/blob/4e93e36843b8f13f06eb088e1a46232ad1fb225d/plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/pod_template.py#L6. So we can put shm into Resource, but it can only be used with limit.

[JiangJiaWei1103]

#take

[JiangJiaWei1103]

Hi @wild-endeavor,

Would it be a good idea to support both ephemeral_storage and disk for now, prioritize disk as the preferred argument, and warn users about the future deprecation of ephemeral_storage? I think this approach could help maintain backward compatibility while encouraging users to a smooth transition.
The question above is clarified by Thomas' response here. Thanks!

Regarding shm, just to confirm, we’re not planning to address it in this ticket—correct?

cc @thomasjpfan @granthamtaylor

Thanks so much!