prefect integration support volcano job

[HeavenS221]

Describe the current behavior

current behavior:
work pool only support to launch kubernetes job, job template is hardcode written in src/integrations/prefect-kubernetes/prefect_kubernetes/.

Describe the proposed behavior

expect behavior:
support volcano job and kubernetes job, when work pool loading base job template['job_configuration']['job_manifest'] config, based on some key value like ['job_type'] = 'kubernetes'/ 'volcano' to decide which job will be launched.

Example Use

VolcanoJob, referred to as vcjob, is a CRD object for Volcano. Different from a Kubernetes job, it provides more advanced features such as specified scheduler, minimum number of members, task definition, lifecycle management, specific queue, and specific priority. VolcanoJob is ideal for high performance computing scenarios such as machine learning, big data applications, and scientific computing.

Our team hope to use prefect to scheduler several task flow, i.e. data processing, LLM model training, model eval, model compiling. There are several plugin functions like pytorch-plugin supported by volcano job.

Additional context

our team hope to use prefect to scheduler several task flow, i.e. data processing, LLM model training, model eval, model compiling. There are several plugin func supported by volcano job.

https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_pytorch_plugin.md

[desertaxle]

Thanks for the request @HeavenS221! I bet you could use an existing Kubernetes work pool to start volcano jobs by editing the job_configuration.job_manifest section of the base job template because the api_version is available for editing there.

Can you try updating the api_version in the base job template for a Kubernetes work pool to batch.volcano.sh/v1alpha1 and see if it works? You can change this in the 'Advanced' tab when editing a work pool in the UI.

[zzstoatzz]

fwiw @kevingrismore wrote a volcano worker a while ago

https://github.com/kevingrismore/volcano-worker/blob/main/prefect_volcano/worker.py

[HeavenS221]

@desertaxle Thanks for response. I've tried this option, but it doesn't work, there is a base job manifest validation in integrations/prefect-kubernetes/worker.py func _get_base_job_manifest()

[HeavenS221]

@zzstoatzz Thanks for info, is there any plan to accept this integration?

[desertaxle]

@HeavenS221 I'd be open to relaxing that validation to enable alternative job implementations. Do you want to submit a PR for that?

[HeavenS221]

@desertaxle sounds good to me, any developer guilds?

[desertaxle]

@HeavenS221 Yep! Here's our guide for contributing to our integrations: https://docs.prefect.io/contribute/contribute-integrations