Enhancing Kubeflow with Batch Processing Gateway for Efficient Multi-Cluster Spark Job Management

[Shekharrajak]

What feature you would like to be added?

Data Scientists and ML Engineers must be able to run Spark application through Kubeflow provided Python APIs in Kubeflow Notebook & pipeline. Batch Processing Gateway can help enterprise environments, managing Apache Spark jobs across multiple Kubernetes clusters presents challenges such as inefficient job distribution, performance bottlenecks, and complex workload balancing. Integrating the Batch Processing Gateway (BPG) into Kubeflow aims to address these issues by providing a centralized mechanism for submitting, monitoring, and managing Spark applications across various clusters.

Why is this needed?

By integrating Batch Processing Gateway with Kubeflow Notebooks, this project provides a cloud-native, scalable, and user-friendly solution for Spark job execution, debugging, and optimization. It will significantly improve performance, efficiency, and developer experience, enabling ML practitioners and data engineers to focus on experimentation and optimization without struggling with job management complexities.

Describe the solution you would like

No response

Describe alternatives you have considered

No response

Additional context

Since this need collaboration and dedicated contributors , timeline and resource - We can have this project in GSoC 2025 proposal.

Love this feature?

Give it a 👍 We prioritize the features with most 👍