k8s middle-manager extension for spawning peon pods [WIP]

[kishan-tr]

Description

This PR contains implementation of a druid extension (apache#10910) that spawns peon processes from middle managers, as kubernetes pods.

Use-cases :

• This extension can help us assign less resources to middle managers as peon processes would directly be spawned in kubernetes pods.
• The first main use-case of this extension could be for compaction tasks as we use middle managers to run compaction tasks. One thing that the extension doesn't have is how do we control where the pods are spawned i.e on which nodes are they created. This PR contains a small customization that can help users to define a pod specification for compaction tasks and let them control the constraints for creating pods on specific nodes
• The second use case of this extension can be integrating it with the dynamic task allocation feature of druid (Dynamic auto scale Kafka-Stream ingest tasks apache/druid#10524) which can help us in handling ingestion lag by automating the entire scaling process. However, this use-case might require additional research as the extension is designed for middle managers and we are not using middle managers for handling ingestion tasks.

References

• Dynamic auto scale Kafka-Stream ingest tasks apache/druid#10524
• A new Druid MiddleManager Resource Scheduling Model Based On K8s(MOK) apache/druid#10824
• KubernetesTaskRunner for running druid tasks as kubernetes jobs apache/druid#8801

---

Key changed/added classes in this PR

• K8sForkingTaskRunner.java
• DefaultK8sApiClient.java
• CliMiddleManager.java
• CliPeon.java

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}