Batch inference over multiple nodes

[boyang-nlp]

I want to run multiple sglang runtimes on multiple nodes, each using only 4 GPUs, and then perform batch inference through a router. How should I proceed?

[zhaochenyang20]

https://docs.sglang.ai/router/router.html

@boyang-nlp Refer to this.