How to start/expose the metrics end point of the Triton Server via openai_frontend/main.py arguments

[shuknk8s]

Could not find an argument similar to --enable-kserve-frontends (which enables exposing Triton Server Kfserve http and grpc end points) for enabling/exposing the metrics end point in openai_frontend/main.py args.

[shuknk8s]

argument for metrics server enabling is missing in the method below in main.py

def parse_args():
parser = argparse.ArgumentParser(
description="Triton Inference Server with OpenAI-Compatible RESTful API server."
)

# Triton Inference Server
triton_group = parser.add_argument_group("Triton Inference Server")
triton_group.add_argument(
    "--model-repository",
    type=str,
    required=True,
    help="Path to the Triton model repository holding the models to be served",
)
triton_group.add_argument(
    "--tokenizer",
    type=str,
    default=None,
    help="HuggingFace ID or local folder path of the Tokenizer to use for chat templates",
)
triton_group.add_argument(
    "--backend",
    type=str,
    default=None,
    choices=["vllm", "tensorrtllm"],
    help="Manual override of Triton backend request format (inputs/output names) to use for inference",
)
triton_group.add_argument(
    "--tritonserver-log-verbose-level",
    type=int,
    default=0,
    help="The tritonserver log verbosity level",
)
triton_group.add_argument(
    "--host",
    type=str,
    default="0.0.0.0",
    help="Address/host of frontends (default: '0.0.0.0')",
)

# OpenAI-Compatible Frontend (FastAPI)
openai_group = parser.add_argument_group("Triton OpenAI-Compatible Frontend")
openai_group.add_argument(
    "--openai-port", type=int, default=9000, help="OpenAI HTTP port (default: 9000)"
)
openai_group.add_argument(
    "--uvicorn-log-level",
    type=str,
    default="info",
    choices=["debug", "info", "warning", "error", "critical", "trace"],
    help="log level for uvicorn",
)

# KServe Predict v2 Frontend
kserve_group = parser.add_argument_group("Triton KServe Frontend")
kserve_group.add_argument(
    "--enable-kserve-frontends",
    action="store_true",
    help="Enable KServe Predict v2 HTTP/GRPC frontends (disabled by default)",
)
kserve_group.add_argument(
    "--kserve-http-port",
    type=int,
    default=8000,
    help="KServe Predict v2 HTTP port (default: 8000)",
)
kserve_group.add_argument(
    "--kserve-grpc-port",
    type=int,
    default=8001,
    help="KServe Predict v2 GRPC port (default: 8001)",
)

return parser.parse_args()

[rmccorm4]

Hi @shuknk8s, for the OpenAI code you linked, you should be able to see the Triton Core metrics published at localhost:{openai_port}/metrics, for example: localhost:9000/metrics.

server/python/openai/openai_frontend/frontend/fastapi/routers/observability.py

Lines 33 to 35 in 153ba03

	@router.get("/metrics", response_class=PlainTextResponse, tags=["Utilities"])
	def metrics(request: Request) -> PlainTextResponse:
	return request.app.engine.metrics()

We could also consider exposing the Triton Core metrics through the --enable-kserve-frontends path at localhost:{kserve_metrics_port}/metrics, such as localhost:8002/metrics - but that isn't there today since the FastAPI version essentially published the same metrics natively.

[shuknk8s]

Thank you so much for your kind attention and prompt response. It would be great if you can provide a command line argument to main.py to enable metrics server explicitly with option to specify port so that we can use it deployment manifest to facilitate scaling using hpa.

one side question if you don't mind attending to : why a very small number like 16 -20 is selected for max token while inferencing if we specify max token to None in client request , by default vllm sets this number to model context length but open ai front end or triton server is overriding the max token value , is there a way to specify the value while using open ai front end and also I am unable to set triton environment variables to modify triton server behavior while using open ai front end.