components Benchmarking documentation

Benchmarking

Components in this category

---

• batch_benchmark_config_generator

  Generates the config for the batch score component.

• batch_benchmark_inference

  Components for batch endpoint inference

• batch_benchmark_inference_claude

  Components for batch endpoint inference

• batch_benchmark_inference_with_inference_compute

  Components for batch endpoint inference with inference compute support.

• batch_benchmark_score

• batch_inference_preparer

  Prepare the jsonl file and endpoint for batch inference component.

• batch_output_formatter

  Output Formatter for batch inference output

• batch_resource_manager

  Resource Manager for batch inference.

• benchmark_embedding_model

  Component for benchmarking an embedding model via MTEB.

• benchmark_result_aggregator

  Aggregate quality metrics, performance metrics and all of the metadata from the pipeline. Also add them to the root run.

• compute_performance_metrics

  Performs performance metric post processing using data from a model inference run.

• dataset_downloader

  Downloads the dataset onto blob store.

• dataset_preprocessor

  Dataset Preprocessor

• dataset_sampler

  Samples a dataset containing JSONL file(s).

• inference_postprocessor

  Inference Postprocessor

• prompt_crafter

  This component is used to create prompts from a given dataset. From a given jinja prompt template, it will generate prompts. It can also create few-shot prompts given a few-shot dataset and the number of shots.