This repository contains compacted and aggregated results of the MLPerf Inference benchmark, MLPerf Training benchmark and TinyMLPerf benchmark in the compact MLCommons Collective Mind format for the Collective Knowledge Playground being developed by the MLCommons taskforce on automation and reproducibility.
The goal is to make it easier for the community to analyze MLPerf results, add derived metrics such as performance/Watt and constraints, generate graphs, prepare reports and link reproducibility reports as shown in these examples:
- Power efficiency to compare Qualcomm, Nvidia and Sima.ai devices
- Reproducibility report for Nvidia Orin
Install MLCommons CMX framework.
Follow this README from the related CM automations script.
You can see aggregated results here.
Follow this README from the related CM automations script.
You can see aggregated results here.
Follow this README from the related CM automations script.
You can see aggregated results here.
You can use this repository to analyze, reuse, update and improve MLPerf results compact by calculating and adding derived metrics (performance/watt) or links to reproducibility reports that will be visible at the MLCommons CK playground.
Install MLCommons CMX framework.
Pull the CMX repositories with automation recipes and MLPerf results:
cmx pull repo mlcommons@ck --dir=cmx4mlops/cmx4mlops
cmx pull repo mlcommons@cm4mlperf-resultsFind CM entries with MLPerf inference v3.1 experiments from CMD:
cmx find experiment --tags=mlperf-inference,v4.1Find CM entries with MLPerf inference v3.1 experiments from Python:
import cmind
r = cmind.access({'action':'find',
'automation':'experiment,a0a2d123ef064bcb',
'tags':'mlperf-inference,v3.1'})
if r['return']>0: cmind.error(r)
lst = r['list']
for experiment in lst:
print (experiment.path)We created a sample CM script in this repository that you can use and extend to add derived metrics:
cmx run script "process mlperf-inference results" --experiment_tags=mlperf-inference,v4.12021-2025 MLCommons
Grigori Fursin and Arjun Suresh.
This project is maintained by the MLCommons taskforce on automation and reproducibility, cTuning foundation and cKnowledge.org.