Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Mechanism for *per-benchmark* iteration control and to clarify the purpose of cpu-mhz #135

Open
Roger-Shepherd opened this issue Apr 19, 2021 · 0 comments

Comments

@Roger-Shepherd
Copy link
Contributor

NB This does not to change the definition of a EMBENCH score.

At present the benchmarking system uses an option cpu_mhz for two (related) purposes:

  1. to define the frequency of the benchmarked CPU
  2. to multiply the baseline number of iterations performed by all benchmarks so that the execution time of those benchmarks is long enough that execution time measurements are large (seconds) compared to the quantum of time.

The baseline number of iterations has been set to provide a weighting between the benchmarks when computing the Embench score. The reported time for any benchmark is actual time / cpu_mhz. A 100 MHz processor runs 10x the number of iterations as a 10 MHz processor and this difference is accounted for by dividing the actual time accordingly.

This works well for similar processors using similar compiler optimismisations. When a processor or a compiler causes a benchmark to run very much faster (10x or even 100x) than the processor frequency would suggest, the actual run time generated by the cpu_mhz scaling becomes too short to be reliable. (Running on an M1 Mac three benchmarks run between 100x and 700x faster than frequency suggests). With the current system, the only way to overcome this is to set the cpu_mhz to be much higher (10x or 100x) than reality. This results in two problems:

  1. the run time for the entire suite is increased greatly
  2. the cpu_mhzparameter no longer reflects the actual frequency of the processor

This proposal is to add an mechanism which allows for individual benchmarks to have their iteration count increased, and the computation of their nominal run-time appropriately adjusted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant