Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quality of life improvements for using benchmarks #257

Merged
merged 1 commit into from
Aug 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -135,3 +135,7 @@ dmypy.json

# Mac OS
.DS_Store

# Airspeed Velocity performance results
_results/
_html/
25 changes: 20 additions & 5 deletions docs/practices/ci_benchmarking.rst
Original file line number Diff line number Diff line change
Expand Up @@ -118,27 +118,42 @@ Running ASV locally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You may want to run ``asv`` locally, during development. Verify that it has been
properly installed on your environment by executing the following command. It runs
the benchmarking suite for your most recent commit.
properly installed on your environment by executing the following command. There
are several questions you'll be asked the first time, and you may need to instal
a new python venv. When your local environment is properly configured, it runs
the benchmarking suite for your most recent commit

.. code:: bash

>> cd benchmarks
>> asv run

Having benchmarks for several revisions, you may compare them with ease.
You will need to commit changes locally for the new code to be picked up by ASV.
Having benchmarks for several revisions, you can find them and compare them with ease.

.. code:: bash

>> asv compare revision1 revision2
>>asv show
Commits with results:

Machine : XPS8104-L
Environment: virtualenv-py3.10-Cython-build-packaging

d02787f1
5dd46d87
>> asv compare d02787f1 5dd46d87

The commands use a very flexible and powerful syntax which allows to specify a range
of commits and even tags. For more information visit ASV's
`Benchmarking section <https://asv.readthedocs.io/en/stable/using.html#benchmarking>`_.

Since you've had to create many commits while working on benchmarks, be sure
to squash before merging to main!

Writing benchmarks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Performance is measured for suites defined under ``benchmarks/benchmarks``.
Performance is measured for suites defined under ``benchmarks``.

The functions benchmarked must follow a predefined prefix.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,16 @@
},
// The directory (relative to the current directory) that benchmarks are
// stored in. If not provided, defaults to "benchmarks".
"benchmark_dir": "benchmarks",
"benchmark_dir": ".",
// The directory (relative to the current directory) to cache the Python
// environments in. If not provided, defaults to "env".
"env_dir": "env",
// The directory (relative to the current directory) that raw benchmark
// results are stored in. If not provided, defaults to "results".
"results_dir": "results",
"results_dir": "_results",
// The directory (relative to the current directory) that the html tree
// should be written to. If not provided, defaults to "html".
"html_dir": "html",
"html_dir": "_html",
// The number of characters to retain in the commit hashes.
// "hash_length": 8,
// `asv` will cache wheels of the recent builds in each
Expand Down