benchmark| Potential solution for performance regressions

[doublethefish]

Steps

• (na) Add yourself to CONTRIBUTORS if you are a new contributor.
• (na) Add a ChangeLog entry describing what your PR does.
• (na) If it's a new feature or an important bug fix, add a What's New entry in doc/whatsnew/<current release.rst>.
• Write a good description on what the PR does.

Description

These commits generate comparable performance data for a run of benchmarked code.

This is a potential 1st step in establishing performance regression tests. Later work would be to actually do the comparison against the target-branch somehow.

We use pytest-benchmark here, rather than asv, because pytest-benchmark was easier to integrate into the existing tests.

Use-case

A use-case for this functionality is:

1. Write a good benchmark test:

• See tests/benchmark/test_baseline_benchmarks.py for an example

2. Run tox -e benchmark

• This generates benchmark .json files in the .tox directory

3. Apply some change that you suspect makes the code faster
4. Re-run tox -e benchmark

• This generates another benchmark .json file, probably 00002-something.json

5. Compare stats:

• In .tox/ run py.test-benchmark compare 0001 0002 - the numbers are used as a glob to find files that match, use the numbers that refer to the tox -e benchmark runs you want to compare.

6. Look for statistically significant differences. You will need to use your common sense here.
7. Profit.

Baseline benchmark for pylint

The baseline benchmark in this PR attempts to benchmark basic runs:

• with no-files
• a single empty file
• lots of empty files
• a single empty checker
• many empty checkers
• a checker that mimics doing expensive work
• across workers and a single-processed

Type of Changes

	Type
✓	🔨 Refactoring (support of)
✓	📜 Docs (new performance metrics)

Related Issue

[coveralls]

Coverage remained the same at 90.449% when pulling ce02b98 on doublethefish:chore/1954/Add_performance_regressions into be5a61b on PyCQA:master.

[Pierre-Sassoulas]

This is a much needed addition to be able to make performances better. I know it's a draft, but I reviewed some things let me know what you think.

[doublethefish]

PR #3498 is a slice of this PR.

[doublethefish]

Having used this code in anger, this PR has been updated to include the most useful change and I think it should go in.

I have removed some of the noise. The files that have been removed muddied the water:

• The changes to tests/test_self.py and tests/test_functional.py were a "lets see what happens" commit and it is very debatable if those changes are useful.
  • This is because of how fast most of the tests are and the fact that we're not aiming for high-performance-computing.
  • There are performance wins to be made, especially in test_functional.py but those should be targeted with specific benchmarks.
• Whilst the changes to tests/unittest_checker_similar.py are moderately useful, they can wait until we establish that this is the right way to do benchmarking.