Skip to content

Commit

Permalink
Add benchmarking scripts
Browse files Browse the repository at this point in the history
useragents.txt sample file kindly provided by @DailyMats out of
DailyMotion's data (2023-04-26).

The provided scripts allow:

- Testing the cache hit rate of various cache configuration (algorithm
  and size) on sample files, this script uses a dummy parser and is
  thus extremely fast.
- Benchmarking the average entry processing of various parser
  configurations (base parser + cache algoritm + cache size) on sample
  files, this is a much slower script but provides a a realistic
  evaluation, and allows using custom rules (`regexes.yaml` files) to
  check their impact on the performance of a given base parser.

Also added a script for testing threaded parsing, as expected this
gets 0 gain over the normal stuff because of the GIL (and re2
seemingly doesn't release the GIL either, though I don't know how
beneficial it would be at ~30us per call).

May be more useful with 3.13, or possibly with a regex-based extension
releasing the GIL, at least the basis for testing things out will be
here.
  • Loading branch information
masklinn committed Feb 11, 2024
1 parent e719a7e commit 9960dbd
Show file tree
Hide file tree
Showing 5 changed files with 75,591 additions and 1 deletion.
Loading

0 comments on commit 9960dbd

Please sign in to comment.