Single-threaded Julia benchmarks

gdalle · Jun 11, 2023 · a302d13 · a302d13
1 parent e686659
commit a302d13
Show file tree

Hide file tree

Showing 3 changed files with 7 additions and 3 deletions.
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
@@ -50,7 +50,7 @@ jobs:
           version: '1'
       - uses: julia-actions/julia-buildpkg@v1
       - run: |
-          julia --threads=auto --project=benchmark -e '
+          julia --threads=1 --project=benchmark -e '
           using Pkg
           Pkg.instantiate()
           include("benchmark/run_benchmarks.jl")'

diff --git a/benchmark/run_benchmarks.jl b/benchmark/run_benchmarks.jl
@@ -19,7 +19,8 @@ run_suite(;
     D_vals=(1, 10),
     T_vals=1000,
     K_vals=1,
-    seconds=5,
+    seconds=10,
+    samples=100,
     path=joinpath(@__DIR__, "results", "results_single_sequence.csv"),
 );
 
@@ -31,5 +32,6 @@ run_suite(;
     T_vals=100,
     K_vals=100,
     seconds=10,
+    samples=100,
     path=joinpath(@__DIR__, "results", "results_multiple_sequences.csv"),
 );
diff --git a/docs/src/benchmarks.md b/docs/src/benchmarks.md
@@ -23,7 +23,7 @@ Full benchmark logs: [`results_single_sequence.csv`](./assets/benchmark/results/
 
 ![Baum-Welch single sequence benchmark](./assets/benchmark/plots/benchmark_single_sequence_baum_welch.svg)
 
-Here pomegranate is not included because it is much slower on very small inputs.
+Here, pomegranate is not included because it is much slower on very small inputs.
 
 ### Multiple sequences
 
@@ -76,6 +76,8 @@ The packages we include have different approaches to parallelism, which can bias
 [^2]: possibly affected by `OPENBLAS_NUM_THREADS`
 [^3]: possibly affected by `MKL_NUM_THREADS`
 
+For a fairer comparison, we set `JULIA_NUM_THREADS=1`, even though it robs HMMs.jl of its parallel speedup on multiple sequences.
+
 In addition, OpenBLAS threads have [negative interactions](https://github.com/JuliaLang/julia/issues/44201#issuecomment-1585656581) with Julia threads.
 To overcome this obstacle, we run the Julia benchmarks (and only those) with `OPENBLAS_NUM_THREADS=1`.