index.jl

#=

In this tutorial we illustrate the main uses of `SolverBenchmark`.

First, let's create fake data. It is imperative that the data for each solver be stored
in `DataFrame`s, and the collection of different solver must be stored in a dictionary of
`Symbol` to `DataFrame`.

In our examples we'll use the following data.

=#
using DataFrames, Printf, Random

Random.seed!(0)

n = 10
names = [:alpha, :beta, :gamma]
stats = Dict(name => DataFrame(:id => 1:n,
         :name => [@sprintf("prob%03d", i) for i = 1:n],
         :status => map(x -> x < 0.75 ? :first_order : :failure, rand(n)),
         :f => randn(n),
         :t => 1e-3 .+ rand(n) * 1000,
         :iter => rand(10:10:100, n),
         :irrelevant => randn(n)) for name in names)
#=

The data consists of a (fake) run of three solvers `alpha`, `beta` and `gamma`.
Each solver has a column `id`, which is necessary for joining the solvers (names
can be repeated), and columns `name`, `status`, `f`, `t` and `iter` corresponding to
problem results. There is also a column `irrelevant` with extra information that will
not be used to produce our benchmarks.

Here are the statistics of solver `alpha`:

=#
stats[:alpha]
#=

## Tables

The first thing we may want to do is produce a table for each solver. Notice that the
solver result is already a DataFrame, so there are a few options available in other
packages, as well as simply printing the DataFrame.
Our concern here is two-fold: producing publication-ready LaTeX tables, and web-ready
markdown tables.

The simplest use is `pretty_stats(io, dataframe)`.
By default, `io` is `stdout`:

=#
using SolverBenchmark

pretty_stats(stats[:alpha])
#=

Printing is LaTeX format is achieved with `pretty_latex_stats`:

=#
pretty_latex_stats(stats[:alpha])
#=

Alternatively, you can print to a file.

=#
open("alpha.tex", "w") do io
  println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
  println(io, "\\usepackage{longtable}")
  println(io, "\\begin{document}")
  pretty_latex_stats(io, stats[:alpha])
  println(io, "\\end{document}")
end
#=

=#
run(`latexmk -quiet -pdf alpha.tex`)
run(`pdf2svg alpha.pdf alpha.svg`)
#=

![](alpha.svg)

If only a subset of columns should be printed, the DataFrame should be indexed accordingly:

=#
df = stats[:alpha]
pretty_stats(df[!, [:name, :f, :t]])
#=

Markdown tables may be generated by supplying the PrettyTables `tf` keyword argument to specify the table format:

=#
pretty_stats(df[!, [:name, :f, :t]], tf=tf_markdown)
#=

All values of `tf` accepted by PrettyTables may be used in SolverBenchmark.

The `fmt_override` option overrides the formatting of a specific column.
The argument should be a dictionary of `Symbol` to format strings, where the format string will be applied to each element of the column.

The `hdr_override` changes the column headers.

=#
fmt_override = Dict(:f => "%+10.3e",
                    :t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
pretty_stats(stdout,
             df[!, [:name, :f, :t]],
             col_formatters = fmt_override,
             hdr_override = hdr_override)
#=

While `col_formatters` is for simple format strings, the PrettyTables API lets us define more elaborate formatters in the form of functions:

=#
fmt_override = Dict(:f => "%+10.3e",
                    :t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
pretty_stats(df[!, [:name, :f, :t]],
             col_formatters = fmt_override,
             hdr_override = hdr_override,
             formatters = (v, i, j) -> begin
               if j == 3  # t is the 3rd column
                 vi = floor(Int, v)
                 minutes = div(vi, 60)
                 seconds = vi % 60
                 micros = round(Int, 1e6 * (v - vi))
                 @sprintf("%2dm %02ds %06dμs", minutes, seconds, micros)
               else
                 v
               end
             end)
#=

See the [PrettyTables.jl documentation](https://ronisbr.github.io/PrettyTables.jl/stable/man/formatters/) for more information.

When using LaTeX format, the output must be understood by LaTeX.
By default, numerical data in the table is wrapped in inline math environments.
But those math environments would interfere with our formatting of the time.
Thus we must first disable them for the `time` column using `col_formatters`, and then apply the PrettyTables formatter as above:

=#
fmt_override = Dict(:f => "%+10.3e",
                    :t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
open("alpha2.tex", "w") do io
  println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
  println(io, "\\usepackage{longtable}")
  println(io, "\\begin{document}")
  pretty_latex_stats(io,
                    df[!, [:name, :status, :f, :t, :iter]],
                    col_formatters = Dict(:t => "%f"),  # disable default formatting of t
                    formatters = (v,i,j) -> begin
                      if j == 4
                        xi = floor(Int, v)
                        minutes = div(xi, 60)
                        seconds = xi % 60
                        micros = round(Int, 1e6 * (v - xi))
                        @sprintf("\\(%2d\\)m \\(%02d\\)s \\(%06d \\mu\\)s", minutes, seconds, micros)
                      else
                        v
                      end
                  end)
  println(io, "\\end{document}")
end
#=

=#
run(`latexmk -quiet -pdf alpha2.tex`)
run(`pdf2svg alpha2.pdf alpha2.svg`)
#=

![](alpha2.svg)

### Joining tables

In some occasions, instead of/in addition to showing individual results, we show
a table with the result of multiple solvers.

=#
df = join(stats, [:f, :t])
pretty_stats(stdout, df)
#=

The column `:id` is used as guide on where to join. In addition, we may have
repeated columns between the solvers. We convery that information with argument `invariant_cols`.

=#
df = join(stats, [:f, :t], invariant_cols=[:name])
pretty_stats(stdout, df)
#=

`join` also accepts `hdr_override` for changing the column name before appending
`_solver`.

=#
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
df = join(stats, [:f, :t], invariant_cols=[:name], hdr_override=hdr_override)
pretty_stats(stdout, df)
#=

=#
hdr_override = Dict(:name => "Name", :f => "\\(f(x)\\)", :t => "Time")
df = join(stats, [:f, :t], invariant_cols=[:name], hdr_override=hdr_override)
open("alpha3.tex", "w") do io
  println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
  println(io, "\\usepackage{longtable}")
  println(io, "\\begin{document}")
  pretty_latex_stats(io, df)
  println(io, "\\end{document}")
end
#=

=#
run(`latexmk -quiet -pdf alpha3.tex`)
run(`pdf2svg alpha3.pdf alpha3.svg`)
#=

![](alpha3.svg)

## Profiles

Performance profiles are a comparison tool developed by [Dolan and
Moré, 2002](https://link.springer.com/article/10.1007/s101070100263/) that takes into
account the relative performance of a solver and whether it has achieved convergence for each
problem. `SolverBenchmark.jl` uses
[BenchmarkProfiles.jl](https://github.com/JuliaSmoothOptimizers/BenchmarkProfiles.jl)
for generating performance profiles from the dictionary of `DataFrame`s.

The basic usage is `performance_profile(stats, cost)`, where `cost` is a function
applied to a `DataFrame` and returning a vector.

=#
# Running on setup to avoid warnings
using Plots
pyplot()

p = performance_profile(stats, df -> df.t)
Plots.svg(p, "profile1")
#=

#=
using Plots
pyplot()

p = performance_profile(stats, df -> df.t)
Plots.svg(p, "profile1")
#=

![](profile1.svg)

Notice that we used `df -> df.t` which corresponds to the column `:t` of the
`DataFrame`s.
This does not take into account that the solvers have failed for a few problems
(according to column :status). The next profile takes that into account.

=#
cost(df) = (df.status .!= :first_order) * Inf + df.t
p = performance_profile(stats, cost)
Plots.svg(p, "profile2")
#=

#=
cost(df) = (df.status .!= :first_order) * Inf + df.t
p = performance_profile(stats, cost)
Plots.svg(p, "profile2")
#=

![](profile2.svg)

### Profile wall

Another profile function is `profile_solvers`, which creates a wall of performance
profiles, accepting multiple costs and doing 1 vs 1 comparisons in addition to the
traditional performance profile.

=#
solved(df) = (df.status .== :first_order)
costs = [df -> .!solved(df) * Inf + df.t, df -> .!solved(df) * Inf + df.iter]
costnames = ["Time", "Iterations"]
p = profile_solvers(stats, costs, costnames)
Plots.svg(p, "profile3")
#=

#=
solved(df) = (df.status .== :first_order)
costs = [df -> .!solved(df) * Inf + df.t, df -> .!solved(df) * Inf + df.iter]
costnames = ["Time", "Iterations"]
p = profile_solvers(stats, costs, costnames)
Plots.svg(p, "profile3")
#=

![](profile3.svg)

### Example of benchmark running 
Here is a useful tutorial on how to use the benchmark with specific solver:
[Run a benchmark with OptimizationProblems](https://juliasmoothoptimizers.github.io/OptimizationProblems.jl/dev/benchmark/)
The tutorial covers how to use the problems from `OptimizationProblems` to run a benchmark for unconstrained optimization.

=#