Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bridge backend of Model if needed #2610

Closed
wants to merge 5 commits into from
Closed

Bridge backend of Model if needed #2610

wants to merge 5 commits into from

Conversation

odow
Copy link
Member

@odow odow commented May 21, 2021

This PR addresses the "time-to-first-solve" issue (jump-dev/MathOptInterface.jl#1313). It is essentially a repeat of jump-dev/MathOptInterface.jl#1252, but at the JuMP level instead of MOI.

Closes

Closes #2520
Closes #1627

Background

The main issues are:

  • Bridges are good, and have a small run-time cost if un-used
  • Due to the design, bridges are not type stable. The main reasons are:
    • New bridges can be added by the user
    • Variable bridges can return a function of a variable, instead of just a variable
    • Determining whether a variable is bridged requires a run-time lookup.
  • Because of the type instability, bridges don't precompile well, and they have a large compile-time cost.
  • This compile time-cost happens even if the bridges are not used. As one example:
    • To set the objective, we must first delete the current objective. Because we can only tell at runtime if the current objective is bridged, inference starts exploring the branch that deals with the bridges. Objective bridges lead to the graph, which leads to other bridges, and so inference continues until we hit the type-instabilities and it gives up.
  • One attempt at solving this was Add auto_bridge to CachingOptimizer MathOptInterface.jl#1252. But this tried to squeeze the logic into CachingOptimizer, and tied CachingOptimizer and LazyBridgeOptimizer together too much.
  • A related PR is Concrete optimizer type in CachingOptimizer #2520. If JuMP's backend is not a concrete type, we don't need the CachingOptimizer to also be non-concrete.

Demo

By default, the LazyBridgeOptimizer is omitted. But, if you do something that requires bridges, they are added:

julia> model = Model(Clp.Optimizer)
A JuMP Model
Feasibility problem with:
Variables: 0
Model mode: AUTOMATIC
CachingOptimizer state: EMPTY_OPTIMIZER
Solver name: Clp

julia> backend(model)
MOIU.CachingOptimizer{Clp.Optimizer, MOIU.UniversalFallback{MOIU.Model{Float64}}}
in state EMPTY_OPTIMIZER
in mode AUTOMATIC
with model cache MOIU.UniversalFallback{MOIU.Model{Float64}}
  fallback for MOIU.Model{Float64}
with optimizer Clp.Optimizer

julia> @variable(model, x[1:4] in MOI.Nonnegatives(4))
4-element Vector{VariableRef}:
 x[1]
 x[2]
 x[3]
 x[4]

julia> backend(model)
MOIU.CachingOptimizer{MOIB.LazyBridgeOptimizer{MOIU.CachingOptimizer{Clp.Optimizer, MOIU.UniversalFallback{MOIU.Model{Float64}}}}, MOIU.UniversalFallback{MOIU.Model{Float64}}}
in state EMPTY_OPTIMIZER
in mode AUTOMATIC
with model cache MOIU.UniversalFallback{MOIU.Model{Float64}}
  fallback for MOIU.Model{Float64}
with optimizer MOIB.LazyBridgeOptimizer{MOIU.CachingOptimizer{Clp.Optimizer, MOIU.UniversalFallback{MOIU.Model{Float64}}}}
  with 0 variable bridges
  with 0 constraint bridges
  with 0 objective bridges
  with inner model MOIU.CachingOptimizer{Clp.Optimizer, MOIU.UniversalFallback{MOIU.Model{Float64}}}
    in state EMPTY_OPTIMIZER
    in mode AUTOMATIC
    with model cache MOIU.UniversalFallback{MOIU.Model{Float64}}
      fallback for MOIU.Model{Float64}
    with optimizer Clp.Optimizer

You can also pass force_bridge_formulation = true to force-add the bridges:

julia> model = Model(Clp.Optimizer; force_bridge_formulation = true)
A JuMP Model
Feasibility problem with:
Variables: 0
Model mode: AUTOMATIC
CachingOptimizer state: EMPTY_OPTIMIZER
Solver name: Clp

julia> backend(model)
MOIU.CachingOptimizer{MOIB.LazyBridgeOptimizer{MOIU.CachingOptimizer{Clp.Optimizer, MOIU.UniversalFallback{MOIU.Model{Float64}}}}, MOIU.UniversalFallback{MOIU.Model{Float64}}}
in state EMPTY_OPTIMIZER
in mode AUTOMATIC
with model cache MOIU.UniversalFallback{MOIU.Model{Float64}}
  fallback for MOIU.Model{Float64}
with optimizer MOIB.LazyBridgeOptimizer{MOIU.CachingOptimizer{Clp.Optimizer, MOIU.UniversalFallback{MOIU.Model{Float64}}}}
  with 0 variable bridges
  with 0 constraint bridges
  with 0 objective bridges
  with inner model MOIU.CachingOptimizer{Clp.Optimizer, MOIU.UniversalFallback{MOIU.Model{Float64}}}
    in state EMPTY_OPTIMIZER
    in mode AUTOMATIC
    with model cache MOIU.UniversalFallback{MOIU.Model{Float64}}
      fallback for MOIU.Model{Float64}
    with optimizer Clp.Optimizer

Benchmark

using JuMP

### No bridging

function foo(optimizer, bridge_formulation)
    model = Model(optimizer; force_bridge_formulation = force_bridge_formulation)
    set_silent(model)
    @variable(model, x >= 0)
    @constraint(model, 2x + 1 <= 1)
    @objective(model, Max, 1.0 * x)
    optimize!(model)
end

### ObjectiveFunction bridging

function foo_obj(optimizer)
    model = Model(optimizer)
    set_silent(model)
    @variable(model, x >= 0)
    @constraint(model, 2x + 1 <= 1)
    @objective(model, Max, x)
    optimize!(model)
end

### variable bridging

function foo_var(optimizer)
    model = Model(optimizer)
    set_silent(model)
    @variable(model, x[1:1] in MOI.Nonnegatives(1))
    @constraint(model, 2x[1] + 1 <= 1)
    @objective(model, Max, 1.0 * x[1])
    optimize!(model)
end

### constraint bridging

function foo_con(optimizer)
    model = Model(optimizer)
    set_silent(model)
    @variable(model, x[1:2])
    @constraint(model, x in MOI.Nonpositives(2))
    @objective(model, Max, 1.0 * x[1])
    optimize!(model)
end

import Clp
import GLPK
const optimizer = ARGS[1] == "clp" ? Clp.Optimizer : GLPK.Optimizer

if ARGS[2] == "--bridge"
    @eval @time foo(optimizer, true)
    @eval @time foo(optimizer, true)
elseif ARGS[2] == "--no-bridge"
    @eval @time foo(optimizer, false)
    @eval @time foo(optimizer, false)
elseif ARGS[2] == "--var"
    @eval @time foo_var(optimizer)
    @eval @time foo_var(optimizer)
elseif ARGS[2] == "--con"
    @eval @time foo_con(optimizer)
    @eval @time foo_con(optimizer)
else
    @assert ARGS[2] == "--obj"
    @eval @time foo_obj(optimizer)
    @eval @time foo_obj(optimizer)
end

Current master

(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --bridge
 10.509123 seconds (25.20 M allocations: 1.570 GiB, 5.64% gc time, 21.39% compilation time)
  0.000679 seconds (1.36 k allocations: 83.797 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --no-bridge
  4.220926 seconds (8.44 M allocations: 537.310 MiB, 3.65% gc time, 29.07% compilation time)
  0.000323 seconds (506 allocations: 42.453 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --var      
 11.194838 seconds (26.03 M allocations: 1.645 GiB, 5.66% gc time, 21.81% compilation time)
  0.000734 seconds (1.59 k allocations: 99.672 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --con
 11.305996 seconds (27.82 M allocations: 1.736 GiB, 5.79% gc time, 27.37% compilation time)
  0.000590 seconds (1.44 k allocations: 89.172 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --obj
 11.233307 seconds (25.30 M allocations: 1.576 GiB, 5.60% gc time, 22.66% compilation time)
  0.000542 seconds (1.37 k allocations: 84.109 KiB)

(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --bridge
  6.395862 seconds (15.22 M allocations: 921.493 MiB, 4.76% gc time, 31.53% compilation time)
  0.000251 seconds (909 allocations: 45.906 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --no-bridge
  3.740884 seconds (5.39 M allocations: 318.667 MiB, 2.13% gc time, 99.96% compilation time)
  0.000174 seconds (358 allocations: 31.203 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --var      
  8.470519 seconds (17.92 M allocations: 1.082 GiB, 5.56% gc time, 27.10% compilation time)
  0.000546 seconds (1.14 k allocations: 61.703 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --con
  8.209768 seconds (19.46 M allocations: 1.158 GiB, 4.48% gc time, 36.46% compilation time)
  0.000514 seconds (1.06 k allocations: 55.734 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --obj
  6.853098 seconds (15.37 M allocations: 931.368 MiB, 4.19% gc time, 36.77% compilation time)
  0.000288 seconds (918 allocations: 46.203 KiB)

This PR

(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --bridge
 10.589091 seconds (25.71 M allocations: 1.604 GiB, 6.78% gc time, 11.96% compilation time)
  0.000610 seconds (1.37 k allocations: 85.797 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --no-bridge
  4.787131 seconds (9.11 M allocations: 577.229 MiB, 4.02% gc time, 27.56% compilation time)
  0.000388 seconds (522 allocations: 44.484 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --var      
 12.136513 seconds (26.08 M allocations: 1.653 GiB, 5.59% gc time, 11.15% compilation time)
  0.000879 seconds (1.65 k allocations: 104.969 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --con
 11.449779 seconds (27.80 M allocations: 1.740 GiB, 6.50% gc time, 15.22% compilation time)
  0.000612 seconds (1.47 k allocations: 93.453 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl clp --obj
 10.283270 seconds (25.37 M allocations: 1.583 GiB, 6.06% gc time, 23.26% compilation time)
  0.000537 seconds (1.40 k allocations: 88.359 KiB)

(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --bridge
  6.067656 seconds (15.35 M allocations: 932.123 MiB, 5.25% gc time, 22.02% compilation time)
  0.000261 seconds (926 allocations: 47.953 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --no-bridge
  3.772164 seconds (5.49 M allocations: 324.804 MiB, 2.41% gc time, 99.95% compilation time)
  0.000162 seconds (376 allocations: 33.266 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --var      
  7.915397 seconds (18.05 M allocations: 1.093 GiB, 5.46% gc time, 15.57% compilation time)
  0.000556 seconds (1.20 k allocations: 66.953 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --con
  7.842836 seconds (19.53 M allocations: 1.165 GiB, 4.78% gc time, 21.45% compilation time)
  0.000452 seconds (1.10 k allocations: 60.031 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. test/perf/force_bridge_formulation.jl glpk --obj
  7.193380 seconds (15.38 M allocations: 933.930 MiB, 4.45% gc time, 37.75% compilation time)
  0.000415 seconds (957 allocations: 50.484 KiB)

The big change is really that its 2x as fast when bridges are not needed.

Alternative

The alternative is to change the default of bridge_constraints to false, and also fix the double abstract-type issue.

This would give everyone a speed-up if they don't use bridges, and for models which use bridges, they can just go

Model(GLPK.Optimizer, bridge_formulation = true)

This would have the added benefit of giving people greater insight into when bridges get used.

TODO

TODOs before merging

  • Better tests

src/optimizer_interface.jl Outdated Show resolved Hide resolved
@blegat
Copy link
Member

blegat commented May 21, 2021

bridge_constraint does not have the same effect as before as setting it to false won't prevent bridges to be automatically added if needed.
If were going to be breaking, we might as well change the name since it should be changed before v1.0 (bridge_constraint does not make sense given that we also bridge variables and objectives).
It should be bridge_mode with

  • AUTO_BRIDGING: add if needed
  • ADD_BRIDGING: add from the start
  • NO_BRIDGING: don't add it even if needed

@codecov
Copy link

codecov bot commented May 28, 2021

Codecov Report

Merging #2610 (e19f712) into master (b300820) will decrease coverage by 0.22%.
The diff coverage is 90.38%.

❗ Current head e19f712 differs from pull request most recent head d4d02b4. Consider uploading reports for the commit d4d02b4 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2610      +/-   ##
==========================================
- Coverage   94.18%   93.96%   -0.23%     
==========================================
  Files          43       43              
  Lines        5522     5564      +42     
==========================================
+ Hits         5201     5228      +27     
- Misses        321      336      +15     
Impacted Files Coverage Δ
src/variables.jl 97.07% <85.36%> (-1.56%) ⬇️
src/optimizer_interface.jl 79.66% <86.66%> (+4.15%) ⬆️
src/objective.jl 94.00% <92.30%> (+0.38%) ⬆️
src/JuMP.jl 82.46% <96.29%> (-2.52%) ⬇️
src/constraints.jl 95.65% <100.00%> (+0.08%) ⬆️
src/copy.jl 95.31% <100.00%> (-0.08%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b300820...d4d02b4. Read the comment docs.

@odow
Copy link
Member Author

odow commented May 28, 2021

Docs are failing because of jump-dev/MathOptInterface.jl#1329. This shows that we're copying straight to GLPK, which should give a nice speedup though!

Alternate fix here: jump-dev/GLPK.jl#183

@odow
Copy link
Member Author

odow commented Sep 27, 2021

Good news on the GLPK front, we're < 4 seconds, compared to 15 seconds in May.

(base) oscar@Oscars-MBP JuMP % ~/julia --project=. ~/Desktop/bench.jl glpk --bridge
  6.649749 seconds (15.36 M allocations: 933.000 MiB, 4.98% gc time, 21.35% compilation time)
  0.000296 seconds (926 allocations: 47.953 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. ~/Desktop/bench.jl glpk --no-bridge
  3.869593 seconds (5.49 M allocations: 324.722 MiB, 2.53% gc time, 99.96% compilation time)
  0.000163 seconds (376 allocations: 33.266 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. ~/Desktop/bench.jl glpk --var      
  7.808467 seconds (18.06 M allocations: 1.094 GiB, 5.11% gc time, 15.47% compilation time)
  0.000547 seconds (1.20 k allocations: 66.953 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. ~/Desktop/bench.jl glpk --con
  8.049347 seconds (19.54 M allocations: 1.166 GiB, 4.39% gc time, 20.44% compilation time)
  0.000466 seconds (1.10 k allocations: 60.031 KiB)
(base) oscar@Oscars-MBP JuMP % ~/julia --project=. ~/Desktop/bench.jl glpk --obj
  6.487435 seconds (15.40 M allocations: 934.871 MiB, 4.42% gc time, 36.21% compilation time)
  0.000282 seconds (957 allocations: 50.484 KiB)

@odow odow changed the title WIP: bridge backend of Model if needed Bridge backend of Model if needed Sep 29, 2021
@odow odow force-pushed the od/auto-bridge branch 2 times, most recently from 2569bf7 to 9c04228 Compare October 1, 2021 00:50
@odow
Copy link
Member Author

odow commented Oct 3, 2021

I need to rerun the benchmark because I don't think I precompiled things properly, but this is the sort of thing our new benchmarking scripts can produce:

  • time_A is the current master
  • time_B is this PR
  • time_ratio is time_B / time_A (smaller is better)
43×4 DataFrame
 Row │ models                             time_A    time_B    time_ratio 
     │ String                             Float64   Float64   Float64    
─────┼───────────────────────────────────────────────────────────────────
   1 │ getting_started_with_JuMP.jl       15.6854   12.0367     0.767382
   2 │ solvers_and_solutions.jl           14.7179   11.8145     0.80273
   3 │ diet.jl                            14.7689   12.3716     0.837675
   4 │ cannery.jl                         14.6897   12.3197     0.838661
   5 │ urban_plan.jl                      16.2603   13.6643     0.840345
   6 │ knapsack.jl                        13.3857   11.5781     0.864959
   7 │ factory_schedule.jl                17.572    15.2579     0.86831
   8 │ transp.jl                          14.4346   12.6629     0.877258
   9 │ multi.jl                           14.6595   12.9403     0.882725
  10 │ cutting_stock_column_generation.…  17.9306   15.8623     0.88465
  11 │ network_flows.jl                   15.9198   14.1367     0.887992
  12 │ variables_constraints_objective.…  20.2857   18.0255     0.888579
  13 │ sudoku.jl                          13.568    12.078      0.890187
  14 │ n-queens.jl                        13.3915   11.949      0.892282
  15 │ working_with_data_files.jl         40.6386   36.3744     0.89507
  16 │ power_systems.jl                   19.587    17.5443     0.895711
  17 │ performance_tips.jl                 7.95099   7.17228    0.902061
  18 │ steelT3.jl                         18.3476   16.5523     0.902149
  19 │ facility_location.jl               25.7838   23.3619     0.906069
  20 │ clnlbeam.jl                        24.7641   23.0942     0.932567
  21 │ nlp_tricks.jl                      23.291    21.8562     0.938394
  22 │ prod.jl                            18.1101   17.0073     0.939106
  23 │ callbacks.jl                       18.1412   17.3113     0.954257
  24 │ qcp.jl                             23.407    22.6249     0.966589
  25 │ rosenbrock.jl                      21.1796   20.7913     0.981668
  26 │ rocket_control.jl                  36.0933   35.5357     0.984552
  27 │ geographic_clustering.jl           25.5792   25.288      0.988618
  28 │ mle.jl                             23.1395   23.0961     0.998125
  29 │ an_introduction_to_julia.jl         7.65756   7.6937     1.00472
  30 │ portfolio.jl                       23.3081   23.4615     1.00658
  31 │ corr_sdp.jl                        24.8389   25.7969     1.03857
  32 │ min_ellipse.jl                     23.7421   24.7362     1.04187
  33 │ logistic_regression.jl             30.0491   31.5174     1.04886
  34 │ min_distortion.jl                  24.5027   25.7974     1.05284
  35 │ space_shuttle_reentry_trajectory…  41.9092   44.1962     1.05457
  36 │ max_cut_sdp.jl                     25.1358   26.6172     1.05894
  37 │ experiment_design.jl               27.9409   29.5918     1.05908
  38 │ tips_and_tricks.jl                 12.0369   13.018      1.08151
  39 │ finance.jl                         15.8194   17.2735     1.09192
  40 │ robust_uncertainty.jl              26.4479   29.0361     1.09786
  41 │ cluster.jl                         25.9001   28.4743     1.09939
  42 │ benders_lazy_constraints.jl        17.6766   21.025      1.18942
  43 │ benders_decomposition.jl           17.329    20.8819     1.20502

@odow odow force-pushed the od/auto-bridge branch 2 times, most recently from 22f844d to ad23255 Compare October 4, 2021 21:33
@odow odow added the Status: Needs developer call This should be discussed on a monthly developer call label Oct 7, 2021
@odow odow requested a review from blegat October 7, 2021 21:30
src/JuMP.jl Outdated Show resolved Hide resolved
src/JuMP.jl Outdated Show resolved Hide resolved
src/optimizer_interface.jl Outdated Show resolved Hide resolved
@@ -186,6 +205,16 @@ function optimize!(
"The solver does not support nonlinear problems " *
"(i.e., NLobjective and NLconstraint).",
)
elseif err isa MOI.UnsupportedConstraint
if _add_bridges_if_needed(model)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also catch unsupported objective function ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also free variables that are not supported ? You should probably try with CSDP see the error that is thrown

end

model = direct_model(
MOI.Utilities.CachingOptimizer(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, we could directly do direct_model(MOIU.UniversalFallback(MOIU.Model{Float64}())). Otherwise we will compile a lot of methods for MOIU.CachingOptimizer{...,MOI.AbstractOptimizer} but we will never use the fact that it's a caching optimizer. If we use direct_model(MOIU.UniversalFallback(MOIU.Model{Float64}())) directly, wouldn't we gain in compile time ?
It won't add more possible states. Currently the possible states are:

  1. The MOI backend is a CachingOptimizer in state NO_OPTIMIZER
  2. An optimizer is set
    and if we do the change I suggest, the possible states are
  3. The MOI backend is UniversalFallback{Model}
  4. An optimizer is set and the MOI backend is a CachingOptimizer

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work for two reasons:

  • UniversalFallback is not an AbstractOptimizer
  • We wouldn't be able to tell if the user was in direct mode

Rebase and fix formatting

bridge_formulation instead of bridge_constraints

Some test fixes

More fixes

Fix docs. Requires new GLPK release

Update models.md

Remove TODO

Another fix

More fixes

Fix formatting

Fix typo

More tests

Update solvers_and_solutions.jl

Rename bridge_formulation to force_bridge_formulation
@odow
Copy link
Member Author

odow commented Oct 14, 2021

I'm having second thoughts about this PR. Perhaps we just need to:

  • Fix the double-abstract-type issue (Concrete optimizer type in CachingOptimizer #2520)
  • Make the documentation much clearer and explicit that people should try bridge_constraints = false to get a speed boost
  • Rename bridge_constraints to add_bridges

Model(GLPK.Optimizer; add_bridges = false) isn't that bad. It's explicit in what it is doing, and it doesn't rely on this hacky trying to catch errors mechanism.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Status: Needs developer call This should be discussed on a monthly developer call Type: Performance
Development

Successfully merging this pull request may close these issues.

Bridges in Manual mode with solver not supporting default_copy_to
2 participants