WIP: Convert optimization.jl to sparse #19
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks @odow. My philosophy is correctness first, performance second. So I would like to keep this in a branch until it can match the level of correctness of the version that is currently on master. |
odow
changed the title
Ah. I forgot to add "WIP:" to the title. I'm digging into the correctness issue |
|
Working with my PR: SciML/Optimization.jl#284 (base) oscar@Oscars-MBP rosetta-opf % ~/julia-1.7 --project=. optimization.jl
[info | PowerModels]: extending matpower format with data: areas 1x3
[info | PowerModels]: removing 1 cost terms from generator 4: [4000.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 1: [1400.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 5: [1000.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 2: [1500.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 3: [3000.0, 0.0]
[info | PowerModels]: updated generator 4 cost function with order 2 to a function of order 3: [0.0, 4000.0, 0.0]
[info | PowerModels]: updated generator 1 cost function with order 2 to a function of order 3: [0.0, 1400.0, 0.0]
[info | PowerModels]: updated generator 5 cost function with order 2 to a function of order 3: [0.0, 1000.0, 0.0]
[info | PowerModels]: updated generator 2 cost function with order 2 to a function of order 3: [0.0, 1500.0, 0.0]
[info | PowerModels]: updated generator 3 cost function with order 2 to a function of order 3: [0.0, 3000.0, 0.0]
variables: 44, 44, 44
constraints: 53, 53, 53
******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit https://github.com/coin-or/Ipopt
******************************************************************************
This is Ipopt version 3.14.4, running with linear solver MUMPS 5.4.1.
Number of nonzeros in equality constraint Jacobian...: 155
Number of nonzeros in inequality constraint Jacobian.: 36
Number of nonzeros in Lagrangian Hessian.............: 240
Total number of variables............................: 44
variables with only lower bounds: 0
variables with lower and upper bounds: 39
variables with only upper bounds: 0
Total number of equality constraints.................: 35
Total number of inequality constraints...............: 18
inequality constraints with only lower bounds: 0
inequality constraints with lower and upper bounds: 6
inequality constraints with only upper bounds: 12
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
0 1.0059989e+02 3.99e+00 2.88e+01 -1.0 0.00e+00 - 0.00e+00 0.00e+00 0
1 8.3066346e+03 2.47e+00 1.01e+02 -1.0 2.78e+00 - 4.11e-03 3.82e-01h 1
2 6.7182484e+03 2.36e+00 9.62e+01 -1.0 1.60e+01 - 7.37e-02 4.44e-02f 1
3 6.6691211e+03 2.30e+00 9.34e+01 -1.0 1.30e+01 - 4.95e-01 2.40e-02f 1
4 6.5744238e+03 2.04e+00 8.25e+01 -1.0 1.29e+01 - 3.67e-01 1.12e-01f 2
5 6.8265929e+03 1.80e+00 7.10e+01 -1.0 1.23e+01 - 8.72e-01 1.20e-01h 2
6 8.8541540e+03 1.08e+00 4.20e+01 -1.0 9.14e+00 - 5.92e-01 4.00e-01h 1
7 1.0572759e+04 8.62e-01 3.58e+01 -1.0 2.94e+00 - 4.94e-01 2.00e-01h 1
8 1.7308372e+04 3.63e-02 1.47e+01 -1.0 2.41e+00 - 7.66e-01 9.58e-01h 1
9 1.7572883e+04 1.33e-02 1.10e+00 -1.0 2.11e+00 - 1.00e+00 1.00e+00h 1
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
10 1.7590632e+04 1.69e-03 1.61e-01 -1.0 5.03e-01 - 1.00e+00 1.00e+00h 1
11 1.7558725e+04 5.24e-03 5.03e-01 -2.5 6.03e-01 - 8.35e-01 9.36e-01f 1
12 1.7553111e+04 3.34e-03 4.11e+00 -2.5 2.84e-01 - 1.00e+00 8.20e-01h 1
13 1.7552956e+04 3.24e-05 1.26e-02 -2.5 6.35e-02 - 1.00e+00 1.00e+00h 1
14 1.7551990e+04 1.35e-05 1.09e+00 -3.8 2.53e-02 - 1.00e+00 9.25e-01h 1
15 1.7551938e+04 4.46e-08 1.23e-02 -3.8 7.00e-03 - 1.00e+00 1.00e+00f 1
16 1.7551940e+04 2.35e-10 2.06e-04 -3.8 3.84e-04 - 1.00e+00 1.00e+00h 1
17 1.7551892e+04 1.75e-07 2.11e-01 -5.7 2.49e-03 - 1.00e+00 9.68e-01f 1
18 1.7551891e+04 6.82e-11 3.10e-05 -5.7 2.38e-04 - 1.00e+00 1.00e+00f 1
19 1.7551891e+04 1.59e-14 6.53e-10 -5.7 5.20e-07 - 1.00e+00 1.00e+00h 1
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
20 1.7551891e+04 6.34e-12 3.03e-07 -8.6 3.52e-05 - 1.00e+00 1.00e+00f 1
21 1.7551891e+04 2.84e-14 2.27e-12 -8.6 3.34e-08 - 1.00e+00 1.00e+00h 1
Number of Iterations....: 21
(scaled) (unscaled)
Objective...............: 4.3879727096486914e+02 1.7551890838594765e+04
Dual infeasibility......: 2.2670978373976439e-12 9.0683913495905758e-11
Constraint violation....: 1.8371792975813150e-14 2.8421709430404007e-14
Variable bound violation: 2.9463905093507492e-08 2.9463905093507492e-08
Complementarity.........: 2.5059076302146477e-09 1.0023630520858591e-07
Overall NLP error.......: 2.5059076302146477e-09 1.0023630520858591e-07
Number of objective function evaluations = 28
Number of objective gradient evaluations = 22
Number of equality constraint evaluations = 28
Number of inequality constraint evaluations = 28
Number of equality constraint Jacobian evaluations = 22
Number of inequality constraint Jacobian evaluations = 22
Number of Lagrangian Hessian evaluations = 21
Total seconds in IPOPT = 1.868
EXIT: Optimal Solution Found.
Summary
case........: /Users/oscar/Documents/lanl-ansi/rosetta-opf/data/pglib_opf_case5_pjm.m
variables...: 44
constraints.: 53
feasible....: true
cost........: 17552
total time..: 58.78733706474304
data time.: 10.700264930725098
build time: 4.490694999694824
solve time: 43.596376180648804 |
|
@odow this last result is with sparsity enabled right? |
Yes.
|
|
Did you dig into the timing comparisons? I ran a few experiments and concluded this - Looks like there's a bit more compilation latency with |
|
I wouldn't pay too much attention to the timing at this time. This is best left for the more comprehensive experiments I run on all of the OPF problem instances. In general, we expect that |
|
@odow, could you have a look at this test on this branch, It seems to be working for building the oracles, but during the Ipopt run I get, |
|
Can reproduce. Full trace is |
|
Okay, digging in, I think this is a problem with problem-size issue with RuntimeGeneratedFunctions. Things work okay for This one is the slightly different: We're still pretty small here, but it requires on the order of 1.5x10^3 hessians of size 10^3 square. It looks like this might be SciML/RuntimeGeneratedFunctions.jl#13 and JuliaLang/julia#45173 since it looks to be a problem in the IR, and we have a very large number of constants. |
|
@Vaibhavdixit02 although this still has this crash issue, it appears to be a net-win for scalability, FD got up to 30 bus systems this PR is getting to about 73 buses (a 2x size increase). So I am going to merge this and make it the new "official" implementation. |
|
Great 🎉 |
I guess as reference, we really want to get to the 30,000 bus system, and ideally even larger. So we still need to find 3 orders of magnitude 😆 |
|
Oh 😂 |
|
But, if it is very sparse, hence not a proportional increase in the amount of symbolic simplifications needed, I think it might be the best bet |
|
@Vaibhavdixit02 here is the precise timing info if you are curious. I did confirm the bug reported by @odow, the code runs into the Julia crash for test cases with more than 73 buses.
|
It doesn't do the symbolic simplifications right now, you'd have to |
|
Is there an easy MWE for @YingboMa to run to try the |
|
The running the From inside the REPL I am usually doing If the documentation for how to run things in the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Okay, this at least constructs the sparse hessian. But it generates an invalid number: