Memory allocation problems

[baggepinnen]

I'm defining a function that solves a problem using JuMP and GLPK. After calling the function some 10+ times it suddenly throws an error like this

GLPKError("invalid GLPK.Prob")

and occasionally the following

glp_free: memory allocation error
Error detected in file env/alloc.c at line 72
double free or corruption (out)

or

error in running finalizer: GLPK.GLPKFatalError(msg="GLPK call failed. All GLPK objects you defined so far are now invalidated.")

I've checked that my input data is okay. There seems to be no issue on GLPK.jl mentioning this. Anyone familiar with it?

The function is the following

import JuMP, GLPK
import JuMP: Model, @objective, @variable, @constraint, termination_status, dual, value

function ot_jump(D, P1, P2; kwargs...)
    n = length(P1)
    @assert all(isfinite, P1) "Got nonfinite P1"
    @assert all(isfinite, P2) "Got nonfinite P2"
    @assert sum(P1) ≈ 1 "Sum(P1) ≠ 1"
    @assert sum(P2) ≈ 1 "Sum(P2) ≠ 1"
    
    model = Model(GLPK.Optimizer)
    @variable(model, γ[1:n^2])
    @objective(model, Min, γ'vec(D))
    Γ = reshape(γ,n,n)
    con1 = JuMP.@constraint(model, con1,  sum(Γ, dims=1)[:] .== P2)
    con2 = @constraint(model, con2,  sum(Γ, dims=2)[:] .== P1)
    con3 = @constraint(model, con3,  γ .>= 0)
    JuMP.optimize!(model)
    if Int(termination_status(model)) != 1
        @error Int(termination_status(model))
    end
    α, β = -dual.(con2)[:], -dual.(con1)[:]
    α .-= mean(α)
    β .-= mean(β)
    reshape(value.(γ),n,n), α, β #, -dual.(con3)[:]
end

and some typical input data are

s1(x) = x./sum(x)
P1 = rand(4) |> s1
P2 = rand(4) |> s1

D = [(i-j)^2 for i in 0:3, j in 0:3]
ot_jump(D, P1, P2)

(@v1.4) pkg> st JuMP
Status `~/.julia/environments/v1.4/Project.toml`
  [4076af6c] JuMP v0.21.1

(@v1.4) pkg> st GLPK
Status `~/.julia/environments/v1.4/Project.toml`
  [60bf3e95] GLPK v0.12.1

[odow]

What platform are you on? Can you reproduce on a different Julia version?

I don't have 1.4 installed, and can't reproduce this on Mac.

I'm guessing this is some sort of race condition, where we GC a GLPK environment, then GLPK's C finalizer runs, but before it runs, we've already created a new GLPK environment and it just happened to have the same memory address? Not really sure. Hard to say without being able to reproduce.

[odow]

@baggepinnen please try and reproduce on the latest master. I significantly changed how we handle the C pointers, so hopefully this should have fixed the errors.

[baggepinnen]

I can't reproduce the error on v0.13.0. Unfortunately, I do no longer have access to the machine where the error occurred the first time.

[odow]

Closing because this isn't reproducible.

[OwenTrokeBillard]

I experienced this error with JuMP + GLPK on Windows following the Travelling Salesperson example after executing it about 10 times. The model runs in an Oxygen.jl endpoint. I started Julia with --threads auto.

Log

glp_free: memory allocation error
Error detected in file env/alloc.c at line 70

[11136] signal (22): SIGABRT
in expression starting at C:\source\opt\salesman\api\src\Api.jl:26     
crt_sig_handler at C:/workdir/src\signals-win.c:95
raise at C:\Windows\System32\msvcrt.dll (unknown line)
abort at C:\Windows\System32\msvcrt.dll (unknown line)
errfunc at /workspace/srcdir/glpk-5.0/src\env\error.c:53
dma at /workspace/srcdir/glpk-5.0/src\env\alloc.c:70
_glp_dmp_delete_pool at /workspace/srcdir/glpk-5.0/src\misc\dmp.c:235  
delete_prob at /workspace/srcdir/glpk-5.0/src\api\prob1.c:1553
glp_delete_prob at /workspace/srcdir/glpk-5.0/src\api\prob1.c:1581     
glp_delete_prob at C:\Users\owent\.julia\packages\GLPK\usyvi\src\gen\libglpk_api.jl:98 [inlined]
#2 at C:\Users\owent\.julia\packages\GLPK\usyvi\src\MOI_wrapper\MOI_wrapper.jl:189
unknown function (ip: 0000026958e30d4b)
run_finalizer at C:/workdir/src\gc.c:318
jl_gc_run_finalizers_in_list at C:/workdir/src\gc.c:408
run_finalizers at C:/workdir/src\gc.c:454
jl_mutex_unlock at C:/workdir/src\julia_locks.h:80 [inlined]
ijl_task_get_next at C:/workdir/src\partr.c:474
poptask at .\task.jl:985
wait at .\task.jl:994
#wait#645 at .\condition.jl:130
wait at .\condition.jl:125 [inlined]
_wait at .\task.jl:310
wait at .\task.jl:350 [inlined]
wait at C:\Users\owent\.julia\packages\HTTP\bDoga\src\Servers.jl:125   
unknown function (ip: 0000026958dc0bcb)
startserver at C:\Users\owent\.julia\packages\Oxygen\A6WKA\src\core.jl:239
unknown function (ip: 0000026958d9f636)
#serve#11 at C:\Users\owent\.julia\packages\Oxygen\A6WKA\src\core.jl:160
serve at C:\Users\owent\.julia\packages\Oxygen\A6WKA\src\core.jl:140   
unknown function (ip: 0000026958d83204)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_call at C:/workdir/src\interpreter.c:126
eval_value at C:/workdir/src\interpreter.c:223
eval_stmt_value at C:/workdir/src\interpreter.c:174 [inlined]
eval_body at C:/workdir/src\interpreter.c:635
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:775        
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:934
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
ijl_toplevel_eval at C:/workdir/src\toplevel.c:943 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:985
eval at .\boot.jl:385 [inlined]
include_string at .\loading.jl:2070
_include at .\loading.jl:2130
include at .\Base.jl:495
jfptr_include_49873.1 at C:\Users\owent\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
exec_options at .\client.jl:318
_start at .\client.jl:552
jfptr__start_86719.1 at C:\Users\owent\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
true_main at C:/workdir/src\jlapi.c:582
jl_repl_entrypoint at C:/workdir/src\jlapi.c:731
mainCRTStartup at C:/workdir/cli\loader_exe.c:58
BaseThreadInitThunk at C:\Windows\System32\KERNEL32.DLL (unknown line) 
RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)     
Allocations: 22188368 (Pool: 22168751; Big: 19617); GC: 33

Update: It happens every ~15 executions.

[odow]

Does it happen if you don't use threads? I don't know if GLPK is thread safe. Do you have a reproducible example?

[OwenTrokeBillard]

It does not happen with only one thread. Indeed, it says GLPK is not thread safe in the README of this repository:

GLPK is not thread-safe and should not be used with multithreading.

Mystery solved 🎉

[odow]

I'm not sure if this is a fundamental limitation of GLPK or how we compile and call it. If anyone has suggestions for improvements we're open to them.

For now, I'd just use Distributed instead of Threads for parallelism. (Although I don't know if/how Oxygen supports this.)