Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERROR: TaskFailedException: Adjacency / distance matrices must be symmetric #102

Closed
linhunt opened this issue Aug 2, 2021 · 9 comments
Closed

ERROR: TaskFailedException: Adjacency / distance matrices must be symmetric #102

linhunt opened this issue Aug 2, 2021 · 9 comments

Comments

@linhunt
Copy link

linhunt commented Aug 2, 2021 •
edited by ranjanan
Loading

Good Morning and Happy Monday!

After about 14 days and at 97% A large run I was performing crashed giving me this error message. Like in my other runs I have just one input file which has the resistances and the sources and targets are derived from this one file. Also this is the same file I have successfully run omniscape on many times. This run just has different resistance values, and different block/radius settings. I have also successfully produced Omniscape outputs for my study area at the same block/radius settings as this run.

██████████████████████ |  ETA: 10:00:5Progress:  97%|███████████████████████████████████████ |  ETA: 9:43:3 ERROR: TaskFailedException:
Adjacency / distance matrices must be symmetric
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] SimpleWeightedGraph at C:\Users\lhunt\.julia\packages\SimpleWeightedGraphs\IDzOp\src\simpleweightedgraph.jl:16 [inlined]
 [3] SimpleWeightedGraph at C:\Users\lhunt\.julia\packages\SimpleWeightedGraphs\IDzOp\src\simpleweightedgraph.jl:27 [inlined]
 [4] compute_advanced_data(::Circuitscape.RasData{Float64,Int64}, ::Circuitscape.RasterFlags, ::Dict{String,String}) at C:\Users\lhunt\.julia\packages\Circuitscape\9x9VD\src\raster\advanced.jl:56
 [5] calculate_current(::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Array{Float64,2}, ::Circuitscape.RasterFlags, ::Dict{String,String}, ::DataType) at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\utils.jl:360
 [6] solve_target!(::Int64, ::Int64, ::Dict{String,Int64}, ::Array{Float64,2}, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Omniscape.OmniscapeFlags, ::Dict{String,String}, ::Circuitscape.RasterFlags, ::Circuitscape.OutputFlags, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::Array{Union{Missing, Float64},2}, ::String, ::String, ::Float64, ::Float64, ::Float64, ::Float64, ::Array{Float64,2}, ::Array{Float64,3}, ::Array{Float64,3}, ::DataType) at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\utils.jl:507
 [7] macro expansion at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\main.jl:273 [inlined]
 [8] (::Omniscape.var"#71#threadsfor_fun#15"{Dict{String,Int64},DataType,Omniscape.OmniscapeFlags,String,String,Float64,Float64,Float64,Float64,Dict{String,String},Int64,Circuitscape.OutputFlags,Circuitscape.RasterFlags,ProgressMeter.Progress,Int64,UnitRange{Int64}})(::Bool) at .\threadingconstructs.jl:81
 [9] (::Omniscape.var"#71#threadsfor_fun#15"{Dict{String,Int64},DataType,Omniscape.OmniscapeFlags,String,String,Float64,Float64,Float64,Float64,Dict{String,String},Int64,Circuitscape.OutputFlags,Circuitscape.RasterFlags,ProgressMeter.Progress,Int64,UnitRange{Int64}})() at .\threadingconstructs.jl:48
Stacktrace:
 [1] wait at .\task.jl:267 [inlined]
 [2] threading_run(::Function) at .\threadingconstructs.jl:34
 [3] macro expansion at .\threadingconstructs.jl:93 [inlined]
 [4] run_omniscape(::Dict{String,String}, ::Array{Union{Missing, Float64},2}; reclass_table::Array{Union{Missing, Float64},2}, source_strength::Array{Union{Missing, Float64},2}, condition1::Array{Union{Missing, Float64},2}, condition2::Array{Union{Missing, Float64},2}, condition1_future::Array{Union{Missing, Float64},2}, condition2_future::Array{Union{Missing, Float64},2}, wkt::String, geotransform::Array{Float64,1}, write_outputs::Bool) at C:\Users\lhunt\.julia\packages\Omniscape\APo6L\src\main.jl:268
 [5] top-level scope at REPL[7]:1
@vlandau vlandau changed the title New Error received @96% ERROR: TaskFailedException: Adjacency / distance matrices must be symmetric Aug 2, 2021
@vlandau
Copy link
Member

vlandau commented Aug 2, 2021 •
edited
Loading

Ugh, it is never nice to get errors so late in a large compute task. Sorry about this. It appears that something is happening in Circuitscape's construction of the graphs, which could either be due to a bug in Circuitscape, or an issue with an input that Omnsicape is passing to Circuitscape.

@ranjanan is there any reason that Circuitscape would produce a non-symmetric adjacency matrix? This certainly seems like a strange edge case. I guess that's one of the challenges with Omniscape -- thousands of CIrcuitscape solves for a given problem will no doubt reveal edge cases! Large problems also make these things hard to reproduce. Maybe when an error occurs, I should catch it in Omniscape and print the row/column indices for the moving window that is being solved so it can be reproduced. This should be easy enough to do -- in fact I'm going to do this now. It will also help with debugging errors related to matrix norm checks failing.

@ranjanan
Copy link
Member

ranjanan commented Aug 2, 2021

Yeah sorry @linhunt, this is never nice. Vincent, if you could track down the particular rows, if you can track down the portion of the moving window which is causing this I can then debug on the Circuitscape. It shouldn't produce an asymmetric matrix in any case.

@linhunt
Copy link
Author

linhunt commented Aug 2, 2021 via email

@vlandau vlandau mentioned this issue Aug 2, 2021
Merged
@ranjanan
Copy link
Member

ranjanan commented Aug 5, 2021

@vlandau we will have to run this job again to reproduce the error right? In that case, @linhunt would have to send us their files?

@vlandau
Copy link
Member

vlandau commented Aug 6, 2021

Yes we would need to re-run, though it is unfortunate that it is for such a large problem. I should go ahead and issue a patch release for Omniscape now so the error catching is in the latest version.

@linhunt
Copy link
Author

linhunt commented Aug 6, 2021 via email

@vlandau
Copy link
Member

vlandau commented Aug 6, 2021 •
edited
Loading

That would be very helpful, but only if it is not costing you extra money to run the problem again. Also, how many threads are you running it on? we might be able to get it done faster since @ranjanan might have access to some more powerful computers, but I'm not sure.

Basically the new error handling will print out the index of the target pixel that is failing (the row and column for that pixel) so then we can reproduce the problem on a much smaller scale by clipping the landscape just to what we need. To do that, we'll still eventually need the full resistance surface and other raster inputs.

@vlandau vlandau mentioned this issue Aug 23, 2021
Closed
@vlandau
Copy link
Member

vlandau commented Nov 10, 2022

@linhunt Is this still giving you problems or have you been able to reproduce it and get a column/row index?

@vlandau
Copy link
Member

vlandau commented Oct 16, 2023

Closing due to lack of updates. Please reopen if it becomes an issue again!

@vlandau vlandau closed this as completed Oct 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ranjanan @vlandau @linhunt