-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
processLandsat: Check if output already exists when overwrite=FALSE #70
Comments
Good point! It would make sense to check for file existence before starting any processing. |
Right, but in the case of the devel branch, it looks like There is also another odd issue I'm experiencing in that running the batcher on a large number of tiles (330 in my case) where some of them have already been processed (the others haven't due to disk getting full), all 330 threads fail stating that the files already exist, even though only ~100 files truly exist. The filenames listed are all from years 1999-2001, even though I have images up to 2017, so it feels like the list somehow wraps around... If I use a pattern to filter the images by year, it works fine; if there are 12 images that year, half of them already processed, the other half does get processed correctly. That's very weird behaviour. |
Maybe the example below better illustrates the problem. library(parallel)
fun <- function(x) {
if(x == 5){
stop('Nope !!!')
}
print(x)
}
mclapply(seq(20), fun, mc.cores = 4)
# With error catcher
funSafe <- function(...){
try(fun(...))
}
mclapply(seq(20), funSafe, mc.cores = 4) |
At the moment when
overwrite=FALSE
is used, the processing is done all the way until the raster is about to be written, and then writing gets aborted because the destination file already exists. That is quite inefficient. The output filenames should be checked first, and processing done only for the output files that do not already exist.The text was updated successfully, but these errors were encountered: