Assertion failed: (dsid > 0), function attach_dimscales, file nc4hdf.c, line 1417. #261
Comments
|
I can reproduce this issue on Linux using your script and file (with NetCDF_jll v400.902.211+0 and v400.902.211+1): How was the input file created ? Does the error persist if you recreate the input file (if this is possible for you)? |
|
This works: NCDataset(infnam, "r") do ds
myvar = ds["MYVAR"]
tax = ds["TIME1_10"]
NCDataset(oufnam, "c") do ou
defVar(ou, "tax", tax)
defVar(ou, "myvar", myvar[:,1], ("TIME1_10",))
end
endIn |
|
Thanks for your response. After I posted my initial report here, I realized my error as you point out. So, my problem has been solved. But . . . I let the report stand as is, because the point of my report was not to ask for your help to fix my code. An assertion error is an internal error that should not be propagated to the user. That is my report here. I'm sure you understand this, but just in case . . . in practice, it's hard to debug your code if the error message is not meaningful to you. In addition, an internal error sometimes mean that you have to wait for the library writer to fix it before you can use it once again. |
|
Yes, an assertion error is always a bug as far as I can tell. But in this case, it is an assertion internal to the netcdf C library which is maintained by the fine folks from unidata (https://github.com/Unidata/netcdf-c) which should be not considered as "my" (or NCDatasets') code. Do you have time to report it to unidata? |
Yes. If not now, I'm willing to do it soon. But to do so, wouldn't we need a C or Fortran program to replicate the problem? That was the reason why I didn't submit the issue to unidata in the first place. What's the call from within the NCDatasets Julia package that's causing this? |
|
Yes, a C code would be ideal, but maybe they can give additional insights even without. But it turns out that this is an known issue.
Yes, this can also be the case. But changing the dimension names (i.e. the data of the function call, not the function calls themselves) avoids the issue. NetCDF dimensions and HDF5 dimensions do not map one to one: https://docs.unidata.ucar.edu/netcdf-c/current/interoperability_hdf5.html As I understand it HDF5 dimension scales have always values attached to them where in netcdf you can have the dimension Avoiding the using NCDatasets
infnam = "tmp-in.nc"
oufnam = "tmp-out.nc"
ds = NCDataset(infnam, "r")
myvar = ds["MYVAR"]
tax = ds["TIME1_10"]
ou = NCDataset(oufnam, "c")
defVar(ou, "tax", tax)
defVar(ou, "myvar", myvar[:,1], ("tax",))
close(ou)
close(ds)This leads to and While looking for
Indeed the following reproduces the error without input files: using NCDatasets
ou = NCDataset("test.nc", "c")
defDim(ou, "TIME1_10", 10)
nctax = defVar(ou, "tax", Int32,("TIME1_10",))
nctax[:] = 1:10
defDim(ou, "tax", 10) # no errors
ncmyvar = defVar(ou, "myvar", Int32, ("tax",))
ncmyvar[:] = 1:10 # HDF error
close(ou) # assertion errorA subsequent call to |
|
Thank you for homing in on the cause of the error. But I can't reproduce your assertion error. I commented out your "HDF error" line: using NCDatasets
ou = NCDataset("test.nc", "c")
defDim(ou, "TIME1_10", 10)
nctax = defVar(ou, "tax", Int32,("TIME1_10",))
nctax[:] = 1:10
defDim(ou, "tax", 10) # no errors
ncmyvar = defVar(ou, "myvar", Int32, ("tax",))
#ncmyvar[:] = 1:10 # HDF error
close(ou) # assertion errorI still got "NetCDF: HDF error (NetCDF error code: -101)". How did my original code reach the assertion error without stopping at the HDF error? If the netCDF library reports the HDF error and stops there, then that's not a bug in the library . . . . The netCDF-library developers will need to be shown a case of assertion error . . . |
|
I've found the answer to my question
I got the HDF error on REPL of julia. Then, when I quitted the Julia interpreter, I got the assertion error. So, I guess that the recipe is to get the HDF error first but to proceed to close the netCDF file, ignoring the error. |
|
I've translated your Julia code into Fortran as faithfully as possible, but then the error disappeared! The generated netCDF file was a correct one! See below. So, we need to know the sequence of netCDF-library calls which the Julia code generates. But then, am I using the right netCDF version? With fortran, I use netCDF 4.6.1. In the below translation, one difference is that you cannot mix program try_netcdf_error
use netcdf
implicit NONE
integer:: ncid, i, dimid_time, varid_tax, dimid_tax, varid_myvar
integer, parameter:: tax(10) = [(i,i=1,10)]
!-- ou = NCDataset("test.nc", "c")
i = nf90_create("test.nc", cmode = NF90_CLOBBER, ncid=ncid)
!-- defDim(ou, "TIME1_10", 10)
i = nf90_def_dim(ncid, name="TIME1_10", len=10, dimid=dimid_time)
!-- nctax = defVar(ou, "tax", Int32,("TIME1_10",))
i = nf90_def_var(ncid, name="tax", xtype=NF90_INT, &
dimids=dimid_time, varid=varid_tax)
!-- nctax[:] = 1:10 . . . Impossible before nf90_enddef().
!! i = nf90_put_var(ncid, varid=varid_tax, values=tax)
!-- defDim(ou, "tax", 10) # no errors
i = nf90_def_dim(ncid, name="tax", len=10, dimid=dimid_tax)
!-- ncmyvar = defVar(ou, "myvar", Int32, ("tax",))
i = nf90_def_var(ncid, name="myvar", xtype=NF90_INT, &
dimids=dimid_tax, varid=varid_myvar)
i = nf90_enddef(ncid) !-- End the def section.
!-- nctax[:] = 1:10
i = nf90_put_var(ncid, varid=varid_tax, values=tax)
if (i/=NF90_NOERR) write(*,*) trim(nf90_strerror(i))
!-- ncmyvar[:] = 1:10 # HDF error
i = nf90_put_var(ncid, varid=varid_myvar, values=tax)
if (i/=NF90_NOERR) write(*,*) trim(nf90_strerror(i))
!-- close(ou) # assertion error
i = nf90_close(ncid)
end program try_netcdf_errorOn the command line, I compile this with $ gfortran -I/opt/homebrew/include try-netcdf-error.f90 -L/opt/homebrew/lib -lnetcdff -lnetcdfas my netCDF library is installed under |
|
Actually the error is HDF5 specific using cmode = nf90_netcdf4 (which is default for NCDataset but not for Fortran). Normally the HDF5 error stop the normal program flow preventing to reach the nc_close function except in a do block where the error is caught and the file is closed and then raised again. The garbage collector also closes the netCDF file the variables goes out-of-scope (garbage collector is called on all objects in your session when you close julia). |
|
I think your conjecture on the cause of the error is very accurate. I've converted your Julia program into a minimal C++ and found that 1) if I guess that the root cause of the problem is that the following conflict is not gracefully handled within the netCDF library:
I think that the netCDF library should issue a proper warning or error when this conflict happens. Perhaps I'll submit this as a bug report to unidata. // try_netcdf_error.cc
#include <iostream>
#include "netcdf.h"
int main() {
int i, ncid, dimid_time, varid_tax, dimid_tax;
const int tax[] = {1,2,3,4,5,6,7,8,9,10};
i = nc_create("test.nc", NC_NETCDF4, &ncid);
i = nc_def_dim(ncid, "TIME", 10, &dimid_time);
//i = nc_def_dim(ncid, "tax", 7, &dimid_tax); //(1) -> fine.
i = nc_def_var(ncid, "tax", NC_INT, 1, (int[1]){dimid_time}, &varid_tax);
i = nc_def_dim(ncid, "tax", 7, &dimid_tax); //(2) -> HDF error on put_var
i = nc_enddef(ncid);
i = nc_put_var(ncid, varid_tax, tax);
if (i != NC_NOERR) {std::cerr << nc_strerror(i) << std::endl;}
i = nc_close(ncid);
} |
|
In oder to use the same NetCDF libraries as julia (including all dependencies), I executed a sample C program within the Binary Builder environment and I can reproduce the core dump: The C code is: #include <stdio.h>
#include <string.h>
#include <netcdf.h>
int main() {
int ncid, var_time_id, dim2_id, dim_id, var_myvar_id, status;
int data[3] = {1,2,3};
status = nc_create("test.nc", NC_CLOBBER|NC_NETCDF4, &ncid);
if(status != NC_NOERR) {
printf("Could not create the file.\n");
}
status = nc_def_dim(ncid, "time2", 3, &dim2_id);
if(status != NC_NOERR) {
printf("Could not create time2.\n");
}
status = nc_def_var(ncid, "time", NC_INT, 1, &dim2_id, &var_time_id);
if(status != NC_NOERR) {
printf("Could not create variable time.\n");
}
status = nc_put_var_int(ncid, var_time_id, data);
if(status != NC_NOERR) {
printf("Could not put var.\n");
}
status = nc_def_dim(ncid, "time", 3, &dim_id);
if(status != NC_NOERR) {
printf("Could not create dim time.\n");
}
status = nc_def_var(ncid, "myvar", NC_INT, 1, &dim_id, &var_myvar_id);
if(status != NC_NOERR) {
printf("Could not create myvar.\n");
}
status = nc_put_var_int(ncid, var_myvar_id, data);
if(status != NC_NOERR) {
printf("Could not put myvar.\n");
}
printf("before calling close\n");
status = nc_close(ncid);
printf("error code after close = %d\n", status);
return 0;
}However, the core dump is not triggered using all the libraries (netcdf and dependencies) from Linux (ubuntu). The error is also not triggered when I recompile netCDF version 4.9.2 compiled from source (but still keeping HDF5 1.10.7 from ubuntu). all dependencies for referenceMy understanding is that the NetCDF developers know already about the "HDF5 error". But I think that the more problematic assertion error on |
|
I can report the assertion error to the NetCDF developers as a new issue unless you want to do it :-) |
It would be wonderful if you do it! Your C code would be much more helpful to the netCDF devs than mine! Thank you. I'll read your report and their responses later. If necessary, then, I may raise the issue of using the same name between a variable and a dimension. Even when the assertion error isn't triggered (it isn't in my C++ code), the use of the same name shouldn't result in error. |
|
Wow, your report is super comprehensive!! Thank you! I have nothing to add. |
|
The upstream PR has been merged: However, it will take some times before it is available to julia users. |
Describe the bug
When I run the test program below, I get an assertion error as shown near the bottom of this report.
To Reproduce
The following test program needs an existing netCDF file
tmp-in.nc, which I'll attach, if possible. ( I've found netCDF isn't allowed. I've zipped it and attached it at the end of this report.) I tried to create a sample netCDF file from within the test program but the simple netCDF file I created didn't result in any error.The test program
try-ncdatasets.jlisExpected behavior
My original purpose is to subset an existing netCDF file.
Environment
juliaup update.versioninfo(): Shown in the following section.using Pkg; Pkg.status(mode=PKGMODE_MANIFEST): Shown in the following section.Full output
The following is the error:
tmp-in.nc.zip
The text was updated successfully, but these errors were encountered: