niftiRead.m: Include an option to call different functions depending on nifti file size #257
Conversation
|
Hi Hiromasa, You are the second person to point out problems with large nifti files. (The first problem was writing large files - yours is reading). These problems do not manifest the same way on all machines. They seem to be dependent on hardware (probably RAM) and maybe the OS. For example, I tested code that allows me to read and write a Nifti that is 12 GB and it succeeds perfectly (that is, the data gets created, written with niftiWrite, read back with niftiRead, and this is identical to the original data). So I am not sure we should make this change mandatory for everyone. One option would be to make a new Vista preference field. You can see the current preferences with: A better option, though will take some work, is to figure out why the mex function fails with large files whereas the matlab function succeeds. I am guessing that it has something to do with the way the C-code allocates memory, but I am not sure. When your niftiRead fails for a large file, do you get an error? By the way, this was my test function to read/write a 12 GB file. I would not advise running this unless you have a lot of RAM! nii = niftiCreate;
data = rand(100,100,100,1500);
s = whos('data');
fprintf('data is %5.3f GB \n', s.bytes * 1e-9);
nii = niftiSet(nii, 'data', data);
nii = niftiSet(nii, 'dim', size(data));
nii = niftiSet(nii, 'pixdim', [1 1 1 1]);
clear data;
fname = fullfile(tempdir, 'myNifit.nii.gz');
niftiWrite(nii, fname);
nii2 = niftiRead(fname);
assert(isequal(nii.data, nii2.data));
|
|
Hi Jon, Thanks for comments! I did not get any error message when the mex code fails to import large nifti files. It produces a matrix with identical size of input data, but some part of the data becomes zero. This may be a source code of mex file (written in C). I have no experience in writing C-code and this might be hard to find out a solution. However it seems to be the code is splitting jobs to allocate memory. There might be something, but a bit difficult to tell. Given the dependency on environment, the tentative solution may be vistasoft preferences, as you suggested. |
|
This will be a problem caused by various potential issues. The total number of elements/voxels in the dataset, which is correlated with the size of the file could be the issue. If your hitting 32bit code from the 2000's. When a data set requires a number bigger than 2^32 somewhere. The exact problems arise depending on what part of the chain is limited to a 32 number. Mex files need to be compiled with -largeArrayDims. Otherwise when a mex file attempts to create a matrix with more than 2^32-2 elements should throw an error "out of memory". Might be good to use that in compiling the mex files. But I don't think this is the problem you're hitting because there's no out of memory message. readFileNifti relies on the niftilib to perform all the nifti reading. nifti1_io.c uses a type "size_t" to represent the number of voxels in a file (As well as in lots of places). Size_t type is for representing memory addresses and should be 64 bit on 64 bit systems. But this can potentially cause differences between systems/compilers. Because what value this type takes depends on both the system used to compile the code and the options used. E.g. The *.mexmaci64 file would handle it fine but not the *.mexmaci binary even though both use the same c code (but that's a silly example because it would bomb out with an out of memory error as explained above). Vistasfot is using an older version of the niftilib (1.37 10 Mar 2009). I'm guessing the real problem is caused by a bug in niftilib that caused it to choke on large gzipped files because of use of an "int" type incorrectly instead of "size_t". From the current niftilib code on "1.42 06 Jul 2010 [rickr]: trouble with large (gz) files\n", Updating nifti* files to the versions from: https://nifti.nimh.nih.gov/pub/dist/src/niftilib/ |
|
I see. Here is what I learned from a little playing around:
The total file size is not the issue.
If one dimension of the data array has 2^15 elements or more, then you get
a failure - either the file doesn't write or it gets truncated.
Example:
``` Matlab
numcols = 2^15-1;
numrows = 1;
nii = niftiCreate('data', ones(numrows, numcols));
fname = fullfile(tempdir, 'myNifti.nii.gz');
niftiWrite(nii, fname);
nii = niftiRead(fname);
disp(size(nii.data))
numcols = 2^15;
numrows = 1;
nii = niftiCreate('data', ones(numrows, numcols));
fname = fullfile(tempdir, 'myNifti.nii.gz');
niftiWrite(nii, fname);
nii = niftiRead(fname);
disp(size(nii.data))
numcols = 1;
numrows = 2^15-1;
nii = niftiCreate('data', ones(numrows, numcols));
fname = fullfile(tempdir, 'myNifti.nii.gz');
niftiWrite(nii, fname);
nii = niftiRead(fname);
disp(size(nii.data))
numcols = 1;
numrows = 2^15;
nii = niftiCreate('data', ones(numrows, numcols));
fname = fullfile(tempdir, 'myNifti.nii.gz');
niftiWrite(nii, fname);
nii = niftiRead(fname);
disp(size(nii.data))
1 32767
1 1
32767 1
nim is NULL! Check to be sure the file exists.
0 0
```
…On Thu, Jul 27, 2017 at 2:59 AM, Justin Ales ***@***.***> wrote:
This will be a problem caused by various potential issues. The total
number of elements/voxels in the dataset, which is correlated with the size
of the file could be the issue. If your hitting 32bit code from the 2000's.
When a data set requires a number bigger than 2^32 somewhere. The exact
problems arise depending on what part of the chain is limited to a 32
number.
Mex files need to be compiled with -largeArrayDims. Otherwise when a mex
file attempts to create a matrix with more than 2^32-2 elements should
throw an error "out of memory". Might be good to use that in compiling the
mex files. But I don't think this is the problem you're hitting because
there's no out of memory message.
readFileNifti relies on the niftilib to perform all the nifti reading.
nifti1_io.c uses a type "size_t" to represent the number of voxels in a
file (As well as in lots of places). Size_t type is for representing memory
addresses and should be 64 bit on 64 bit systems. But this can potentially
cause differences between systems/compilers. Because what value this type
takes depends on both the system used to compile the code and the options
used. E.g. The *.mexmaci64 file would handle it fine but not the *.mexmaci
binary even though both use the same c code (but that's a silly example
because it would bomb out with an out of memory error as explained above).
Vistasfot is using an older version of the niftilib (1.37 10 Mar 2009).
I'm guessing the real problem is caused by a bug in niftilib that caused it
to choke on large gzipped files because of use of an "int" type incorrectly
instead of "size_t".
From the current niftilib code on
"1.42 06 Jul 2010 [rickr]: trouble with large (gz) files\n",
" - noted/investigated by M Hanke and Y Halchenko\n"
" - fixed znzread/write, noting example by M Adler\n"
" - changed nifti_swap_* routines/calls to take size_t (6)\n"
Updating nifti* files to the versions from: https://nifti.nimh.nih.gov/
pub/dist/src/niftilib/
and recompiling the mex files should fix the problem. I'd also suggest
adding -largeArrayDims to the mex options.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACBX3uhTgzycnY69Mu5hZVAWbqs3WozMks5sSDVegaJpZM4OjbWP>
.
--
Jonathan Winawer
Assistant Professor of Psychology and Neural Science
New York University
6 Washington Place
New York, NY, 10003
(212) 998-7922 (phone)
(212) 995-4018 (fax)
[email protected]
http://psych.nyu.edu/winawer/
|
|
That's a limitation of the nifti1 specification. 2^15-1 is the maximum value for a signed 16 bit integer. |
|
Yes, @j-ales, you are right, and the 2^15-1 limit is probably unrelated to @htakemur 's problem. It is probably reasonable enough to assume that no dimension of the nifti needs more than 2^15 elements. Separately, I noticed that readFileNifti casts dims as an int. I think this is wrong. I think it should be cast as mwSize. I changed this and recompiled the mex (for mexmaci64) and put this in a new branch called readFileNiftiFix. @htakemur : Could you try this branch and see if you are able to read large files? If you are not using 64 bit Mac, then you will want to recompile readFileNifti after checking out the new branch. You could test this code (changing the variable GB to whatever value you want), or try reading your own large files. This works fine for me, even when GB is set to more than 10 (I tested up to 12 ). |
|
A comment/question from the peanut gallery: is file IO really a performance bottle-neck for anyone? Would it make sense to get rid of the maintenance burden of keeping this C code up-to-date and error free, and rely only on the Matlab code for file IO? I imagine that file IO used to be very slow when this solution was first adopted. Is it still too slow to rely on the uncompiled version? |
|
Good question. I wondered the same thing this morning. The matlab function
is slower but I don't know how much slower. I will do some testing.
One option is to rename the m file the same as the mex, and then put the
mex functions in a folder with a .gitignore. This wY, by default the user
will use the .m file, but could switch to the mex file by compiling.
In any case, let me do some timing tests and get back to you.
On Sat, Jul 29, 2017 at 3:48 PM Ariel Rokem ***@***.***> wrote:
A comment/question from the peanut gallery: is file IO really a
performance bottle-neck for anyone? Would it make sense to get rid of the
maintenance burden of keeping this C code up-to-date and error free, and
rely only on the Matlab code for file IO? I imagine that file IO used to be
very slow when this solution was first adopted. Is it still too slow to
rely on the uncompiled version?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACBX3jT3-9CeFp1lVFNoJP-ywf5CA8NBks5sS4yngaJpZM4OjbWP>
.
--
Sent from Gmail Mobile
|
|
Methinks you are correct. (Tested from 0.04GB to 4GB) GB = 4*(.1:.1:1).^2;
n = length(GB);
niftiTime = zeros(1,n);
matTime = zeros(1,n);
for ii = 1:n
disp(ii)
dimsize = round((GB(ii)*1e9/8)^0.25);
numElements = dimsize^4;
data = reshape(1:numElements, dimsize, dimsize, dimsize, dimsize);
s = whos('data');
GB(ii) = s.bytes * 1e-9;
nii = niftiCreate('data', data); clear data;
fname = fullfile(tempdir, 'myNifti.nii');
disp('writing...')
niftiWrite(nii, fname); clear nii;
disp('reading...')
tic, niftiRead(fname); niftiTime(ii)=toc;
tic, niftiReadMatlab(fname); matTime(ii) = toc;
end
figure; set(gcf, 'Color', 'w');
plot(GB, niftiTime, 'o-', GB, matTime, 'x-', 'LineWidth', 4, 'MarkerSize', 12)
set(gca, 'FontSize', 16);
legend('niftiRead Mex', 'niftiRead M-file')
xlabel('File Size (GB)')
ylabel('Time (seconds)') |
|
Yes, I agree with you both. @htakemur, if you have a Linux machine, maybe you can run my code and test the speed of compiled v uncompiled? |
|
And in case anyone was wondering about writing, same story: no advantage for the compiled c-code.
So yes, I propose changing the niftiRead/Write wrappers to call m-files rather than mex files. We can leave the mex files in the repository in case anyone was calling them directly, or otherwise finds them useful. We need to validate the m-files to make sure they perform correctly (e.g., in terms of units, transforms, etc). I'll do this in the next week or two. |
|
Hi @JWinawer I have tested your code in my computer. The computer is Ubuntu 12.04 LTS. The MATLAB calls the readFileNifti.mexa64. I used matlab2016a. In my environment, the mex file is significantly slower than m file. I don't know why, but here is the result. (I used the same code which you described above).
|
|
That test only test the *.nii pipeline. It's the gzip pipeline that's the problem. The matlab code for reading nii is fine. It's what it does when it encounters a gz file that is slow: It copies the file to a new location, gunzips the file in that location, then reads the *.nii file. So instead of reading once it: reads,writes,reads,writes,reads which should be around 5x slower. But matlab gunzip() is also slow so for GZ files I get 15x slower operation in total. This depends somewhat on the compressibility of the file. If you want to test. I modified your benchmark code to include gzipping the file and made half of the data random so gz would compress it by a factor of ~2. For the c code there is no difference between gz and non gz files. Note: In the below test the x-axis is for the uncompressed file size for the GZ case. The nii.gz file sizes are half the .gz files. I'm also using a version of readFileNifti compiled myself against the most recent version of niftilib. Regarding @arokem comment about maintainability. You're right. But the vista c-code is just a mex wrapper for the nimh nifti reference code. The original problem is no one grabbed the new version of the niftilib code that fixed a gzip bug. Using the nifti reference code might be desirable as that's widely used code. However, it might be nice to have a matlab native version. The problem is in writing the code to efficiently handle directly reading a gzipped file. Bob wrote some code to use the java gzip class to read a gz file in matlab. Presumably that code could be used in the matlab load_nii commands. However, that trades compiled c for java. Java isn't guaranteed to be available to matlab code (although it's much more guaranteed than 10 years ago). So I'm not sure if that's a net positive. @htakemur I get the opposite from you. I get the matlab code being 1.5x slower then the c-code for the *.nii case. Using OSX 10.9.5, and matlab 2014b.
|
|
This is helpful - I get the same result, meaning that for gzipped files, there is a big advantage for the compiled function. To summarize, we have two different problems requiring two different solutions. The first problem is that @htakemur could not read large nifti files. I am guessing that @j-ales is right that the reason is we are using out of data niftilib files. I updated these fils with the newer ones from NIH and out them in the branch readFileNiftiFix. This branch also has the newly compiled mex function for 64-bit mac (I cannot recompile the others as I don't have other types of machines). Hopefully, this will fix @htakemur 's problem. It should, given that the header comments in the newer nifti1_io.c indicate a 2010 change to deal with problems with large files. I'll create a pull request for that branch. The second problem is that the m-file is slow, for all the reasons Justin mentioned. I can easily fix the problem of copying the zipped file and reading it multiple times, but that is only a small gain. The biggest problem is that Matlab's gunzip is slow, as you noted. I'd like to fix this too at some point but will leave it aside for now. Maybe java will be a solution. Not sure. |
|
Hi @JWinawer Thanks. I am now testing compiling, but it isn't very straightforward. Following the notes for remembering for myself (not necessarily asking for a help).
in order to call older version of gcc (4.7).
|
|
I will look for an ubuntu machine in my dept and see if I can compile the c-code in the readFileNiftiFix branch. |
|
@JWinawer Sorry for taking your time... |
|
Maybe worth checking PR #330 for speed as well? It uses nii_tool instead of the mex file for reading, but I haven't check for speed |
Currently, niftiRead.m produces problem to read a huge nifti file (e.g. 10GB). Revision includes that the niftiRead.m will call older (slower) code more robust for bigger file, if the nifti file size is over a certain threshold (3GB by default).
I'm not sure how much this makes sense to you or not, and the selection of 3GB is arbitrary. Also, this may affect fMRI processing as well.
Please give me any feedbacks if you had any thoughts.