Possible Collaboration

[lesserwhirls]

Interesting work @jamesmudd! We've been working on a pure java HDF5 read implementation for quite a few years now for many of the reasons you describe in the [README]
(https://github.com/jamesmudd/jhdf#why-did-i-start-jhdf) (especially the point "[a]lso, as a widely used file format for storing scientific, engineering, and commercial data, it seem like a good idea to be able to read HDF5 files with more than one library."). It is a challenge, for sure! The goal of our reader is to read HDF5 into our Common Data Model, so likely not all of the code is applicable to the work here, but there might be some interesting parts in there that could be useful.

Currently, we only handle Superblock 2 and lower features, but are looking to start adding Superblock 3 features. Anyways, I wanted to at least reach out and say "hi!", and offer to help when/where I can. If you are interested in what we have done, it can be found in the ucar.nc2.internal.iosp.hdf5.

[lesserwhirls]

There is also a pure java HDF5 writer out there from NCAR called nujan that might also be of interest, if you are considering writing capabilities. We punted on writing (specifically for netCDF-4, which is really like an HDF5 metadata spec) and wrap the netCDF-C library.

[jamesmudd]

Hi @lesserwhirls

Thanks a lot for the message, and looking at jhdf. I have seen the netCDF library before, but have not taken a deep look at the code so I will do that.

I have also seen the nujan library there might be some useful code there as well so will take a better look. I have a kind of WIP POC for writing on a branch https://github.com/jamesmudd/jhdf/tree/writing but still very early!

Collaborating could be good any areas you think would make sense? Of course your free to look and reuse this code if its helpful.

[lesserwhirls]

Sounds good! It looks like you are using MIT? It looks like you have a very nice collection of test data going in this repo, which would actually help me quite a bit as I implement superblock v3. Most of our HDF5 test data comes in the form of scientific datasets, which are not as nice. Would it be ok if I used those (not sure how you've licensed those). I'll take a look more closely at the jHDF code and the open issues to see if there is anything that jumps out to me where I might be able to help.

Would you rather I keep adding to this issue for discussions? Have you considered enabling the GitHub discussions on your repo?

[jamesmudd]

Yes your welcome to use the test files they are also MIT. Most of them have an associated python script using h5py to generate them, a few are from other places, like issue submissions etc.

I have enabled discussions and converted this issue.