Difference h5netcdf and netcdf4 while using xarray and dask

[renaudjester]

Hi all!

I was reading through the documentation about parallel computation with xarray and dask, and there is this note:

Note that writing netCDF files with Dask’s distributed scheduler is only supported for the netcdf4 backend.

Does it mean the the h5netcdf engine cannot be used with dask? What exactly would it mean for my processes, would they be slower (from some tests I made it seems like it doesn't seem like)?

PS: Also if you have any input on what are the pros and cons for h5netcdf vs netcdf4 in the specific context of xarray and dask, I would happy to hear then :D

[doronbehar]

I'm just a user of xarray, and only been using it for a few months, so perhaps my word should be taken with a grain of salt, but I think a clue to the answer to your question is the distinction between "engine" and "format". According to the docs of .to_netcdf, h5netcdf is an engine, and netcdf4 is a format. Using a different engine doesn't mean the format is changed, only means that the Python library used to do the job is different.

[renaudjester]

Thanks a lot @doronbehar you might be right 🤔

I guess my confusion comes from the fact that:

1. there is a format called netcdf4 and an engine called netcdf4
2. The possible engines in xarray have different capabilities so it could also have been that on engine works with dask properly the others not.

[doronbehar]

1. there is a format called netcdf4 and an engine called netcdf4

Indeed that is confusing and perhaps can be better explained in the docs!

• The possible engines in xarray have different capabilities so it could also have been that on engine works with dask properly the others not.

If you observe that a certain engine works with dask and other engines don't that's also an important information to mention in the docs of .to_netcdf.