Limits of open_mfdataset for many netcdf files

[betolink]

I'm curious to know, what's the max number of NetCDF/HDF5 files (remote or local) that someone has opened with xarray's xr.open_mfdataset(files, parallel=True)? This is using a Dask cluster. Context: I'm opening ~4000 small files and I'm seeing a lot of memory usage just on the opening step, these are small files (3MB each), even loaded into memory they should fit on my instance.

[douglatornell]

We routinely process ~30 netCDF4 files in the 25M-75M size range to calculate month-averaged fields from day-averaged fields. Less frequently, we process 365 netCDF4 files in the 200M-2G size range to construct year-long datasets from daily hour-averaged fields. The files are all locally stored on a Linux file system. We use a processes=True dask cluster. parallel=True does not work well for us. That and more than 1 thread per process seemed to be the cause of random netCDF and HDF5 errors as described in xCDAT/xcdat#561 (comment)

[dcherian]

I've done O(10,000) files with a big-ish distributed cluster.

Your memory usage might result from useless comparisons. See the "Note" in https://docs.xarray.dev/en/stable/user-guide/io.html#reading-multi-file-datasets (xref #8778)

[betolink]

Thanks @dcherian, this is somewhat related to a comment you made not long ago on how to once that we have these many files open, operations should be faster if we use flox. I think your suggestion of changing the default values in open_mfdataset makes total sense to me.

For context, I'm prototyping a way in which earthaccess will take advantage of knowing how compliant/regular a given dataset is and use this information to speed up access by inferring sensible defaults in xarray and using the right caching strategy (via fsspec).

[dcherian]

Very nice, we'd be happy to publish a blog post on this topic at the Xarray blog if you want.

[ghiggi]

Hi @betolink. I often used xr.open_mfdataset(files, parallel=True) to open O(100,000) files.

In my experience was very important to:

• increase the maximum allowed number of opened file with ulimit -n <number>. The default on Linux is often 1024
• set os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE" before the imports (or as environment variable)
• Wtih some servers/NFS configurations, to set dask.config.set({'distributed.worker.use-file-locking': 'False'})
• constrain the dask cluster to threads_per_worker=1 to avoid netcdf locking

If the reason to open lot of files to then rechunk it (i.e. with rechunker) you should then also pay attention to the max chunk size you are gonna to create and the compressor you use. Some compressor have an upper limit on the chunk size they can deal with ... and cause some cryptic error messages or failure of the rechunking.

I saw the recent efforts you put on earthaccess and I tackled similar problems to facilitate the access and analysis of the GPM data archive nested group HDFs with xarray (see https://gpm-api.readthedocs.io/en/latest/). We can have a chat in the coming days if you want ;)

[dcherian]

Great answer. Thanks for taking the time to write this up!

[betolink]

I'd love to compare notes @ghiggi! I'll send you a message on Twitter, I have some time this Friday or next week.

[dcherian]

Can I suggest the Pangeo weekly meeting as a venue for this discussion please? I bet many are interested, and public discussions help reduce bus factor, and build up a broad pool of expertise.

[betolink]

That's a great idea @dcherian!

[ghiggi]

Hi @betolink and @dcherian. I will try to attend this evening meeting ;)