pip install failed with HDF5_DIR environment variable issue

[leewujung]

With python=3.9-3.11, pip install echopype failed with error HDF5_DIR environment variable not set shown below.

Error messages

$ pip install echopype
Collecting echopype
  Using cached echopype-0.9.1-py3-none-any.whl.metadata (8.7 kB)
Collecting bottleneck (from echopype)
  Using cached Bottleneck-1.4.2-cp39-cp39-macosx_11_0_arm64.whl.metadata (7.7 kB)
Collecting dask[array,distributed] (from echopype)
  Using cached dask-2024.8.0-py3-none-any.whl.metadata (3.8 kB)
Collecting dask-image (from echopype)
  Using cached dask_image-2024.5.3-py3-none-any.whl.metadata (2.8 kB)
Collecting jinja2 (from echopype)
  Using cached jinja2-3.1.5-py3-none-any.whl.metadata (2.6 kB)
Collecting netCDF4>1.6 (from echopype)
  Using cached netcdf4-1.7.2.tar.gz (835 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      reading from setup.cfg...
          HDF5_DIR environment variable not set, checking some standard locations ..
      checking /Users/wujung/miniconda3/envs/echopype_20241230_py39/include...
      hdf5 headers not found in /Users/wujung/miniconda3/envs/echopype_20241230_py39/include
      checking /Users/wujung/miniconda3/envs/echopype_20241230_py39/Library/include...
      hdf5 headers not found in /Users/wujung/miniconda3/envs/echopype_20241230_py39/Library/include
      checking /Users/wujung/include...
      hdf5 headers not found in /Users/wujung/include
      checking /usr/local/include...
      hdf5 headers not found in /usr/local/include
      checking /sw/include...
      hdf5 headers not found in /sw/include
      checking /opt/include...
      hdf5 headers not found in /opt/include
      checking /opt/local/include...
      hdf5 headers not found in /opt/local/include
      checking /opt/homebrew/include...
      hdf5 headers not found in /opt/homebrew/include
      checking /usr/include...
      hdf5 headers not found in /usr/include
      Traceback (most recent call last):
        File "/Users/wujung/miniconda3/envs/echopype_20241230_py39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/wujung/miniconda3/envs/echopype_20241230_py39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/wujung/miniconda3/envs/echopype_20241230_py39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/1m/8nxc8r_900778tkhqfgh0nqh0000gn/T/pip-build-env-0nk5vu5u/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/private/var/folders/1m/8nxc8r_900778tkhqfgh0nqh0000gn/T/pip-build-env-0nk5vu5u/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/1m/8nxc8r_900778tkhqfgh0nqh0000gn/T/pip-build-env-0nk5vu5u/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 320, in run_setup
          exec(code, locals())
        File "<string>", line 277, in <module>
        File "<string>", line 226, in _populate_hdf5_info
      ValueError: did not find HDF5 headers
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Using conda/mamba and conda-forge installs everything without problem: mamba install -c conda-forge echopype

Some searches led to the same issues reported by others:

• HDF5 headers error on install Unidata/netcdf4-python#901
• on OS-X
• on Windows: though an earlier version than our pinned >1.6.0 so not a problem for us

[leewujung]

Looks like this is a wheel problem from netCDF4. Pinning netCDF4<1.7.1 would allow pip install without problem.

With python=3.11:

• pip pulled the following with netCDF4<1.7.1:

  netcdf4                   1.6.5                    pypi_0    pypi
  

• whereas conda-forge without max version pinning pulled:

  netcdf4                   1.7.2           nompi_py311ha5aeccf_101    conda-forge
  

It doesn'tseem like we have anything at the moment that requires the latest version of netcdf4, so we can probably let this sit and see if at the next release this continue to be a problem, and if so, considering pinning netCDF4<1.7.1 then.

I'll leave this issue open in case someone ran into this problem.