Numpy incompatibility across required packages on python 3.10/OS X

[ceesem]

I'm working on some tutorials right now that involve fresh installs of various CAVE packages that depend on cloud-volume. After doing a standard pip install cloud-volume in a python 3.10 environment, importing it throws the error:

ValueError                                Traceback (most recent call last)
Cell In[1], line 1
----> 1 import cloudvolume

File ~/miniconda3/envs/swdb2023-em/lib/python3.10/site-packages/cloudvolume/__init__.py:74
     71 __version__ = '8.25.0'
     73 # Register plugins
---> 74 from .datasource.precomputed import register as register_precomputed
     75 from .datasource.graphene import register as register_graphene
     77 register_precomputed()

File ~/miniconda3/envs/swdb2023-em/lib/python3.10/site-packages/cloudvolume/datasource/precomputed/__init__.py:3
      1 from typing import Optional, Union
----> 3 from .image import PrecomputedImageSource
      4 from .metadata import PrecomputedMetadata
      5 from .mesh import PrecomputedMeshSource

File ~/miniconda3/envs/swdb2023-em/lib/python3.10/site-packages/cloudvolume/datasource/precomputed/image/__init__.py:25
     23 from cloudvolume.scheduler import schedule_jobs, DEFAULT_THREADS
     24 from ....lib import Bbox, Vec, sip, first, BboxLikeType, toiter
---> 25 from .... import sharedmemory, chunks
     27 from ... import autocropfn, readonlyguard, ImageSourceInterface
     28 from .. import sharding

File ~/miniconda3/envs/swdb2023-em/lib/python3.10/site-packages/cloudvolume/chunks.py:23
     21 import pyspng
     22 import simplejpeg
---> 23 import compresso
     24 import crackle
     25 import fastremap

File compresso.pyx:1, in init compresso()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

It turns out that pip installs numpy 1.21.6 whenever it sees cloud-volume as an install requirement, and compresso seems allergic to that. This can be worked around by running pip install --upgrade numpy to bring numpy up to version 1.25.2. Importing works fine and basic functionality, at least on precomputed and graphene datasets, seems fine.

Looking at the pip dependencies, it appears that zfpy has oldest-supported-numpy as one of its requirements, which is forcing numpy to the lowest version supported by the current python version compatible with the platform+python version.

[ceesem]

After a bit more digging, I see that you brought up this issue before on zfpy and is still unresolved.

[ceesem]

This is broadly related to #576 in that this is another case of brittle interactions between supporting a great number of compression formats without optional installs.