RuntimeError: Decompression 'SNAPPY' not available. Options: ['BROTLI', 'GZIP', 'UNCOMPRESSED']

[datalw]

What happened:
I got this error as in the title while trying to read parquet files.

What you expected to happen:
This parque file could be read.

Minimal Complete Verifiable Example:

from fastparquet import ParquetFile
pf = ParquetFile(myfile.parquet)
df = pf.to_pandas()

Anything else we need to know?:
Then I tried the following codes from this post

import snappy

def snappy_decompress(data, uncompressed_size):
    return snappy.decompress(data)
compressions['SNAPPY'] = snappy.compress
decompressions['SNAPPY'] = snappy_decompress

And got the following error:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\mne\lib\site-packages\snappy\snappy.py", line 48, in <module>
ImportError: DLL load failed while importing _snappy: Das angegebene Modul wurde nicht gefunden.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "e:\GoogleDriveBB\Program\bb_git_repo\test.py", line 1, in <module>
    import snappy
  File "C:\ProgramData\Anaconda3\envs\mne\lib\site-packages\snappy\__init__.py", line 3, in <module>
    from .snappy import (
  File "C:\ProgramData\Anaconda3\envs\mne\lib\site-packages\snappy\snappy.py", line 51, in <module>
    from .snappy_cffi import UncompressError, compress, decompress, \
  File "C:\ProgramData\Anaconda3\envs\mne\lib\site-packages\snappy\snappy_cffi.py", line 3, in <module>
    from ._snappy_cffi import ffi, lib
ModuleNotFoundError: No module named 'snappy._snappy_cffi'

Could someone please tell me where the problem is?

Environment:

• Dask version: 2021.1.1
• Python version: 3.8.5
• Operating System: Windows 10
• Install method (conda, pip, source): conda partially, details pls see below

# Name                    Version                   Build  Channel
appdirs                   1.4.4                      py_0
backcall                  0.2.0              pyhd3eb1b0_0
blas                      1.0                         mkl
blosc                     1.21.0               h0e60522_0    conda-forge
brotli                    1.0.9                ha925a31_2
brotlipy                  0.7.0           py38h2bbff1b_1003
bzip2                     1.0.8                he774522_0
ca-certificates           2021.1.19            haa95532_0
cached-property           1.5.2                      py_0
certifi                   2020.12.5        py38haa95532_0
cffi                      1.14.4           py38hcd4344a_0
cftime                    1.3.1            py38h080aedc_0
chardet                   4.0.0           py38haa95532_1003
charls                    2.1.0                h33f27b4_2
cloudpickle               1.6.0                      py_0
colorama                  0.4.4              pyhd3eb1b0_0
cryptography              3.3.1            py38hcd4344a_0
curl                      7.71.1               h2a8f88b_1
cvxpy                     1.1.8            py38haa244fe_0    conda-forge
cvxpy-base                1.1.8            py38h4c96930_0    conda-forge
cycler                    0.10.0                   py38_0
cython                    0.29.21          py38hd77b12b_0
cytoolz                   0.11.0           py38he774522_0
dask-core                 2021.1.1           pyhd3eb1b0_0
decorator                 4.4.2              pyhd3eb1b0_0
dipy                      1.3.0            py38hbc2243c_1    conda-forge
double-conversion         3.1.5                ha925a31_1
ecos                      2.0.7.post1      py38h2a96729_0
eigen                     3.3.7                h74a9793_0
expat                     2.2.10               h33f27b4_2
fastparquet               0.5.0            py38h347fdf6_0    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
freetype                  2.10.4               hd328e21_0
giflib                    5.2.1                h62dcd97_0
gl2ps                     1.4.2                h0597ee9_0    conda-forge
glew                      2.1.0                h39d44d4_2    conda-forge
h5py                      3.1.0           nompi_py38h022eade_100    conda-forge
hdf4                      4.2.13               h712560f_2
hdf5                      1.10.6          nompi_h5268f04_1114    conda-forge
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha925a31_3
idna                      2.10               pyhd3eb1b0_0
imagecodecs               2021.1.11        py38h5da4933_1
imageio                   2.9.0                      py_0
imageio-ffmpeg            0.4.3              pyhd8ed1ab_0    conda-forge
intel-openmp              2020.2                      254
ipykernel                 5.3.4            py38h5ca1d4c_0
ipython                   7.20.0           py38hd4e2768_1
ipython_genutils          0.2.0              pyhd3eb1b0_1
jedi                      0.17.2           py38haa95532_1
joblib                    1.0.0              pyhd3eb1b0_0
jpeg                      9d                   h8ffe710_0    conda-forge
jsoncpp                   1.9.4                h2d74725_1    conda-forge
jupyter_client            6.1.7                      py_0
jupyter_core              4.7.1            py38haa95532_0
kiwisolver                1.3.1            py38hd77b12b_0
krb5                      1.18.2               hc04afaa_0
lcms2                     2.11                 hc51a39a_0
lerc                      2.2.1                hd77b12b_0
libaec                    1.0.4                h33f27b4_1
libblas                   3.9.0                     8_mkl    conda-forge
libcurl                   7.71.1               h2a8f88b_1
libdeflate                1.7                  h2bbff1b_5
libiconv                  1.15                 h1df5818_7
liblapack                 3.9.0                     8_mkl    conda-forge
libnetcdf                 4.7.4           nompi_h3a9aa94_107    conda-forge
libogg                    1.3.4                h8ffe710_1    conda-forge
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.18               h62dcd97_0
libssh2                   1.9.0                h7a1dbc1_1
libtheora                 1.1.1             h62dcd97_1004    conda-forge
libtiff                   4.2.0                hc10be44_0    conda-forge
libxml2                   2.9.10               hb89e7f3_3
libzopfli                 1.0.3                ha925a31_0
llvmlite                  0.34.0           py38h1a82afc_4
loguru                    0.5.3            py38h32f6830_2    conda-forge
lz4-c                     1.9.3                h2bbff1b_0
matplotlib-base           3.3.2            py38hba9282a_0
meshio                    4.3.8              pyhd8ed1ab_0    conda-forge
mffpy                     0.6.0              pyhd8ed1ab_0    conda-forge
mkl                       2020.4             hb70f87d_311    conda-forge
mkl-service               2.3.0            py38h196d8e1_0
mkl_fft                   1.2.0            py38h45dec08_0
mkl_random                1.1.1            py38h47e9c7a_0
mock                      4.0.3              pyhd3eb1b0_0
netcdf4                   1.5.5.1         nompi_py38h5338a22_101    conda-forge
networkx                  2.5                        py_0
nibabel                   3.2.1              pyhd8ed1ab_0    conda-forge
nilearn                   0.7.0              pyhd8ed1ab_0    conda-forge
numba                     0.51.2           py38hf9181ef_1
numexpr                   2.7.2            py38hcbcaa1e_0
numpy                     1.19.2           py38hadc3359_0
numpy-base                1.19.2           py38ha3acd2a_0
olefile                   0.46                       py_0
openjpeg                  2.3.0                h5ec785f_1
openssl                   1.1.1i               h2bbff1b_0
osqp                      0.6.2            py38h4c96930_1    conda-forge
packaging                 20.9               pyhd3eb1b0_0
pandas                    1.2.1            py38hf11a4ad_0
parso                     0.7.0                      py_0
patsy                     0.5.1                    py38_0
pickleshare               0.7.5           pyhd3eb1b0_1003
pillow                    8.1.0            py38h4fa10fc_0
pip                       20.3.3           py38haa95532_0
proj                      7.2.0                h3e70539_1    conda-forge
prompt-toolkit            3.0.8                      py_0
psutil                    5.8.0            py38h2bbff1b_1
pugixml                   1.11.4               h0e60522_0    conda-forge
pyarrow                   3.0.0                    pypi_0    pypi
pycparser                 2.20                       py_2
pydicom                   2.1.2              pyhd3deb0d_0    conda-forge
pyedflib                  0.1.20                   pypi_0    pypi
pygments                  2.7.4              pyhd3eb1b0_0
pyopenssl                 20.0.1             pyhd3eb1b0_1
pyparsing                 2.4.7              pyhd3eb1b0_0
pyreadline                2.1                      py38_1
pysocks                   1.7.1            py38haa95532_0
pytables                  3.6.1            py38h153c448_3    conda-forge
python                    3.8.5                h5fd99cc_1
python-dateutil           2.8.1              pyhd3eb1b0_0
python-picard             0.4                pyh5ca1d4c_1    conda-forge
python-snappy             0.5.4            py38hc5270cd_3    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2021.1             pyhd3eb1b0_0
pyvista                   0.28.1             pyhd8ed1ab_1    conda-forge
pywavelets                1.1.1            py38he774522_2
pywin32                   227              py38he774522_1
pyyaml                    5.4.1            py38h2bbff1b_1
pyzmq                     20.0.0           py38hd77b12b_1
qdldl-python              0.1.5            py38h4c96930_0    conda-forge
qt                        5.9.7            vc14h73c81de_0
qtpy                      1.9.0                      py_0
requests                  2.25.1             pyhd3eb1b0_0
scikit-image              0.17.2           py38h1e1f486_0
scikit-learn              0.23.2           py38h47e9c7a_0
scipy                     1.6.0            py38h14eb087_0
scooby                    0.5.6              pyh9f0ad1d_0    conda-forge
scs                       2.1.2            py38h7465cd0_2    conda-forge
setuptools                52.0.0           py38haa95532_0
six                       1.15.0           py38haa95532_0
snappy                    1.1.8                ha925a31_3    conda-forge
spyder-kernels            1.10.1           py38haa95532_0
sqlite                    3.34.0               h8ffe710_0    conda-forge
statsmodels               0.12.1           py38h2bbff1b_0
tbb                       2020.3               h74a9793_0
tbb-devel                 2020.3               h74a9793_0
threadpoolctl             2.1.0              pyh5ca1d4c_0
thrift                    0.13.0                   pypi_0    pypi
tifffile                  2021.1.14          pyhd3eb1b0_1
tk                        8.6.10               he774522_0
toolz                     0.11.1             pyhd3eb1b0_0
tornado                   6.1              py38h2bbff1b_0
tqdm                      4.56.0             pyhd3eb1b0_0
traitlets                 5.0.5              pyhd3eb1b0_0
traits                    6.2.0            py38h2bbff1b_0
transforms3d              0.3.1                      py_0    conda-forge
urllib3                   1.26.3             pyhd3eb1b0_0
utfcpp                    3.1.2                         0    conda-forge
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
vtk                       9.0.1           no_osmesa_py38h3499093_104    conda-forge
wcwidth                   0.2.5                      py_0
wheel                     0.36.2             pyhd3eb1b0_0
win32_setctime            1.0.3                      py_0    conda-forge
win_inet_pton             1.1.0            py38haa95532_0
wincertstore              0.2                      py38_0
xlrd                      2.0.1              pyhd3eb1b0_0
xz                        5.2.5                h62dcd97_0
yaml                      0.2.5                he774522_0
zeromq                    4.3.3                ha925a31_3
zfp                       0.5.5                hd77b12b_4
zlib                      1.2.11               h62dcd97_4
zstd                      1.4.5                h04227a9_0

[martindurant]

This seems to be a problem with snappy, not fastparquet. However, I am surprised, as your conda listing does suggest you have snappy/python-snappy correctly installed, and you should not need anything else. Maybe you can try installing them from defaults rather than conda-forge? If you search on your error message, you'll find lots of hits and suggestions, but I don't know what would work for you.

cc #488 (this PR stalled, but is not a bad idea)

[datalw]

@martindurant Thanks a lot for reply! I tried to reinstall them from default, unfortunately it did not help. Weirdly, I tried the same codes as above in jupyter notebook and it works. Do you happen to know what the problem is?
Btw. I tried .py and .ipynb both in VS code.

[martindurant]

In that case it sounds like a paths problem, perhaps you are not running the ptyhon you thought you were. Check you have activated the correct conda environment. One thing that sometimes happens to me, is I try to run ipython, but it's not installed in this environment, so the system picks up one in another environment, and then I have a mix of pythons conflicting.

[datalw]

Indeed, it just happend to me somehow. I just uninstalled snappy from all my environments (as default python-snappy also uninstalled together by conda). Then I installed python-snappy only in one environment - let's say env1 (as default snappy installed together by conda). And I ran the same codes in .py and .ipython both in env1. Again .py did not work (same error as above) and .ipython worked...
Then I ran them in another environment (env2, where I did not installed python-snappy yet), and .ipython worked again?! And in this environment, .py did not work, which I can understand, since I did not install it yet. I am really confused what happened there and why .py won't work properly. I would appreciate any hints.

[yohplala]

In that case it sounds like a paths problem, perhaps you are not running the ptyhon you thought you were. Check you have activated the correct conda environment. One thing that sometimes happens to me, is I try to run ipython, but it's not installed in this environment, so the system picks up one in another environment, and then I have a mix of pythons conflicting.

@martindurant, a related question. Shouldn't fastparquet tests not run if one does not have snappy installed?
It turns out I had the same error message when running python3 -m pytest to check my fastparquet branch, and as a result I installed SNAPPY to have the test running properly.

[martindurant]

Ideally there should be a pytest.importorskip to safely deal with the lack of snappy

…

On February 13, 2021 10:44:55 AM EST, yohplala ***@***.***> wrote: > In that case it sounds like a paths problem, perhaps you are not running the ptyhon you thought you were. Check you have activated the correct conda environment. One thing that sometimes happens to me, is I try to run `ipython`, but it's not installed in this environment, so the system picks up one in another environment, and then I have a mix of pythons conflicting. @martindurant, a related question. Shouldn't fastparquet tests not run if one does not have snappy installed? It turns out I had the same error message when running `python3 -m pytest` to check my fastparquet branch, and as a result I installed SNAPPY to have the test running properly. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #563 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[datalw]

I come to make an update for a workaround I just found to solve this temprarily: I downloaded the wheel file of python-snappy from the PyPI website and manually installed it via pip install --force-reinstall python_snappy-0.6.0-cp38-cp38-win_amd64.whl and this error disappeared.

[datalw]

Btw. the reason why I used --force-reinstall is because I updated python-snappy via conda from 0.5.4 to 0.6.0, but this error still existed. Since I already have the version 0.6.0, in order to reinstall it --force-reinstall has to be used.