RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED'] #366

lqueryvg · 2018-08-27T13:56:05Z

Output:

RuntimeError: Decompression 'SNAPPY' not available.  Options: ['GZIP', 'UNCOMPRESSED']

Code:

from fastparquet import ParquetFile

filename = 'somefile.parquet'
pf = ParquetFile(filename)

Environment:

$ python -V; pip list | grep -e fastparquet -e snapp
Python 3.6.5
fastparquet      0.1.6
python-snappy    0.5.3

I've tried install snappy instead of python-snappy. Still no joy because with these installed...

python-snappy    0.5.3
snappy           2.6.1
snappy-manifolds 1.0

I get the error:

AttributeError: module 'snappy' has no attribute 'compress'

The text was updated successfully, but these errors were encountered:

martindurant · 2018-08-27T14:01:11Z

The block that inserts Snappy into the list of compressions looks like this:

    import snappy
    def snappy_decompress(data, uncompressed_size):
        return snappy.decompress(data)
    compressions['SNAPPY'] = snappy.compress
    decompressions['SNAPPY'] = snappy_decompress

You can try to run that yourself to find out where it is failing.

lqueryvg · 2018-08-27T14:11:58Z

Exactly the same error:

$ cat ./fastparquet_test.py

import snappy
def snappy_decompress(data, uncompressed_size):
    return snappy.decompress(data)
compressions['SNAPPY'] = snappy.compress
decompressions['SNAPPY'] = snappy_decompress

$ python ./fastparquet_test.py
Traceback (most recent call last):
  File "./fastparquet_test.py", line 5, in <module>
    compressions['SNAPPY'] = snappy.compress
AttributeError: module 'snappy' has no attribute 'compress'

martindurant · 2018-08-27T14:16:27Z

Perhaps you have another package that happens to be called snappy?
What do you get with print(snappy), help(snappy), snappy?? (the later in ipython) ?

lqueryvg · 2018-08-27T14:29:18Z

I think you are probably right. See below. But without snappy or snappy-manifolds I'm back to original runtime error.

% python
Python 3.6.5 (default, Jul 23 2018, 17:29:46)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import snappy
>>> print(snappy)
<module 'snappy' from '/Users/john.buxton/.pyenv/versions/parquet/lib/python3.6/site-packages/snappy/__init__.py'>
>>> snappy.
snappy.AbelianGroup(                           snappy.NonalternatingKnotExteriors(            snappy.export_stl
snappy.AlternatingKnotExteriors(               snappy.NonorientableClosedCensus(              snappy.filedialog
snappy.CensusKnots(                            snappy.NonorientableCuspedCensus(              snappy.horoviewer
snappy.Crossing(                               snappy.OctahedralNonorientableCuspedCensus(    snappy.infodialog
snappy.CubicalNonorientableClosedCensus(       snappy.OctahedralOrientableCuspedCensus(       snappy.isometry_signature(
snappy.CubicalNonorientableCuspedCensus(       snappy.OrientableClosedCensus(                 snappy.known_manifold_packages
snappy.CubicalOrientableClosedCensus(          snappy.OrientableCuspedCensus(                 snappy.link_objects
snappy.CubicalOrientableCuspedCensus(          snappy.RationalTangle(                         snappy.manifold_package
snappy.CuspNeighborhood(                       snappy.SnapPeaFatalError(                      snappy.manifolds
snappy.CuspNeighborhoodHP(                     snappy.SnapPy                                  snappy.method
snappy.CyOpenGL                                snappy.SnapPyHP                                snappy.mfld_class(
snappy.DTcodec(                                snappy.SnapPy_doc                              snappy.name
snappy.DirichletDomain(                        snappy.Strand(                                 snappy.number
snappy.DirichletDomainHP(                      snappy.SymmetryGroup(                          snappy.numericOutputChecker
snappy.DodecahedralNonorientableClosedCensus(  snappy.Tangle(                                 snappy.pari(
snappy.DodecahedralNonorientableCuspedCensus(  snappy.TetrahedralNonorientableCuspedCensus(   snappy.polyviewer
snappy.DodecahedralOrientableClosedCensus(     snappy.TetrahedralOrientableCuspedCensus(      snappy.ptolemy
snappy.DodecahedralOrientableCuspedCensus(     snappy.Triangulation(                          snappy.random_link(
snappy.FXrays                                  snappy.TriangulationHP(                        snappy.release_info
snappy.FundamentalGroup(                       snappy.ZeroTangle(                             snappy.required
snappy.HTLinkExteriors(                        snappy.all_translations(                       snappy.sage_helper
snappy.HolonomyGroup(                          snappy.app_menus                               snappy.snap
snappy.HolonomyGroupHP(                        snappy.browser                                 snappy.snappy_module
snappy.IcosahedralNonorientableClosedCensus(   snappy.cache                                   snappy.sys
snappy.IcosahedralOrientableClosedCensus(      snappy.canonical_retriangulation(              snappy.table(
snappy.IdentityBraid(                          snappy.cusp_translations(                      snappy.table_dict
snappy.InfinityTangle(                         snappy.database                                snappy.theme
snappy.Link(                                   snappy.database_objects                        snappy.togl
snappy.LinkExteriors(                          snappy.db_utilities                            snappy.twister
snappy.Manifold(                               snappy.decorated_isosig                        snappy.verify
snappy.ManifoldHP(                             snappy.exceptions                              snappy.version(
>>> snappy.version()
'2.6.1'
>>>

martindurant · 2018-08-27T14:31:06Z

Yes, it looks like intake/python-snappy#73 . You will have to remove or remove SnaPy , or make a new environment without it, I'm afraid.

lqueryvg · 2018-08-27T14:51:05Z

I think I've been through that loop before - FYI I've been trying to work out how to read parquet files in either Javascript or Python for several hours now :-(

Here we go with a brand new virtual env...

$ pip install python-snappy
Collecting python-snappy
  Using cached https://files.pythonhosted.org/packages/c8/9f/6a84f3b8e0eadcc86853be05e972f02561397a920d06c996966a739c10be/python_snappy-0.5.3-cp36-cp36m-macosx_10_7_x86_64.whl
Installing collected packages: python-snappy
Successfully installed python-snappy-0.5.3
[john.buxton:~/projects/athena/py] [parquet] % pip list
Package       Version
------------- -------
pip           18.0
python-snappy 0.5.3
setuptools    39.0.1

$ python
Python 3.6.5 (default, Jul 23 2018, 17:29:46)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import snappy
Traceback (most recent call last):
  File "/Users/john.buxton/.pyenv/versions/parquet/lib/python3.6/site-packages/snappy/snappy.py", line 48, in <module>
    from ._snappy import UncompressError, compress, decompress, \
ImportError: dlopen(/Users/john.buxton/.pyenv/versions/parquet/lib/python3.6/site-packages/snappy/_snappy.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libsnappy.1.dylib
  Referenced from: /Users/john.buxton/.pyenv/versions/parquet/lib/python3.6/site-packages/snappy/_snappy.cpython-36m-darwin.so
  Reason: image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/john.buxton/.pyenv/versions/parquet/lib/python3.6/site-packages/snappy/__init__.py", line 3, in <module>
    from .snappy import (
  File "/Users/john.buxton/.pyenv/versions/parquet/lib/python3.6/site-packages/snappy/snappy.py", line 51, in <module>
    from .snappy_cffi import UncompressError, compress, decompress, \
  File "/Users/john.buxton/.pyenv/versions/parquet/lib/python3.6/site-packages/snappy/snappy_cffi.py", line 3, in <module>
    from ._snappy_cffi import ffi, lib
ModuleNotFoundError: No module named 'snappy._snappy_cffi'
>>>

Messy.

martindurant · 2018-08-27T14:55:58Z

It seems that snappy didn't compile successfully for you, or for some reason your pip isn't using binaries. I use conda for everything, where binaries and their dependencies are much better defined and reliable. I suggest installing fastparquet and python-snappy with conda ( https://conda.io/miniconda.html for minimal install).

lqueryvg · 2018-08-27T15:14:54Z

Argh I hope it works nicely with pyenv. I really like pyenv.

martindurant · 2018-08-27T15:19:21Z

The only wheel that seems to exist is 0.5.3-cp36-cp36m-macosx_10_7_x86_64, so unless you match that, you'd need to compile your own, and figure out what is failing. You may want to download the source from https://github.com/andrix/python-snappy/ and follow instructions there by hand.

lqueryvg · 2018-08-27T18:21:38Z

Ooh - I think I might have got somewhere with the following...

$ brew install snappy # snappy library from Google
$ CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy

Taken from the bottom of your link
https://github.com/andrix/python-snappy/

@martindurant thanks for the help - fingers crossed I can finally start to have a play with this module.

martindurant · 2018-08-27T18:58:56Z

I'll close this, then - @lqueryvg , feel free to post on python-snappy if you have more info to give people that might be looking there.

matanox · 2019-07-23T19:18:03Z

On Ubuntu linux, I simply installed python-snappy from conda-forge, and it simply avoids the error.

RabiaMelhem · 2020-01-26T15:15:33Z

RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED']

Try this please :

pip install fastparquet pyarrow dask

ek-nath · 2020-07-09T23:46:22Z

I only had to install python-snappy, which had the dependency: sudo apt-get install libsnappy-dev

bhazard · 2020-07-11T21:31:41Z

@RabiaMelhem -- I'm trying to replace pyarrow with fastparquet. As I understand, only one of those should be needed. So installing both not a good answer for me.

quasiben · 2020-07-13T14:55:04Z

@bhazard if you are running into problems I would recommend open a new issue. With that said, I understand the desire to reduce space but installing pyarrow and fastparquet into the same python env should not cause any conflicts

martindurant · 2020-07-13T14:59:11Z

Indeed, fastparquet does not pull in snappy since, although it is common, you can handle many parquet files without it. It appears that pyarrow has an everything philosophy on this (perhaps it even bundles and compiles the snappy code with pip, not sure). In any case, if you pip install snappy, you will need the C libraries - for example by executing the apt-get command above, or the equivalent for your system. If you install via conda, then the binary compiled version can be installed directly (python-snappy: the python library along, snappy: the compiled C library).

martindurant closed this as completed Aug 27, 2018

anjakefala mentioned this issue Oct 11, 2018

Support for snappy compression format ? saulpw/visidata#195

Closed

talebzeghmi mentioned this issue Nov 7, 2019

Use pyarrow & fix version lookup & integrate with Travis & Coveralls … zillow/intake-dal#2

Merged

arnabbiswas1 mentioned this issue Nov 19, 2020

Azure : RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'LZ4', 'UNCOMPRESSED'] dask/dask-cloudprovider#177

Open

datalw mentioned this issue Feb 10, 2021

RuntimeError: Decompression 'SNAPPY' not available. Options: ['BROTLI', 'GZIP', 'UNCOMPRESSED'] #563

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED'] #366

RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED'] #366

lqueryvg commented Aug 27, 2018 •

edited

Loading

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

matanox commented Jul 23, 2019

RabiaMelhem commented Jan 26, 2020

ek-nath commented Jul 9, 2020

bhazard commented Jul 11, 2020

quasiben commented Jul 13, 2020

martindurant commented Jul 13, 2020

RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED'] #366

RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'UNCOMPRESSED'] #366

Comments

lqueryvg commented Aug 27, 2018 • edited Loading

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

lqueryvg commented Aug 27, 2018

martindurant commented Aug 27, 2018

matanox commented Jul 23, 2019

RabiaMelhem commented Jan 26, 2020

ek-nath commented Jul 9, 2020

bhazard commented Jul 11, 2020

quasiben commented Jul 13, 2020

martindurant commented Jul 13, 2020

lqueryvg commented Aug 27, 2018 •

edited

Loading