Skip to content
This repository has been archived by the owner on Jan 20, 2022. It is now read-only.

IOError: [Errno 13] Permission denied when run python include pandas #709

Closed
matthewshine opened this issue May 23, 2019 · 10 comments
Closed

IOError: [Errno 13] Permission denied when run python include pandas #709

matthewshine opened this issue May 23, 2019 · 10 comments

Comments

@matthewshine
Copy link

The source code is quite simple(pandas),it can be performed when be running with out SGX;

import pandas as pd
import numpy as np
norminal_data = pd.DataFrame([1,2,3],columns=['aa'])
print norminal_data

when I run the python code above,error message is like this:

root@/software/graphene/LibOS/shim/test/apps/python# ./python.manifest.sgx scripts/test/pandans.py 
file_map does not currently support writeable pass-through mappings on SGX.  You may add the PAL_PROT_WRITECOPY (MAP_PRIVATE) flag to your file mapping to keep the writes inside the enclave but they won't be reflected outside of the enclave.
..............................
WARNING: no physical memory support, process creation may be slow.
restore_checkpoint() at vma (-13)
shim_init() in restore_checkpoint (-13)
Saturation error in exit code -13, getting rounded down to 243
process creation failed
Traceback (most recent call last):
  File "scripts/test/pandans.py", line 11, in <module>
    print norminal_data                                        
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/base.py", line 58, in __str__
    return self.__bytes__()
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/base.py", line 70, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 634, in __unicode__
    line_width=width, show_dimensions=show_dimensions)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 720, in to_string
    line_width=line_width)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/formats/format.py", line 428, in __init__
    self._chk_truncate()
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/formats/format.py", line 444, in _chk_truncate
    (w, h) = get_terminal_size()
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/formats/terminal.py", line 37, in get_terminal_size
    current_os = platform.system()
  File "/usr/lib/python2.7/platform.py", line 1286, in system
    return uname()[0]
  File "/usr/lib/python2.7/platform.py", line 1253, in uname
    processor = _syscmd_uname('-p','')
  File "/usr/lib/python2.7/platform.py", line 987, in _syscmd_uname
    output = string.strip(f.read())
IOError: [Errno 13] Permission denied

In summary there are 3 types of err msg:

  1. file_map does not currently support writeable pass-through mappings on SGX
  2. no physical memory support, process creation may be slow.
  3. IOError: [Errno 13] Permission denied

I am not sure which is the reason that lead to the interrupt. can you give me some direction ?

Here is some addtional infomation. I have performed another python using numpy lib successfully. Both pgms are simple use one lib of pandas and numpy.But the pandas one could go well.

import numpy as np
arr = np.linspace(1,10)
print(arr)

Result:
root:/software/graphene/LibOS/shim/test/apps/python# ./python.manifest.sgx scripts/simple.py 
[ 1.          1.18367347  1.36734694  1.55102041  1.73469388  1.91836735
  2.10204082  2.28571429  2.46938776  2.65306122  2.83673469  3.02040816
  3.20408163  3.3877551   3.57142857  3.75510204  3.93877551  4.12244898
  4.30612245  4.48979592  4.67346939  4.85714286  5.04081633  5.2244898
  5.40816327  5.59183673  5.7755102   5.95918367  6.14285714  6.32653061
  6.51020408  6.69387755  6.87755102  7.06122449  7.24489796  7.42857143
  7.6122449   7.79591837  7.97959184  8.16326531  8.34693878  8.53061224
  8.71428571  8.89795918  9.08163265  9.26530612  9.44897959  9.63265306
  9.81632653 10.        ]

#267
@donporter
Copy link
Contributor

Issue 2 is definitely benign.
Issue 1 is possible, but unlikely.

I think 3 is your culprit, and 1 may be related.

I wonder how python implements _syscmd_uname. If it forks and runs /bin/uname, and that is not an allowed file in your manifest, that is probably the issue. You can probably figure this out by stracing the original.

If this is the case, this is also a situation where our error messages could be better..

@dimakuv
Copy link

dimakuv commented May 23, 2019

@matthewshine Can you attach your manifest file?

@dimakuv
Copy link

dimakuv commented May 23, 2019

Here is the implementation of _syscmd_uname: https://github.com/enthought/Python-2.7.3/blob/69fe0ffd2d85b4002cacae1f28ef2eb0f25e16ae/Lib/platform.py#L995

def _syscmd_uname(option,default=''):
    ...
    try:
        f = os.popen('uname %s 2> %s' % (option, DEV_NULL))
    except (AttributeError,os.error):
        return default
    output = string.strip(f.read())
    ...

So as Don speculated, it runs os.popen('uname') which forks and runs uname executable. As Don suggested, please strace your application and try to add /bin/uname as a trusted child to the manifest.

@matthewshine
Copy link
Author

matthewshine commented May 24, 2019 •
edited
Loading

@dimakuv thanks for your time
python.manifest.sgx.txt

I just added the /bin/uname to the manifest.template and regenerated the mainifestfile
sgx.allowed_files.lib5 = file:/bin/uname

WARNING: no physical memory support, process creation may be slow.
restore_checkpoint() at vma (-13)
shim_init() in restore_checkpoint (-13)
Saturation error in exit code -13, getting rounded down to 243
process creation failed
Traceback (most recent call last):
  File "scripts/test/pandans.py", line 11, in <module>
    print norminal_data                                        
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/base.py", line 58, in __str__
    return self.__bytes__()
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/base.py", line 70, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 634, in __unicode__
    line_width=width, show_dimensions=show_dimensions)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 720, in to_string
    line_width=line_width)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/formats/format.py", line 428, in __init__
    self._chk_truncate()
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/formats/format.py", line 444, in _chk_truncate
    (w, h) = get_terminal_size()
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/formats/terminal.py", line 37, in get_terminal_size
    current_os = platform.system()
  File "/usr/lib/python2.7/platform.py", line 1286, in system
    return uname()[0]
  File "/usr/lib/python2.7/platform.py", line 1253, in uname
    processor = _syscmd_uname('-p','')
  File "/usr/lib/python2.7/platform.py", line 987, in _syscmd_uname
    output = string.strip(f.read())
IOError: [Errno 13] Permission denied

@matthewshine
Copy link
Author

matthewshine commented May 24, 2019 •
edited
Loading

@donporter @dimakuv
sorry for so many probelms, I am a beginner in programming and not be able to do source reading well. I search "IOError: [Errno 13] Permission denied" with google and find it happened when writting file .So i edit the source like bellow

import pandas as pd
import numpy as np
norminal_data = pd.DataFrame([1,2,3],columns=['aa'])
print norminal_data.values

after I changed print dataframe to normal string. I finished finally

print norminal_data  ==> print norminal_data.values

root@ak47wyh-System-Product-Name:/software/graphene/LibOS/shim/test/apps/python# ./python.manifest.sgx scripts/test/pandans.py 
[[1]
 [2]
 [3]]

After this I tried other lib but found not even can import sklearn like this
only import inside the source

 cat scripts/test/test3.py 
import sklearn

when i run it

joblog without inline

./python.manifest.sgx scripts/test/test3.py 

/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/_multiprocessing_helpers.py:38: UserWarning: [Errno 38] Function not implemented.  joblib will operate in serial mode
  warnings.warn('%s.  joblib will operate in serial mode' % (e,))
file_map does not currently support writeable pass-through mappings on SGX.  You may add the PAL_PROT_WRITECOPY (MAP_PRIVATE) flag to your file mapping to keep the writes inside the 
file_map does not currently support writeable pass-through mappings on SGX.  You may add the 
WARNING: no physical memory support, process creation may be slow.
cannot open manifest file: sh.manifest.sgx
USAGE: /software/graphene/Pal/src/../../Runtime/pal-Linux-SGX [executable|manifest] args ...
Traceback (most recent call last):
  File "scripts/test/test3.py", line 1, in <module>
    import sklearn
  File "/usr/local/lib/python2.7/dist-packages/sklearn/__init__.py", line 64, in <module>
    from .base import clone
  File "/usr/local/lib/python2.7/dist-packages/sklearn/base.py", line 14, in <module>
    from .utils.fixes import signature
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/__init__.py", line 14, in <module>
    from . import _joblib
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/_joblib.py", line 22, in <module>
    from ..externals import joblib
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/__init__.py", line 119, in <module>
    from .parallel import Parallel
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.py", line 33, in <module>
    from .externals import loky
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/externals/loky/__init__.py", line 11, in <module>
    from .backend.context import cpu_count
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/externals/loky/backend/__init__.py", line 4, in <module>
    from .context import get_context
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/externals/loky/backend/context.py", line 21, in <module>
    from .process import LokyProcess, LokyInitMainProcess
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/externals/loky/backend/process.py", line 11, in <module>
    from .compat import BaseProcess
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/externals/loky/backend/compat.py", line 25, in <module>
    from .compat_posix import wait
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/externals/loky/backend/compat_posix.py", line 11, in <module>
    from ._posix_wait import wait
  File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/externals/loky/backend/_posix_wait.py", line 14, in <module>
    SYSTEM = platform.system()
  File "/usr/lib/python2.7/platform.py", line 1286, in system
    return uname()[0]
  File "/usr/lib/python2.7/platform.py", line 1253, in uname
    processor = _syscmd_uname('-p','')
  File "/usr/lib/python2.7/platform.py", line 987, in _syscmd_uname
    output = string.strip(f.read())
IOError: [Errno 13] Permission denied
[inline.log](https://github.com/oscarlab/graphene/files/3215662/inline.log)

joblog with inline

inline.log

mainfest file

python.manifest.sgx.txt

  1. could you help me to check my manifest file ? is there anythin missed
  2. have any one test libs like sklearn,joblib? does that works well in your enviroment?
import sklearn

@donporter
Copy link
Contributor

I suspect the issue here is that it is forking a child that runs sh, and there isn't a manifest for sh.

I would have thought the correct behavior would be to inherit the rules from the python manifest.

The closest thing to a curated python environment we have at the moment is what is built in for unit tests, but these are good issues for us to try to debug.

@matthewshine
Copy link
Author

matthewshine commented May 27, 2019 •
edited by mkow
Loading

I suspect the issue here is that it is forking a child that runs sh, and there isn't a manifest for sh.

I would have thought the correct behavior would be to inherit the rules from the python manifest.

The closest thing to a curated python environment we have at the moment is what is built in for unit tests, but these are good issues for us to try to debug.

Thanks Don and your greate work.
Although I'm still confused,I am going to try other ways to figure out the issue.
maybe I should rebuild the ubuntu with sgx bug mode.In current stage I hope a python with common libs like sklearn could run inside SGX without modify,which will be very useful and convenient .

@dimakuv
Copy link

dimakuv commented May 28, 2019

Hi Matthew,

So I looked into your issue. The package sklearn can be successfully imported. Here is the summary of what I did:

My system: Ubuntu 16.04, Python 2.7.

Installed pandas/numpy and sklearn using sudo -H pip install pandas and sudo apt install python-sklearn. Pandas and numpy are installed in /usr/local/lib/python2.7/dist-packages. Sklearn is installed in /usr/lib/python2.7/dist-packages.

My toy python script:

#!/usr/bin/env python2
import sklearn
import pandas as pd
import numpy as np
norminal_data = pd.DataFrame([1,2,3],columns=['aa'])
print norminal_data.values
print "Hello World"

The manifest is very similar to the latest one you attached here. I attach mine. The main changes I made are more trusted libraries:

...
# NOTE: in-Graphene libm is too old (2.19), while libquadmath requires at least 2.23
#sgx.trusted_files.libm = file:$(LIBCDIR)/libm.so.6
sgx.trusted_files.libmsystem = file:/lib/x86_64-linux-gnu/libm.so.6

sgx.trusted_files.lapack = file:/usr/lib/liblapack.so.3
sgx.trusted_files.gfortran = file:/usr/lib/x86_64-linux-gnu/libgfortran.so.3
sgx.trusted_files.blas = file:/usr/lib/libblas.so.3
sgx.trusted_files.quadmath = file:/usr/lib/x86_64-linux-gnu/libquadmath.so.0
sgx.trusted_files.jpeg = file:/usr/lib/x86_64-linux-gnu/libjpeg.so.8
sgx.trusted_files.tiff = file:/usr/lib/x86_64-linux-gnu/libtiff.so.5
sgx.trusted_files.lzma = file:/lib/x86_64-linux-gnu/liblzma.so.5
sgx.trusted_files.jbig = file:/usr/lib/x86_64-linux-gnu/libjbig.so.0
...

Note that these lines in the manifest are important (you actually already have them in your manifest):

sgx.allowed_files.lib4 = file:/bin/sh
sgx.allowed_files.lib5 = file:/bin/uname

Now I run cd LibOS/shim/test/apps/python; make SGX=1; make SGX_RUN=1 and this is my run of Python:

~/graphene/LibOS/shim/test/apps/python$ ./python.manifest.sgx scripts/helloworld.py
manifest file: file:python.manifest.sgx
...
Accessing file:../../../../../Runtime/libm.so.6 is denied. (Unknown error) This file is not trusted or allowed.
/usr/lib/python2.7/dist-packages/joblib/_multiprocessing_helpers.py:29: UserWarning: [Errno 38] Function not implemented.  joblib will operate in serial mode
  warnings.warn('%s.  joblib will operate in serial mode' % (e,))
/usr/lib/python2.7/dist-packages/matplotlib/__init__.py:820: UserWarning: Could not find matplotlibrc; using defaults
  warnings.warn('Could not find matplotlibrc; using defaults')
/usr/lib/python2.7/dist-packages/matplotlib/__init__.py:974: UserWarning: could not find rc file; returning defaults
  warnings.warn(message)
[[1]
 [2]
 [3]]
Hello World

So I got a couple warnings (one about libm.so which makes sense because of my dirty hack; others from Python but seem to be non-fatal), but my Python script ran to completion without errors.

@dimakuv
Copy link

dimakuv commented May 28, 2019

Hm, I cannot attach files here. Ok, here is my python.manifest.template:

#!$(PAL)

loader.preload = file:$(SHIMPATH)
loader.exec = file:/usr/bin/python
loader.execname = python
loader.env.LD_LIBRARY_PATH = /graphene:/graphene/resolv:/host:/usr/lib:/usr/lib/x86_64-linux-gnu
loader.env.PATH = /usr/bin:/bin
loader.env.USERNAME =
loader.env.HOME =
loader.env.PWD =
loader.debug_type = none

fs.mount.lib1.type = chroot
fs.mount.lib1.path = /graphene
fs.mount.lib1.uri = file:$(LIBCDIR)

fs.mount.lib2.type = chroot
fs.mount.lib2.path = /host
fs.mount.lib2.uri = file:/lib/x86_64-linux-gnu

fs.mount.bin.type = chroot
fs.mount.bin.path = /bin
fs.mount.bin.uri = file:/bin

fs.mount.usr.type = chroot
fs.mount.usr.path = /usr
fs.mount.usr.uri = file:/usr

fs.mount.etc.type = chroot
fs.mount.etc.path = /etc
fs.mount.etc.uri = file:

sys.stack.size = 256K
sys.brk.size = 4M
glibc.heap_size = 16M
sgx.thread_num = 6
sgx.enclave_size = 1024M

sgx.trusted_files.ld = file:$(LIBCDIR)/ld-linux-x86-64.so.2
sgx.trusted_files.libc = file:$(LIBCDIR)/libc.so.6
sgx.trusted_files.libdl = file:$(LIBCDIR)/libdl.so.2
# NOTE: in-Graphene libm is too old (2.19), while libquadmath requires at least 2.23
#sgx.trusted_files.libm = file:$(LIBCDIR)/libm.so.6
sgx.trusted_files.libmsystem = file:/lib/x86_64-linux-gnu/libm.so.6
sgx.trusted_files.libpthread = file:$(LIBCDIR)/libpthread.so.0
sgx.trusted_files.liburil = file:$(LIBCDIR)/libutil.so.1
sgx.trusted_files.libz = file:/lib/x86_64-linux-gnu/libz.so.1
sgx.trusted_files.libnss1 = file:/lib/x86_64-linux-gnu/libnss_compat.so.2
sgx.trusted_files.libnss2 = file:/lib/x86_64-linux-gnu/libnss_files.so.2
sgx.trusted_files.libnss3 = file:$(LIBCDIR)/libnss_dns.so.2
sgx.trusted_files.libssl = file:/lib/x86_64-linux-gnu/libssl.so.1.0.0
sgx.trusted_files.libcrypto = file:/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
sgx.trusted_files.libresolv = file:$(LIBCDIR)/libresolv.so.2
sgx.trusted_files.hosts = file:hosts
sgx.trusted_files.resolv = file:resolv.conf
sgx.trusted_files.gai = file:gai.conf
sgx.trusted_files.ffi = file:/usr/lib/x86_64-linux-gnu/libffi.so.6
sgx.trusted_files.stdcpp = file:/usr/lib/x86_64-linux-gnu/libstdc++.so.6
sgx.trusted_files.gccs = file:/lib/x86_64-linux-gnu/libgcc_s.so.1
sgx.trusted_files.bz2 = file:/lib/x86_64-linux-gnu/libbz2.so.1.0

sgx.trusted_files.lapack = file:/usr/lib/liblapack.so.3
sgx.trusted_files.gfortran = file:/usr/lib/x86_64-linux-gnu/libgfortran.so.3
sgx.trusted_files.blas = file:/usr/lib/libblas.so.3
sgx.trusted_files.quadmath = file:/usr/lib/x86_64-linux-gnu/libquadmath.so.0
sgx.trusted_files.jpeg = file:/usr/lib/x86_64-linux-gnu/libjpeg.so.8
sgx.trusted_files.tiff = file:/usr/lib/x86_64-linux-gnu/libtiff.so.5
sgx.trusted_files.lzma = file:/lib/x86_64-linux-gnu/liblzma.so.5
sgx.trusted_files.jbig = file:/usr/lib/x86_64-linux-gnu/libjbig.so.0

sgx.allowed_files.sh = file:/bin/sh
sgx.allowed_files.uname = file:/bin/uname

sgx.allowed_files.pyhome = file:/usr/lib/python2.7
sgx.allowed_files.scripts = file:scripts
sgx.allowed_files.pythonpackages = file:/usr/local/lib/python2.7/dist-packages
sgx.allowed_files.pythonpackages2 = file:/usr/lib/python2.7/dist-packages

This manifest can be cleaned up a bit, but I didn't care. Works for me.

@diwata11 diwata11 mentioned this issue Jul 17, 2019
Closed
@mkow
Copy link
Member

mkow commented Jan 19, 2021

The issue seems out-of-date, closing.

@mkow mkow closed this as completed Jan 19, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dimakuv @mkow @donporter @matthewshine