[Errno 14] Bad address; Error occurs when run python include numpy.

[diwata11]

The source code is quite simple(the same as #709), it can be performed when be running without SGX.

import numpy as np
arr = np.linspace(1,10)
print(arr)

---

supposed result:
[ 1.          1.18367347  1.36734694  1.55102041  1.73469388  1.91836735
  2.10204082  2.28571429  2.46938776  2.65306122  2.83673469  3.02040816
  3.20408163  3.3877551   3.57142857  3.75510204  3.93877551  4.12244898
  4.30612245  4.48979592  4.67346939  4.85714286  5.04081633  5.2244898
  5.40816327  5.59183673  5.7755102   5.95918367  6.14285714  6.32653061
  6.51020408  6.69387755  6.87755102  7.06122449  7.24489796  7.42857143
  7.6122449   7.79591837  7.97959184  8.16326531  8.34693878  8.53061224
  8.71428571  8.89795918  9.08163265  9.26530612  9.44897959  9.63265306
  9.81632653 10.        ]

When I run the python code above, [Errno 14] error message occurs like this(some code masks like "xxx").:

xxx@xxx:~/SGX/verify/graphene/LibOS/shim/test/apps/python$ ./python.manifest scripts/pysample.py
Get sealing key: xxx
enclave (software) key hash: xxx
trusted: [1] xxx file:/usr/bin/python
trusted: [2] xxx file:../../../../../Runtime/libsysdb.so
trusted: [3] xxx file:../../../../../Runtime/ld-linux-x86-64.so.2
trusted: [4] xxx file:../../../../../Runtime/libc.so.6
trusted: [5] xxx file:../../../../../Runtime/libdl.so.2
trusted: [6] xxx file:../../../../../Runtime/libm.so.6
trusted: [7] xxx file:../../../../../Runtime/libpthread.so.0
trusted: [8] xxx file:../../../../../Runtime/libutil.so.1
trusted: [9] xxx file:/lib/x86_64-linux-gnu/libz.so.1
trusted: [10] xxx file:/lib/x86_64-linux-gnu/libnss_compat.so.2
trusted: [11] xxx file:/lib/x86_64-linux-gnu/libnss_files.so.2
trusted: [12] xxx file:../../../../../Runtime/libnss_dns.so.2
trusted: [13] xxx file:/lib/x86_64-linux-gnu/libssl.so.1.1
trusted: [14] xxx file:/lib/x86_64-linux-gnu/libcrypto.so.1.1
trusted: [15] xxx file:../../../../../Runtime/libresolv.so.2
trusted: [16] xxx file:hosts
trusted: [17] xxx file:resolv.conf
trusted: [18] xxx file:gai.conf
trusted: [19] xxx file:/usr/lib/x86_64-linux-gnu/libffi.so.6
trusted: [20] xxx file:/lib/x86_64-linux-gnu/libbz2.so.1.0
allowed: file:/usr/lib/python2.7
allowed: file:/usr/local/lib/python2.7
allowed: file:scripts
allowed: file:/dev/urandom
  File "scripts/pysample.py", line 1, in <module>
    import numpy as np
  File "/usr/local/lib/python2.7/dist-packages/numpy/__init__.py", line 187, in <module>
    from .testing import Tester
  File "/usr/local/lib/python2.7/dist-packages/numpy/testing/__init__.py", line 12, in <module>
    from ._private.utils import *
  File "/usr/local/lib/python2.7/dist-packages/numpy/testing/_private/utils.py", line 16, in <module>
    from tempfile import mkdtemp, mkstemp
  File "/usr/lib/python2.7/tempfile.py", line 35, in <module>
    from random import Random as _Random
  File "/usr/lib/python2.7/random.py", line 887, in <module>
    _inst = Random()
  File "/usr/lib/python2.7/random.py", line 97, in __init__
    self.seed(x)
  File "/usr/lib/python2.7/random.py", line 115, in seed
    a = long(_hexlify(_urandom(2500)), 16)
[Errno 14] Bad addressDkProcessExit: Returning exit code 1

I think that the cause of this error is urandom function, so I check it using following code.

from os import urandom as _urandom
from binascii import hexlify as _hexlify

x = _urandom(2500)
print(x)

#y = _hexlify(x)
#print(y)
#a = long(y, 16)

When I run the python code above, error message is as follows.

xxx@xxx:~/SGX/verify/graphene/LibOS/shim/test/apps/python$ ./python.manifest scripts/pysample.py
Get sealing key: xxx
enclave (software) key hash: xxx
trusted: [1] xxx file:/usr/bin/python
trusted: [2] xxx file:../../../../../Runtime/libsysdb.so
trusted: [3] xxx file:../../../../../Runtime/ld-linux-x86-64.so.2
trusted: [4] xxx file:../../../../../Runtime/libc.so.6
trusted: [5] xxx file:../../../../../Runtime/libdl.so.2
trusted: [6] xxx file:../../../../../Runtime/libm.so.6
trusted: [7] xxx file:../../../../../Runtime/libpthread.so.0
trusted: [8] xxx file:../../../../../Runtime/libutil.so.1
trusted: [9] xxx file:/lib/x86_64-linux-gnu/libz.so.1
trusted: [10] xxx file:/lib/x86_64-linux-gnu/libnss_compat.so.2
trusted: [11] xxx file:/lib/x86_64-linux-gnu/libnss_files.so.2
trusted: [12] xxx file:../../../../../Runtime/libnss_dns.so.2
trusted: [13] xxx file:/lib/x86_64-linux-gnu/libssl.so.1.1
trusted: [14] xxx file:/lib/x86_64-linux-gnu/libcrypto.so.1.1
trusted: [15] xxx file:../../../../../Runtime/libresolv.so.2
trusted: [16] xxx file:hosts
trusted: [17] xxx file:resolv.conf
trusted: [18] xxx file:gai.conf
trusted: [19] xxx file:/usr/lib/x86_64-linux-gnu/libffi.so.6
trusted: [20] xxx file:/lib/x86_64-linux-gnu/libbz2.so.1.0
allowed: file:/usr/lib/python2.7
allowed: file:/usr/local/lib/python2.7
allowed: file:scripts
allowed: file:/dev/urandom
  File "scripts/check.py", line 14, in <module>
    x = _urandom(2500)
[Errno 14] Bad addressclose failed in file object destructor:
sys.excepthook is missing
DkProcessExit: Returning exit code 1

What is the root cause of this problem?
I know that you are very busy, but I would appreciate it very much if you could advise me.
※ Codes above and manifest are in Codes.zip.

[dimakuv]

Can you run it with debug_type = inline in your Manifest? And copy here the last ~50 lines of the output so we can see at which point Graphene-SGX fails internally.

[diwata11]

Error message with debug_type = inline is like this.

...
[     1] bkeep_mprotect: 0x81c886000-0x81e886000
[     1] bkeep_munmap: 0x81c886000-0x81e886000
[     1] ---- return from shim_munmap(...) = 0
[     1] ---- shim_munmap(0x81a684000,33554432) ... munmap
[     1] bkeep_mprotect: 0x81a684000-0x81c684000
[     1] bkeep_munmap: 0x81a684000-0x81c684000
[     1] ---- return from shim_munmap(...) = 0
[     1] ---- shim_munmap(0x818583000,33554432) ... munmap
[     1] bkeep_mprotect: 0x818583000-0x81a583000
[     1] bkeep_munmap: 0x818583000-0x81a583000
[     1] ---- return from shim_munmap(...) = 0
[     1] ---- shim_munmap(0x816482000,33554432) ... munmap
[     1] bkeep_mprotect: 0x816482000-0x818482000
[     1] bkeep_munmap: 0x816482000-0x818482000
[     1] ---- return from shim_munmap(...) = 0
[     1] ---- shim_munmap(0x814381000,33554432) ... munmap
[     1] bkeep_mprotect: 0x814381000-0x816381000
[     1] bkeep_munmap: 0x814381000-0x816381000
[     1] ---- return from shim_munmap(...) = 0
[     1] ---- shim_munmap(0x812280000,33554432) ... munmap
[     1] bkeep_mprotect: 0x812280000-0x814280000
[     1] bkeep_munmap: 0x812280000-0x814280000
[     1] ---- return from shim_munmap(...) = 0
[     1] ---- shim_munmap(0x81027e000,33554432) ... munmap
[     1] bkeep_mprotect: 0x81027e000-0x81227e000
[     1] bkeep_munmap: 0x81027e000-0x81227e000
[     1] ---- return from shim_munmap(...) = 0
[     1] ---- shim_exit_group (returning 1)
[     1] now kill other threads in the process
[     1] walk_thread_list(callback=0x83b27787b)
[     1] now exit the process
[     1] ipc broadcast: IPC_CLD_EXIT(1, 1, -1)
[     1] found port 0x83b1870c0 (handle 0x83bcd8590) for process 0 (type 0002)
[     1] found port 0x83b187048 (handle 0x83bcd7e10) for process 0 (type 0001)
[     1] parent not here, need to tell another process
[     1] ipc broadcast: IPC_CLD_EXIT(1, 1, -1)
[     1] found port 0x83b1870c0 (handle 0x83bcd8590) for process 0 (type 0002)
[     1] found port 0x83b187048 (handle 0x83bcd7e10) for process 0 (type 0001)
[     1] clear child tid at 0x821419a50
[     1] this is the only thread 1
[     1] exiting ipc helper
[P21790] ipc helper thread terminated
[     1] deleting port 0x83b1870c0 (handle 0x83bcd8590) for process 0
[     1] deleting port 0x83b187048 (handle 0x83bcd7e10) for process 0
[     1] process 21790 exited with status 1
DkProcessExit: Returning exit code 1

All of error message is in ErrorMessage.txt.

[dimakuv]

Looking at this log, the actual error happens during this:

[     1] ---- shim_write(2,0x83b6e8555,7) ... write
[     1] ---- return from shim_write(...) = -14
[     1] ---- shim_write(2,0x83b7a60d3,2) ... write
[     1] ---- return from shim_write(...) = -14
[Errno 14] Bad address

Since stream 2 == STDERR, it looks like Graphene-SGX wanted to write some string to standard error stream, but the pointer to the string was incorrect: https://github.com/oscarlab/graphene/blob/a0803a22b4cad839d47a7e1e3bcf0542eafd6f46/LibOS/shim/src/sys/shim_open.c#L92. If it works under non-SGX Graphene, then it could be a bug test_user_memory() function of Graphene-SGX.

Looks like a legit bug. If we'll have time, we will look into this. Thanks for reporting.

[diwata11]

I run code under non-SGX Graphene(maybe PAL_LOADER = pal_loader on graphene/LibOS/shim/test/Makefile), it works successfully.

https://github.com/oscarlab/graphene/blob/a0803a22b4cad839d47a7e1e3bcf0542eafd6f46/LibOS/shim/test/Makefile#L13

Debug messages are in non-SGX Graphene debug.txt.

Sorry to trouble you, but I look forward to check and fix it as soon as possible...

[kanakraju73]

@dimakuv Any Updates on this??

[dimakuv]

@kanakraju73 @diwata11 This was fixed on the latest (master branch) Graphene. I just tested it (in particular, the script and manifest attached in Codes.zip). Please clone the latest Graphene and try for yourself. This is what I get:

$ ./python.manifest.sgx scripts/pysample.py
Manifest file: file:python.manifest.sgx
Executable file: file:./python
Token file: file:python.token
... ...
allowed: file:/usr/lib/python2.7
allowed: file:/usr/local/lib/python2.7
allowed: file:scripts
[ 1.          1.18367347  1.36734694  1.55102041  1.73469388  1.91836735
  2.10204082  2.28571429  2.46938776  2.65306122  2.83673469  3.02040816
  3.20408163  3.3877551   3.57142857  3.75510204  3.93877551  4.12244898
  4.30612245  4.48979592  4.67346939  4.85714286  5.04081633  5.2244898
  5.40816327  5.59183673  5.7755102   5.95918367  6.14285714  6.32653061
  6.51020408  6.69387755  6.87755102  7.06122449  7.24489796  7.42857143
  7.6122449   7.79591837  7.97959184  8.16326531  8.34693878  8.53061224
  8.71428571  8.89795918  9.08163265  9.26530612  9.44897959  9.63265306
  9.81632653 10.        ]

By the way, there is no need for this line in the manifest:

sgx.allowed_files.urandom = file:/dev/urandom  # NOT NEEDED!

This is because Graphene emulates /dev/urandom by itself and never consults the host-OS urandom device.

[diwata11]

Thanks for checking!
I got latest graphene today( 9am on August 19, in Japan time ), but almost the same ERROR occurred.

Could you tell me the solution if possible? ( ERROR messages are in ERRORs_20190819.txt. )

[kanakraju73]

I've also tested with the latest master branch and end up with the same error.

With SGX
ERRORS
Manifest File

Without SGX
Errors Without SGX

[kanakraju73]

@dimakuv seems like you forgot to attach Codes.zip file. Can you upload it now?

[dimakuv]

Codes.zip was taken from the first comment in this thread: https://github.com/oscarlab/graphene/files/3400677/Codes.zip.

Unfortunately I have no time currently to verify this again. It worked on my Ubuntu 16.04 though...

[mkow]

@diwata11 @kanakraju73 I'm cleaning up old issues right now. Does this issue reproduce on current master?

[mkow]

No response from the reporters and we couldn't reproduce the issue, closing. It's probably already fixed on master.