Segmentation fault in 0.29 - due to bogus cidr pointer?

[mutax]

I am sanitizing larger amounts of bgpdumps and check prefixes against bogon lists.
As a result I keep larger amounts of data (multiple GB) in various data structures, mainly sets().

I am using python 3.5.3 and SubnetTree 0.29 from pip.
It seems the segfault does not occur using python 2.7 - at least not with the amount of data I put in. Will report back here once I observe a segfault.

After initial load the SubnetTree is used readonly using the "in" operator.

For some bgpdumps I can reproduce a Segmentation fault when checking if a certain IP-adress is in the SubnetTree. It is alway the same position in the file, so I guess the amount of data stored on the heap or stack is relevant.

A Segfault does not happen if I do not store data in the python sets.

I was able to backtrace this using gdb up to SubnetTree_wrap.cc

Program received signal SIGSEGV, Segmentation fault.
__strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32
32      ../sysdeps/x86_64/multiarch/../strchr.S: No such file or directory.
(gdb) bt
#0  __strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32
#1  0x00007f8ab61330a6 in strchr (__c=47, __s=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at /usr/include/string.h:226
#2  parse_cidr (mask=<synthetic pointer>, subnet=0x7ffd0252efe0, family=<synthetic pointer>, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at SubnetTree.cc:59
#3  SubnetTree::lookup (this=this@entry=0x56249ae10310, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>, size=<optimized out>) at SubnetTree.cc:257
#4  0x00007f8ab6137288 in SubnetTree___contains____SWIG_0 (size=<optimized out>, cidr=<optimized out>, self=0x56249ae10310) at SubnetTree_wrap.cc:3422
#5  _wrap_SubnetTree___contains____SWIG_0 (args=<optimized out>) at SubnetTree_wrap.cc:4585
#6  _wrap_SubnetTree___contains__ (self=<optimized out>, args=<optimized out>) at SubnetTree_wrap.cc:4664
#7  0x00005624989bf6df in PyCFunction_Call ()
#8  0x0000562498981ab2 in PyEval_EvalFrameEx ()
#9  0x00005624989850df in PyEval_EvalCodeEx ()
#10 0x00005624989c15d3 in ?? ()
#11 0x0000562498a08647 in PyObject_Call ()
#12 0x000056249892894e in ?? ()
#13 0x0000562498a08647 in PyObject_Call ()
#14 0x00005624989c9a15 in ?? ()
#15 0x000056249897fa6e in PyEval_EvalFrameEx ()
#16 0x000056249897f93f in PyEval_EvalFrameEx ()
#17 0x0000562498984286 in ?? ()
#18 0x0000562498984f9f in PyEval_EvalCode ()
#19 0x0000562498a528f2 in ?? ()
#20 0x0000562498a54e1d in PyRun_FileExFlags ()
#21 0x0000562498a555be in PyRun_SimpleFileExFlags ()
#22 0x0000562498a834d7 in Py_Main ()
#23 0x0000562498913c01 in main ()


(gdb) up
#1  0x00007f8ab61330a6 in strchr (__c=47, __s=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at /usr/include/string.h:226
226       return __builtin_strchr (__s, __c);
(gdb) 
#2  parse_cidr (mask=<synthetic pointer>, subnet=0x7ffd0252efe0, family=<synthetic pointer>, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>) at SubnetTree.cc:59
59      SubnetTree.cc: No such file or directory.
(gdb) print subnet
$1 = (inx_addr *) 0x7ffd0252efe0
(gdb) print *subnet
$2 = {sin = {s_addr = 3057479128}, sin6 = {__in6_u = {__u6_addr8 = "\330m=\266\212\177\000\000\005\000\000\000\000\000\000", __u6_addr16 = {28120, 46653, 32650, 0, 5, 0, 0, 0}, __u6_addr32 = {3057479128, 32650, 5, 0}}}}
(gdb) print *cidr
Cannot access memory at address 0x7f8ab4073050
(gdb) up
#3  SubnetTree::lookup (this=this@entry=0x56249ae10310, cidr=0x7f8ab4073050 <error: Cannot access memory at address 0x7f8ab4073050>, size=<optimized out>) at SubnetTree.cc:257
257     in SubnetTree.cc
(gdb) 
#4  0x00007f8ab6137288 in SubnetTree___contains____SWIG_0 (size=<optimized out>, cidr=<optimized out>, self=0x56249ae10310) at SubnetTree_wrap.cc:3422
3422    SubnetTree_wrap.cc: No such file or directory.
(gdb) 
#5  _wrap_SubnetTree___contains____SWIG_0 (args=<optimized out>) at SubnetTree_wrap.cc:4585
4585    in SubnetTree_wrap.cc
(gdb) 
#6  _wrap_SubnetTree___contains__ (self=<optimized out>, args=<optimized out>) at SubnetTree_wrap.cc:4664
4664    in SubnetTree_wrap.cc
(gdb)  

[mutax]

when converting the string before checking, no Segfault is triggered, e.g.:

    #!/usr/bin/env python3.5
    # ...
    t = SubnetTree.SubnetTree()
    # ...
    is_match = str(address).encode() in t

[jsiwek]

This looks like the same underlying issue as #14, which has a fix in master and v0.30 in PyPI now. Can you give it a try and let me know if you still see a a problem ?

[gombasg]

I'm seeing this as well from time to time, and I think I've found the problem with the help of Valgrind. _wrap_SubnetTree___getitem__() has this code (abbreviated):

#if PY_MAJOR_VERSION >= 3
  if ( PyUnicode_Check(obj1) )
  {
    PyObject* ascii = PyUnicode_AsASCIIString(obj1);
    PyBytes_AsStringAndSize(ascii, &arg2, &len);
    arg3 = len;
    Py_DECREF(ascii);
  }
  else if ( PyBytes_Check(obj1) )
[...]
#endif

  result = (PyObject *)SubnetTree___getitem__(arg1,arg2,arg3);

The problem is, PyBytes_AsStringAndSize() sets arg2 to point to memory owned by ascii, which is freed when the reference is dropped. So SubnetTree___getitem__() is called with an invalid pointer. This also explains why using bytes avoids the problem.