seg fault errors during span extraction

[Dhivyakumaresan]

Hi,

we are receiving below seg faults. i have given the backtrace from core dump. could someone look at and provide some insights why the error occurs. we suspect the error is due to missing span headers.

we are using Datadog tracer.

Jul 14 16:49:45 my-instance kernel: nginx[7330]: segfault at 4 ip 00007efebda15a1b sp 00007ffdec40f470 error 4 in libdd_opentracing_plugin.so[7efebd9c3000+f0000]

nginx version: openresty/1.19.9.1

[my-instance cores]$ sudo gdb /usr/local/openresty/nginx/sbin/nginx ./core.31598
GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.amzn2.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/openresty/nginx/sbin/nginx...done.
[New LWP 31598]
[New LWP 31599]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `nginx: worker process '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&) (__functor=...,
__args#0=..., __args#1=...) at /usr/include/c++/7/bits/std_function.h:302

warning: Source file is more recent than executable.
302 std::forward<_ArgTypes>(__args)...);
[Current thread is 1 (Thread 0x7fabd30cc780 (LWP 31598))]
Missing separate debuginfos, use: debuginfo-install kong-enterprise-edition-2.8.1.1-1.noarch
(gdb) bt
#0 std::_Function_handler<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view), datadog::opentracing::SpanContext::deserialize(std::shared_ptr, const opentracing::v3::TextMapReader&, const datadog::opentracing::HeadersImpl&)::<lambda(opentracing::v3::string_view, opentracing::v3::string_view)> >::_M_invoke(const std::_Any_data &, opentracing::v3::string_view &&, opentracing::v3::string_view &&) (__functor=...,
__args#0=..., __args#1=...) at /usr/include/c++/7/bits/std_function.h:302
#1 0x0000000000552950 in std::function<opentracing::v3::expected<void, std::error_code> (opentracing::v3::string_view, opentracing::v3::string_view)>::operator()(opentracing::v3::string_view, opentracing::v3::string_view) const (__args#1=...,
__args#0=..., this=0x7ffd797fc990) at /usr/include/c++/4.8.2/functional:2471
#2 ngx_opentracing::(anonymous namespace)::NgxHeaderCarrierReader::__lambda0::operator() (header=..., __closure=) at /tmp/nginx-opentracing/opentracing/src/extract_span_context.cpp:36
#3 ngx_opentracing::for_each<ngx_table_elt_t, ngx_opentracing::(anonymous namespace)::NgxHeaderCarrierReader::ForeachKey(std::function<opentracing::v3::expected(opentracing::v3::string_view, opentracing::v3::string_view)>) const::__lambda0> (
f=..., list=...) at /tmp/nginx-opentracing/opentracing/src/utility.h:74
#4 ngx_opentracing::(anonymous namespace)::NgxHeaderCarrierReader::ForeachKey(std::function<opentracing::v3::expected<void, std::error_code>(opentracing::v3::string_view, opentracing::v3::string_view)>) const (this=this@entry=0x7ffd797fcb30, f=...)
at /tmp/nginx-opentracing/opentracing/src/extract_span_context.cpp:37
#5 0x00007fabd06ed326 in datadog::opentracing::SpanContext::deserialize (logger=std::shared_ptr (count 4, weak 0) 0x2a4a210, reader=..., headers_impl=...) at /root/dd-opentracing-cpp/src/propagation.cpp:469
#6 0x00007fabd06eeb6e in datadog::opentracing::SpanContext::deserialize (logger=std::shared_ptr (count 4, weak 0) 0x2a4a210, reader=..., styles=std::set with 2 elements = {...}) at /root/dd-opentracing-cpp/src/propagation.cpp:438
#7 0x00007fabd07191da in datadog::opentracing::Tracer::Extract (this=0x2a42910, reader=...) at /root/dd-opentracing-cpp/src/tracer.cpp:368
#8 0x0000000000552a3a in ngx_opentracing::extract_span_context (tracer=..., request=0x29ceaf0) at /tmp/nginx-opentracing/opentracing/src/extract_span_context.cpp:52
#9 0x00000000005581ff in ngx_opentracing::RequestTracing::RequestTracing (this=0x28bb1b0, request=0x29ceaf0, core_loc_conf=, loc_conf=, parent_span_context=0x0)
at /tmp/nginx-opentracing/opentracing/src/request_tracing.cpp:101
#10 0x00000000005563da in __gnu_cxx::new_allocator<ngx_opentracing::RequestTracing>::construct<ngx_opentracing::RequestTracing, ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (__p=,
this=0x2983110) at /usr/include/c++/4.8.2/ext/new_allocator.h:120
#11 std::allocator_traits<std::allocator<ngx_opentracing::RequestTracing> >::_S_construct<ngx_opentracing::RequestTracing, ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (__p=, __a=...)
at /usr/include/c++/4.8.2/bits/alloc_traits.h:254
#12 std::allocator_traits<std::allocator<ngx_opentracing::RequestTracing> >::construct<ngx_opentracing::RequestTracing, ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (__p=, __a=...)
at /usr/include/c++/4.8.2/bits/alloc_traits.h:393
#13 std::vector<ngx_opentracing::RequestTracing, std::allocator<ngx_opentracing::RequestTracing> >::_M_emplace_back_aux<ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (this=this@entry=0x2983110)
at /usr/include/c++/4.8.2/bits/vector.tcc:408
#14 0x0000000000555f73 in std::vector<ngx_opentracing::RequestTracing, std::allocator<ngx_opentracing::RequestTracing> >::emplace_back<ngx_http_request_s*&, ngx_http_core_loc_conf_s*&, ngx_opentracing::opentracing_loc_conf_t*&> (this=0x2983110)
at /usr/include/c++/4.8.2/bits/vector.tcc:101
#15 ngx_opentracing::OpenTracingContext::OpenTracingContext (this=0x2983110, request=0x29ceaf0, core_loc_conf=0xba9870, loc_conf=0xbab5b8) at /tmp/nginx-opentracing/opentracing/src/opentracing_context.cpp:21
#16 0x00000000005552ce in ngx_opentracing::on_enter_block (request=0x29ceaf0) at /tmp/nginx-opentracing/opentracing/src/opentracing_handler.cpp:39
#17 0x000000000046508c in ngx_http_core_rewrite_phase (r=0x29ceaf0, ph=) at src/http/ngx_http_core_module.c:932
#18 0x0000000000460a85 in ngx_http_core_run_phases (r=r@entry=0x29ceaf0) at src/http/ngx_http_core_module.c:878
#19 0x0000000000460b6c in ngx_http_handler (r=r@entry=0x29ceaf0) at src/http/ngx_http_core_module.c:861
#20 0x000000000046b583 in ngx_http_process_request (r=r@entry=0x29ceaf0) at src/http/ngx_http_request.c:2106
#21 0x000000000046bac1 in ngx_http_process_request_headers (rev=rev@entry=0x7fabadbd7670) at src/http/ngx_http_request.c:1508
#22 0x000000000046be14 in ngx_http_process_request_line (rev=0x7fabadbd7670) at src/http/ngx_http_request.c:1175
#23 0x00000000004543a7 in ngx_epoll_process_events (cycle=, timer=, flags=) at src/event/modules/ngx_epoll_module.c:901
#24 0x000000000044b673 in ngx_process_events_and_timers (cycle=cycle@entry=0xb32a60) at src/event/ngx_event.c:257
#25 0x0000000000452722 in ngx_worker_process_cycle (cycle=cycle@entry=0xb32a60, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:782
#26 0x00000000004510c0 in ngx_spawn_process (cycle=cycle@entry=0xb32a60, proc=proc@entry=0x4526b0 <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x5a21dd "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:199
#27 0x0000000000452b7c in ngx_start_worker_processes (cycle=cycle@entry=0xb32a60, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:382
#28 0x0000000000453328 in ngx_master_process_cycle (cycle=cycle@entry=0xb32a60) at src/os/unix/ngx_process_cycle.c:135
#29 0x000000000042afa6 in main (argc=, argv=) at src/core/nginx.c:386
(gdb)

Thanks in advance!!

[sriemer]

Cannot tell you from just the backtrace. Look at registers (info registers) and disassemble (disasm). There is an option to see the C++ source between the disassembly. Enable full argument listing.
Request support from Datadog as this is their tracer where this crashes.