Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Httpclient calls with OpenSSL 1.0.1e crash #188

Closed
y1015860449 opened this issue Jul 18, 2019 · 28 comments
Closed

Httpclient calls with OpenSSL 1.0.1e crash #188

y1015860449 opened this issue Jul 18, 2019 · 28 comments
Labels

Comments

@y1015860449
Copy link

y1015860449 commented Jul 18, 2019

Crash core:

#0  0x00007fa48b4d3277 in raise () from /lib64/libc.so.6
#1  0x00007fa48b4d4968 in abort () from /lib64/libc.so.6
#2  0x00007fa48b515d37 in __libc_message () from /lib64/libc.so.6
#3  0x00007fa48b51c5e4 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fa48b521c39 in _int_realloc () from /lib64/libc.so.6
#5  0x00007fa48b522eb2 in realloc () from /lib64/libc.so.6
#6  0x00007fa48d2b5649 in CRYPTO_realloc () from ../lib/libcrypto.so.10
#7  0x00007fa48d330b11 in lh_insert () from ../lib/libcrypto.so.10
#8  0x00007fa48d3331e4 in int_thread_set_item () from ../lib/libcrypto.so.10
#9  0x00007fa48d333d6e in ERR_get_state () from ../lib/libcrypto.so.10
#10 0x00007fa48d333ef5 in ERR_clear_error () from ../lib/libcrypto.so.10
#11 0x00007fa48d6665b5 in ssl23_connect () from ../lib/libssl.so.10
#12 0x000000000049add6 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}::operator()(ssl_st*) const ()
#13 0x000000000049ed1b in bool httplib::detail::read_and_close_socket_ssl<httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}>(int, unsigned long, ssl_ctx_st*, std::mutex&, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}) ()
#14 0x000000000049afb0 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&) ()
#15 0x0000000000499c88 in httplib::Client::send(httplib::Request&, httplib::Response&) ()
#16 0x000000000049a57e in httplib::Client::Post(char const*, std::multimap<std::string, std::string, httplib::detail::ci, std::allocator<std::pair<std::string const, std::string> > > const&, std::string const&, char const*) ()
@yhirose
Copy link
Owner

yhirose commented Jul 18, 2019

@y1015860449, thanks for the report. Could you please give me more information about it to reproduce the problem on my machine for debugging?

@PixlRainbow
Copy link
Contributor

Might want to have a catch somewhere in SSLClient or SSLServer to print the SSL error stack with ERR_print_errs()

@y1015860449
Copy link
Author

@yhirose There are four crashes in two days. Use the thread pool to continuously send post requests.

@yhirose
Copy link
Owner

yhirose commented Jul 19, 2019

@y1015860449, sorry but I don't understand what you mean by 'the thread pool'. Could you explain clearly what you are actually doing by showing the small code snippet of your code? Also what OS and compiler are you using? Thanks!

@y1015860449
Copy link
Author

y1015860449 commented Jul 19, 2019

Crash core:

#0  0x00007fa48b4d3277 in raise () from /lib64/libc.so.6
#1  0x00007fa48b4d4968 in abort () from /lib64/libc.so.6
#2  0x00007fa48b515d37 in __libc_message () from /lib64/libc.so.6
#3  0x00007fa48b51c5e4 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fa48b521c39 in _int_realloc () from /lib64/libc.so.6
#5  0x00007fa48b522eb2 in realloc () from /lib64/libc.so.6
#6  0x00007fa48d2b5649 in CRYPTO_realloc () from ../lib/libcrypto.so.10
#7  0x00007fa48d330b11 in lh_insert () from ../lib/libcrypto.so.10
#8  0x00007fa48d3331e4 in int_thread_set_item () from ../lib/libcrypto.so.10
#9  0x00007fa48d333d6e in ERR_get_state () from ../lib/libcrypto.so.10
#10 0x00007fa48d333ef5 in ERR_clear_error () from ../lib/libcrypto.so.10
#11 0x00007fa48d6665b5 in ssl23_connect () from ../lib/libssl.so.10
#12 0x000000000049add6 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}::operator()(ssl_st*) const ()
#13 0x000000000049ed1b in bool httplib::detail::read_and_close_socket_ssl<httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}>(int, unsigned long, ssl_ctx_st*, std::mutex&, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}) ()
#14 0x000000000049afb0 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&) ()
#15 0x0000000000499c88 in httplib::Client::send(httplib::Request&, httplib::Response&) ()
#16 0x000000000049a57e in httplib::Client::Post(char const*, std::multimap<std::string, std::string, httplib::detail::ci, std::allocator<std::pair<std::string const, std::string> > > const&, std::string const&, char const*) ()
#17 0x00000000004d8aad in CMiPushClient::pushDataHandle(im::ANDPushMsg, bool) ()
#18 0x00000000004da364 in void std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)>::operator()<im::ANDPushMsg&, bool&, void>(CMiPushClient*, im::ANDPushMsg&, bool&) const ()
#19 0x00000000004da26a in void std::_Bind<std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)> (CMiPushClient*, im::ANDPushMsg, bool)>::__call<void, , 0ul, 1ul, 2ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul>) ()
#20 0x00000000004d9f74 in void std::_Bind<std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)> (CMiPushClient*, im::ANDPushMsg, bool)>::operator()<, void>() ()
---Type <return> to continue, or q <return> to quit---
#21 0x00000000004d9de0 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)> (CMiPushClient*, im::ANDPushMsg, bool)> >::_M_invoke(std::_Any_data const&) ()
#22 0x000000000049c250 in std::function<void ()>::operator()() const ()
#23 0x0000000000493bd1 in CThreadPool::run_task() ()
#24 0x00000000004d3ed1 in void std::_Mem_fn<void (CThreadPool::*)()>::operator()<, void>(CThreadPool*) const ()
#25 0x00000000004d3e3a in void std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) ()
#26 0x00000000004d3d00 in void std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)>::operator()<, void>() ()
#27 0x00000000004d3b88 in void std::_Bind_simple<std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)> ()>::_M_invoke<>(std::_Index_tuple<>) ()
#28 0x00000000004d39a1 in std::_Bind_simple<std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)> ()>::operator()() ()
#29 0x00000000004d352e in std::thread::_Impl<std::_Bind_simple<std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)> ()> >::_M_run() ()
#30 0x00007fa48be37c70 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-4.9.2/libstdc++-v3/src/c++11/thread.cc:84
#31 0x00007fa48dab3e25 in start_thread () from /lib64/libpthread.so.0
#32 0x00007fa48b59bbad in clone () from /lib64/libc.so.6

code:

void CMiPushClient::addMiPushData(std::shared_ptr<im::ANDPushMsg> msg)
{
	m_threadPool.add_task(&CMiPushClient::pushDataHandle, this, msg, false);
}

void CMiPushClient::pushDataHandle(std::shared_ptr<im::ANDPushMsg> msg, bool bGlobal)
{
	std::string sPostData = getPostData(msg);
	std::string sUrl = bGlobal ? mi_global_url : mi_url;
	httplib::SSLClient client(mi_url.c_str(), 443, 5);
	auto res = client.Post(mi_fun.c_str(), m_header, sPostData.c_str(), "application/x-www-form-urlencoded");
...............
}

@yhirose
Copy link
Owner

yhirose commented Jul 19, 2019

@y1015860449, thanks for the code snippet. It helps me a lot. I have some additional simple questions for you, so that I could reproduce it on my machine:

  1. It rarely happens, right? (four crashes out of thousands of requests in two days, right?)
  2. OpenSSL library version (1.0.2, 1.1.0, or 1.1.1?)
  3. OS (Linux, MacOS or Windows?)

I especially want to know the answer to the question 2. Thanks.

@yhirose yhirose added the bug label Jul 19, 2019
@y1015860449
Copy link
Author

@yhirose 1、yes。
2、OpenSSL 1.0.1e
3、centos7

@yhirose
Copy link
Owner

yhirose commented Jul 19, 2019

@y1015860449, thanks for the answers. I now have to work seriously on the thread issue which happens with versions prior to 1.1.0.

cpp-httplib/httplib.h

Lines 2361 to 2365 in 3d1ae3a

read_and_close_socket_ssl(socket_t sock, size_t keep_alive_max_count,
// TODO: OpenSSL 1.0.2 occasionally crashes...
// The upcoming 1.1.0 is going to be thread safe.
SSL_CTX *ctx, std::mutex &ctx_mutex,
U SSL_connect_or_accept, V setup, T callback) {

@yhirose
Copy link
Owner

yhirose commented Jul 19, 2019

@y1015860449, I made a necessary thread safety change for versions prior to 1.1.0 according to the OpenSSL documentation. Could you try with the latest httplib.h?

@y1015860449
Copy link
Author

@yhirose I will test it today. Does OPENSSL_VERSION_NUMBER require conditional compilation? How to compile with CMakeList if needed?

@yhirose
Copy link
Owner

yhirose commented Jul 22, 2019

@y1015860449, as you can see my changes below, I have already added the #if OPENSSL_VERSION_NUMBER < 0x10100000L condition. So all you need to do is just to replace with the latest httplib.h. Thanks for taking time!

cpp-httplib/httplib.h

Lines 2484 to 2523 in 7267b3f

#if OPENSSL_VERSION_NUMBER < 0x10100000L
class SSLThreadLocks {
public:
SSLThreadLocks() {
CRYPTO_set_locking_callback(locking_callback);
}
~SSLThreadLocks() {
CRYPTO_set_locking_callback(nullptr);
}
private:
static void locking_callback(int mode, int type, const char * /*file*/, int /*line*/) {
if (mode & CRYPTO_LOCK) {
locks_[type].lock();
} else {
locks_[type].unlock();
}
}
static std::vector<std::mutex> locks_;
};
std::vector<std::mutex> SSLThreadLocks::locks_(CRYPTO_num_locks());
#endif
class SSLInit {
public:
SSLInit() {
SSL_load_error_strings();
SSL_library_init();
}
~SSLInit() { ERR_free_strings(); }
private:
#if OPENSSL_VERSION_NUMBER < 0x10100000L
SSLThreadLocks thread_init_;
#endif
};

@y1015860449
Copy link
Author

y1015860449 commented Jul 22, 2019

@yhirose Compile Error.

CMakeFiles/apush_server.dir/hw_push_client.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/imappframe.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/imframe.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/mi_push_client.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/push_provide_server.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make[2]: *** [apush_server] Error 1
make[1]: *** [CMakeFiles/apush_server.dir/all] Error 2
make: *** [all] Error 2

@PixlRainbow
Copy link
Contributor

Were you using #include inside files which were then themselves included? Or is cmake cache still using the old version of cpp-httplib?

@y1015860449
Copy link
Author

@PixlRainbow The intermediate file generated by cmake has been cleaned up and recompiled. The result is still incorrect.

@yhirose
Copy link
Owner

yhirose commented Jul 22, 2019

@y1015860449 and @PixlRainbow, thanks for the investigation. I reproduced the linker errors with multiple translation units all of which include httplib.h. Could you try with the latest?

By the way, according to the official OpenSSL site, the version 1.0.1e is no longer supported and has security risks. https://www.openssl.org/policies/releasestrat.html

@y1015860449
Copy link
Author

@yhirose Thank you for your answer, I am now verifying that there is still a crash.
I will check if there is a memory leak in the library, because I will increase the memory for a long time and will not recycle. Libcurl will have a memory leak when using ssl in multithreading. My previous project used libcurl.

@yhirose
Copy link
Owner

yhirose commented Jul 28, 2019

@y1015860449, thanks for your further investigation. Is it possible for you to test with the latest version 1.1.1 on your project to see if the same crash happens? Thanks!

@y1015860449
Copy link
Author

@yhirose thanks for your answer. After the crash did not appear (openssl 1.01e), we have already tested it, but we have been testing it for a long time, the memory is constantly increasing, it feels like there is a memory leak. I looked up the code of my own, there is no memory that is not released, so please look at the libhttp library for memory leaks, I am currently using multiple threads to call httpclient.

@y1015860449
Copy link
Author

@yhirose I will find a time to test the latest openssl library later.

@yhirose
Copy link
Owner

yhirose commented Aug 7, 2019

@y1015860449, is it possible for you to test with httplib::Client to see if the same memory leak problem also happens? If it doesn't, the memory leak should be caused by OpenSSL library.

@yhirose yhirose changed the title Httpclient calls ssl crash Httpclient calls with OpenSSL 1.0.1e crash Aug 28, 2019
@yhirose
Copy link
Owner

yhirose commented Aug 28, 2019

@y1015860449, I'll close it because the crash problem seems to be gone. If you verify the memory leak problem, could you submit another issue? Thanks for your help!

@yhirose yhirose closed this as completed Aug 28, 2019
@zhuge20100104
Copy link

@yhirose OpenSSL 1.1.1l 24 Aug 2021 This version's client still crash on latest fedora platform for multi thread.

Segmentation fault (thread 139884551083584, pid 1437) Stack trace: [root@07882a6dc489 build]# [__memcmp_avx2_movbe] :? [std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const] /lib64/libstdc++.so.6:0x14803e [bool std::operator==<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*)] /usr/include/c++/11/bits/basic_string.h:6249 [httplib::detail::if2ip(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)] http/httplib.h:2605 (discriminator 1) [httplib::detail::create_client_socket(char const*, char const*, int, int, bool, std::function<void (int)>, long, long, long, long, long, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, httplib::Error&)::{lambda(int, addrinfo&)#1}::operator()(int, addrinfo&) const] http/httplib.h:2633 (discriminator 1) [int httplib::detail::create_socket<httplib::detail::create_client_socket(char const*, char const*, int, int, bool, std::function<void (int)>, long, long, long, long, long, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, httplib::Error&)::{lambda(int, addrinfo&)#1}>(char const*, char const*, int, int, int, bool, std::function<void (int)>, httplib::detail::create_client_socket(char const*, char const*, int, int, bool, std::function<void (int)>, long, long, long, long, long, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, httplib::Error&)::{lambda(int, addrinfo&)#1})] http/httplib.h:2541 [httplib::detail::create_client_socket(char const*, char const*, int, int, bool, std::function<void (int)>, long, long, long, long, long, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, httplib::Error&)] http/httplib.h:2627 [httplib::ClientImpl::create_client_socket(httplib::Error&) const] http/httplib.h:5616 [httplib::ClientImpl::create_and_connect_socket(httplib::ClientImpl::Socket&, httplib::Error&)] http/httplib.h:5637 [httplib::SSLClient::create_and_connect_socket(httplib::ClientImpl::Socket&, httplib::Error&)] http/httplib.h:7262 (discriminator 1) [httplib::ClientImpl::send(httplib::Request&, httplib::Response&, httplib::Error&)] http/httplib.h:5732 [httplib::ClientImpl::send_(httplib::Request&&)] http/httplib.h:5803 [httplib::ClientImpl::Get(char const*, std::multimap<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, httplib::detail::ci, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (unsigned long, unsigned long)>)] http/httplib.h:6290 [httplib::ClientImpl::Get(char const*, std::multimap<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, httplib::detail::ci, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)] http/httplib.h:6279 [HttpUtil::get_str(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)] http/impl/http_util.cpp:129 [Req::make_a_async_api_query_and_get_report_url(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)] utils/req.cpp:84 (discriminator 3)

@zhuge20100104
Copy link

If you have time, pls help to resolve it.
I am a Automation QA using boost::asio::thread_pool to create new thread and then met this error.

@zhuge20100104
Copy link

Thanks you.

@zhuge20100104
Copy link

zhuge20100104 commented Nov 18, 2021

OS: Fedora, docker env
Linux Kernel version: Linux 07882a6dc489 5.10.25-linuxkit

Multi thread library use boost::asio::thread_pool,
code as belows,

`AsyncConCurrResult async_concurr_decorator_run(void (*f)(const std::string&, json&), const std::string& path) {
auto curr_count = (std::size_t)async_concurr_count;
boost::asio::thread_pool pool{curr_count};
std::vector async_results_{};
for(int j=0; j<async_concurr_count; ++j) {
json js {};
async_results_.push_back(js);
}

    for (int i = 0; i < async_concurr_count; ++i) {
        boost::asio::post(pool, std::bind(f, std::cref(path),
                                          std::ref(async_results_[i])));
    }

    pool.join();`

@zhuge20100104
Copy link

The decorator func just use cpp-httplib as belows,

`bool HttpUtil::get_str(std::string host, std::string path, const std::map<std::string, std::string> & headers, std::string &result_string) {
try {
#ifdef CPPHTTPLIB_OPENSSL_SUPPORT
auto port = 443;
httplib::SSLClient cli(host, port);
#else
auto port = 80;
httplib::Client cli(host, port);
#endif

    cli.set_connection_timeout(ConnectionTimeout);
    cli.set_read_timeout(ReadTimeout);

    httplib::Headers headers_ {};
    for(auto&& item: headers) {
        headers_.insert(item);
    }
    
    auto res = cli.Get(path.c_str(), headers_);
    
    if(res == nullptr) {
        cli.stop();
        return false;
    }

    result_string = res->body;
    cli.stop();
    return true;
} catch(const std::exception & e) {
    std::cerr << "Exception: " << e.what() << std::endl;
    return false;
}

}`

@yhirose
Copy link
Owner

yhirose commented Nov 19, 2021

According to #1097, it's not a bug in httplib.h.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants