Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: enhance ASAN/LSAN memory leak detection #1074

Merged
merged 1 commit into from
Feb 11, 2025

Conversation

MamziB
Copy link
Collaborator

@MamziB MamziB commented Feb 6, 2025

This PR enhances memory leak detection during testing by configuring AddressSanitizer (ASan) and LeakSanitizer (LSan) with more robust options. Specifically, it disables fast unwinding on malloc (fast_unwind_on_malloc=0) for more reliable leak detection, enables leak detection (detect_leaks=1), and disables printing of suppressions (print_suppressions=0) in ASan. For LSan, it enables reporting of leaked objects (report_objects=1) to provide more detailed information about memory leaks.
These changes will provide more comprehensive memory leak reports, helping to identify and fix potential memory issues.

An example of such a feature is the below CI output (taken from #1023 after applying this new changes) that detects where the memory is leaking and which memory addresses are leaking:


Indirect leak of 288 byte(s) in 64 object(s) allocated from:
    #0 0x6304a4 in strdup (/home/runner/work/ucc/ucc/test/gtest/gtest+0x6304a4)
    #1 0x7f4e6d59fa4c in ucc_config_names_array_dup /home/runner/work/ucc/ucc/src/utils/ucc_parser.c:249:25
    #2 0x7f4e692a715d in ucc_cl_basic_get_lib_attr /home/runner/work/ucc/ucc/src/components/cl/basic/cl_basic_lib.c:64:18
    #3 0x7f4e6d557db0 in ucc_cl_lib_init /home/runner/work/ucc/ucc/src/core/ucc_lib.c:143:18
    #4 0x7f4e6d557db0 in ucc_init_version /home/runner/work/ucc/ucc/src/core/ucc_lib.c:349:14
    #5 0x734b96 in ucc_init(ucc_lib_params const*, ucc_lib_config*, ucc_lib_info**) /home/runner/work/ucc/ucc/src/ucc/api/ucc.h:783:12
    #6 0x734b96 in UccProcess::UccProcess(int, ucc_lib_params const&, ucc_context_params const&) /home/runner/work/ucc/ucc/test/gtest/common/test_ucc.cc:28:14
    #7 0x73c09e in void __gnu_cxx::new_allocator<UccProcess>::construct<UccProcess, int&>(UccProcess*, int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/ext/new_allocator.h:156:23
    #8 0x73c09e in void std::allocator_traits<std::allocator<UccProcess> >::construct<UccProcess, int&>(std::allocator<UccProcess>&, UccProcess*, int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/alloc_traits.h:512:8
    #9 0x73c09e in std::_Sp_counted_ptr_inplace<UccProcess, std::allocator<UccProcess>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<int&>(std::allocator<UccProcess>, int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr_base.h:551:4
    #10 0x73c09e in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<UccProcess, std::allocator<UccProcess>, int&>(UccProcess*&, std::_Sp_alloc_shared_tag<std::allocator<UccProcess> >, int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr_base.h:683:6
    #11 0x73c09e in std::__shared_ptr<UccProcess, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<UccProcess>, int&>(std::_Sp_alloc_shared_tag<std::allocator<UccProcess> >, int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr_base.h:1376:14
    #12 0x73c09e in std::shared_ptr<UccProcess>::shared_ptr<std::allocator<UccProcess>, int&>(std::_Sp_alloc_shared_tag<std::allocator<UccProcess> >, int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr.h:408:4
    #13 0x73c09e in std::shared_ptr<UccProcess> std::allocate_shared<UccProcess, std::allocator<UccProcess>, int&>(std::allocator<UccProcess> const&, int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr.h:861:14
    #14 0x73c09e in std::shared_ptr<UccProcess> std::make_shared<UccProcess, int&>(int&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr.h:877:14
    #15 0x73c09e in UccJob::UccJob(int, UccJob::ucc_job_ctx_mode_t, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >) /home/runner/work/ucc/ucc/test/gtest/common/test_ucc.cc:301:25
    #16 0x259b72b in test_active_set_single_Test::TestBody() /home/runner/work/ucc/ucc/test/gtest/active_set/test_active_set.cc:171:15
    #17 0x71511a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:3924:10
    #18 0x71511a in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:3960:14
    #19 0x6a33f0 in testing::Test::Run() /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:3999:5
    #20 0x6a7484 in testing::TestInfo::Run() /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:4175:11
    #21 0x6a8c09 in testing::TestSuite::Run() /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:4307:28
    #22 0x6d36ca in testing::internal::UnitTestImpl::RunAllTests() /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:6833:44
    #23 0x717d41 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:3924:10
    #24 0x717d41 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:3960:14
    #25 0x6d272f in testing::UnitTest::Run() /home/runner/work/ucc/ucc/test/gtest/common/gtest-all.cc:6420:10
    #26 0x734955 in RUN_ALL_TESTS() /home/runner/work/ucc/ucc/test/gtest/common/gtest.h:14812:46
    #27 0x734955 in main /home/runner/work/ucc/ucc/test/gtest/common/main.cc:27:11
    #28 0x7f4e6ceda082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)
    #29 0x5c994d in _start (/home/runner/work/ucc/ucc/test/gtest/gtest+0x5c994d)

Objects leaked above:
0x602000012730 (4 bytes)
0x60200004e9d0 (5 bytes)
0x6020000578d0 (5 bytes)
0x602000072650 (4 bytes)
0x60200007bb90 (5 bytes)
0x60200009c730 (5 bytes)
0x6020000ba450 (4 bytes)
0x6020000d4e10 (4 bytes)
0x60200010c3d0 (4 bytes)
0x602000127bb0 (5 bytes)
0x60200012be10 (4 bytes)
0x60200014a750 (5 bytes)
0x60200014fe50 (5 bytes)
0x602000154610 (4 bytes)
0x602000155ff0 (4 bytes)
0x602000161f30 (5 bytes)
0x60200016bd70 (4 bytes)
0x602000179b90 (4 bytes)
0x6020001a0390 (5 bytes)
0x6020001ba2d0 (4 bytes)
0x6020001bffb0 (5 bytes)
0x6020001f3e30 (4 bytes)
0x60200020b470 (5 bytes)
0x6020002234d0 (5 bytes)
0x60200023dbf0 (4 bytes)
0x60200023e5b0 (4 bytes)
0x602000245db0 (4 bytes)
0x60200024d370 (5 bytes)
0x602000284a70 (4 bytes)
0x60200028d690 (4 bytes)
0x6020002ae2d0 (4 bytes)
0x6020002b1730 (4 bytes)
0x6020002c8ff0 (5 bytes)
0x6020002cda90 (5 bytes)
0x6020002eea70 (5 bytes)
0x6020002fc550 (5 bytes)
0x602000303910 (5 bytes)
0x602000314550 (5 bytes)
0x602000317af0 (4 bytes)
0x602000322090 (4 bytes)
0x602000322f90 (4 bytes)
0x602000326330 (4 bytes)
0x6020003295d0 (5 bytes)
0x6020003298d0 (5 bytes)
0x602000336150 (5 bytes)
0x602000354cf0 (4 bytes)
0x6020003754b0 (5 bytes)
0x602000376410 (4 bytes)
0x6020003944d0 (4 bytes)
0x6020003d34b0 (5 bytes)
0x6020003df0f0 (4 bytes)
0x6020003e72d0 (5 bytes)
0x6020003e7410 (4 bytes)
0x6020003e7930 (4 bytes)
0x6020003efd50 (5 bytes)
0x6020003f3170 (5 bytes)
0x6020004160d0 (5 bytes)
0x602000420e10 (5 bytes)
0x60200042f890 (5 bytes)
0x60200044c610 (4 bytes)
0x602000475fb0 (5 bytes)
0x602000484a10 (4 bytes)
0x602000491ab0 (4 bytes)
0x602000493510 (5 bytes)


SUMMARY: AddressSanitizer: 6575 byte(s) leaked in 789 allocation(s).
Error: Process completed with exit code 1.

@MamziB MamziB self-assigned this Feb 6, 2025
@janjust
Copy link
Collaborator

janjust commented Feb 7, 2025

awesome find and fix Mamzi, thanks for the PR

@MamziB MamziB force-pushed the mamzi/ci-leak-detect branch from d9e32a7 to 83635b0 Compare February 10, 2025 22:16
@janjust janjust force-pushed the mamzi/ci-leak-detect branch from 83635b0 to 13cddde Compare February 11, 2025 14:22
@MamziB
Copy link
Collaborator Author

MamziB commented Feb 11, 2025

@janjust can you please merge this? I rebased it yesterday, and not all checks have passed. Thank you

@janjust janjust merged commit 7405dfb into openucx:master Feb 11, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants