Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make ConsoleStream/GeneralRegistry thread-safe #27996

Merged
merged 3 commits into from
Jun 27, 2024

Conversation

lindsayad
Copy link
Member

@lindsayad lindsayad commented Jun 26, 2024

  • Use the same mutex for both operator<< overloads.
  • Use a deque instead of a vector for general registry items as the former will not have its references invalidated

Refs CIVET failure https://civet.inl.gov/job/2295949/ on libMesh/libmesh#3883

Closes #23926

Use the same mutex for both `operator<<` overloads. Refs CIVET
failure https://civet.inl.gov/job/2295949/ on libMesh/libmesh#3883
@lindsayad lindsayad marked this pull request as ready for review June 26, 2024 14:13
@lindsayad lindsayad requested a review from roystgnr as a code owner June 26, 2024 14:13
@lindsayad lindsayad self-assigned this Jun 26, 2024
Copy link
Contributor

@roystgnr roystgnr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoa, good catch.

@roystgnr
Copy link
Contributor

Assuming it's the right catch, this is great... how on Earth did you debug that so quickly? Was it actually easy to reproduce??

@lindsayad
Copy link
Member Author

yea thread sanitizer made it pretty easy

WARNING: ThreadSanitizer: data race (pid=1601763)
  Write of size 8 at 0x7b5400036b00 by thread T25 (mutexes: write M0):
    #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:899 (libtsan.so.2+0x42899) (BuildId: cf5585f598b67115e57d8d7b41c8ccf7af1a689a)
    #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:891 (libtsan.so.2+0x42899)
    #2 std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) <null> (libstdc++.so.6+0x1571fd) (BuildId: 7e8507d31f51627e18fc00bd04ee5a2e01069a96)
    #3 SolutionInvalidity::printDebug(unsigned int) const /home/lindad/projects/thread-debugging/framework/src/utils/SolutionInvalidity.C:142 (libmoose-devel.so.0+0x15af1dd) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #4 SolutionInvalidInterface::flagInvalidSolutionInternal(unsigned int) const /home/lindad/projects/thread-debugging/framework/src/interfaces/SolutionInvalidInterface.C:30 (libmoose-devel.so.0+0x2db069c) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #5 NonsafeMaterial::computeQpProperties() /home/lindad/projects/thread-debugging/test/src/materials/NonsafeMaterial.C:57 (libmoose_test-devel.so.0+0xfc03bb) (BuildId: 48001259c4f518342ff7372a9c9d6b1fe0007d4d)
    #6 Material::computeProperties() /home/lindad/projects/thread-debugging/framework/src/materials/Material.C:139 (libmoose-devel.so.0+0x3907432) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #7 void MaterialData::reinit<std::vector<std::shared_ptr<MaterialBase>, std::allocator<std::shared_ptr<MaterialBase> > > >(std::vector<std::shared_ptr<MaterialBase>, std::allocator<std::shared_ptr<MaterialBase> > > const&) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/MaterialData.h:329 (libmoose-devel.so.0+0x2bcd323) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #8 FEProblemBase::reinitMaterials(unsigned short, unsigned int, bool) /home/lindad/projects/thread-debugging/framework/src/problems/FEProblemBase.C:3668 (libmoose-devel.so.0+0x2bcd323)
    #9 ThreadedElementLoop<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> >::prepareElement(libMesh::Elem const*) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/ThreadedElementLoop.h:190 (libmoose-devel.so.0+0x27f1f08) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #10 NonlinearThread::onElement(libMesh::Elem const*) /home/lindad/projects/thread-debugging/framework/src/loops/NonlinearThread.C:127 (libmoose-devel.so.0+0x2798346) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #11 ThreadedElementLoopBase<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> >::operator()(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&, bool) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/ThreadedElementLoopBase.h:256 (libmoose-devel.so.0+0x126589a) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #12 NonlinearThread::operator()(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&, bool) /home/lindad/projects/thread-debugging/framework/src/loops/NonlinearThread.C:60 (libmoose-devel.so.0+0x27a8e7e) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #13 void* libMesh::Threads::run_body<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*>, ComputeResidualThread>(void*) /home/lindad/projects/thread-debugging/scripts/../libmesh/installed/include/libmesh/threads_pthread.h:236 (libmoose-devel.so.0+0x37b205c) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)

  Previous read of size 8 at 0x7b5400036b00 by thread T24 (mutexes: write M1, write M2):
    #0 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:899 (libtsan.so.2+0x42899) (BuildId: cf5585f598b67115e57d8d7b41c8ccf7af1a689a)
    #1 memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:891 (libtsan.so.2+0x42899)
    #2 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) <null> (libstdc++.so.6+0x158a43) (BuildId: 7e8507d31f51627e18fc00bd04ee5a2e01069a96)
    #3 ConsoleStream::operator<<(std::ostream& (* const&)(std::ostream&)) const /home/lindad/projects/thread-debugging/framework/src/outputs/ConsoleStream.C:34 (libmoose-devel.so.0+0x2a60317) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)
    #4 SolutionInvalidity::printDebug(unsigned int) const /home/lindad/projects/thread-debugging/framework/src/utils/SolutionInvalidity.C:142 (libmoose-devel.so.0+0x15af24a) (BuildId: f24aca780a9805c304cc3f1db4ac1bb14e8fe35c)

@moosebuild
Copy link
Contributor

moosebuild commented Jun 26, 2024

Job Documentation on 174d8a9 wanted to post the following:

View the site here

This comment will be updated on new commits.

@moosebuild
Copy link
Contributor

moosebuild commented Jun 26, 2024

Job Coverage on 174d8a9 wanted to post the following:

Framework coverage

d964da #27996 174d8a
Total Total +/- New
Rate 85.06% 85.06% +0.00% 100.00%
Hits 104413 104417 +4 4
Misses 18334 18334 - 0

Diff coverage report

Full coverage report

Modules coverage

Coverage did not change

Full coverage reports

Reports

This comment will be updated on new commits.

@lindsayad
Copy link
Member Author

Ok new one for me to work out. Only triggered this with TSAN with 16 threads

WARNING: ThreadSanitizer: data race (pid=1651682)
  Write of size 8 at 0x7b14000eca40 by thread T69 (mutexes: write M0, write M1):
    #0 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_data(char*) /usr/include/c++/12/bits/basic_string.h:224 (libmoose-devel.so.0+0x1724536) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #1 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) /usr/include/c++/12/bits/basic_string.h:696 (libmoose-devel.so.0+0x1724536)
    #2 moose::internal::SolutionInvalidityName::SolutionInvalidityName(moose::internal::SolutionInvalidityName&&) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/SolutionInvalidityRegistry.h:34 (libmoose-devel.so.0+0x1724536)
    #3 moose::internal::SolutionInvalidityInfo::SolutionInvalidityInfo(moose::internal::SolutionInvalidityInfo&&) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/SolutionInvalidityRegistry.h:71 (libmoose-devel.so.0+0x1724536)
    #4 void std::__new_allocator<moose::internal::SolutionInvalidityInfo>::construct<moose::internal::SolutionInvalidityInfo, moose::internal::SolutionInvalidityInfo>(moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo&&) /usr/include/c++/12/bits/new_allocator.h:175 (libmoose-devel.so.0+0x1724536)
    #5 void std::allocator_traits<std::allocator<moose::internal::SolutionInvalidityInfo> >::construct<moose::internal::SolutionInvalidityInfo, moose::internal::SolutionInvalidityInfo>(std::allocator<moose::internal::SolutionInvalidityInfo>&, moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo&&) /usr/include/c++/12/bits/alloc_traits.h:516 (libmoose-devel.so.0+0x1724536)
    #6 void std::__relocate_object_a<moose::internal::SolutionInvalidityInfo, moose::internal::SolutionInvalidityInfo, std::allocator<moose::internal::SolutionInvalidityInfo> >(moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, std::allocator<moose::internal::SolutionInvalidityInfo>&) /usr/include/c++/12/bits/stl_uninitialized.h:1064 (libmoose-devel.so.0+0x1724536)
    #7 moose::internal::SolutionInvalidityInfo* std::__relocate_a_1<moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, std::allocator<moose::internal::SolutionInvalidityInfo> >(moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, std::allocator<moose::internal::SolutionInvalidityInfo>&) /usr/include/c++/12/bits/stl_uninitialized.h:1092 (libmoose-devel.so.0+0x1724536)
    #8 moose::internal::SolutionInvalidityInfo* std::__relocate_a<moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, std::allocator<moose::internal::SolutionInvalidityInfo> >(moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, std::allocator<moose::internal::SolutionInvalidityInfo>&) /usr/include/c++/12/bits/stl_uninitialized.h:1133 (libmoose-devel.so.0+0x1724536)
    #9 std::vector<moose::internal::SolutionInvalidityInfo, std::allocator<moose::internal::SolutionInvalidityInfo> >::_S_relocate(moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, moose::internal::SolutionInvalidityInfo*, std::allocator<moose::internal::SolutionInvalidityInfo>&) /usr/include/c++/12/bits/stl_vector.h:504 (libmoose-devel.so.0+0x1724536)
    #10 void std::vector<moose::internal::SolutionInvalidityInfo, std::allocator<moose::internal::SolutionInvalidityInfo> >::_M_realloc_insert<moose::internal::SolutionInvalidityInfo>(__gnu_cxx::__normal_iterator<moose::internal::SolutionInvalidityInfo*, std::vector<moose::internal::SolutionInvalidityInfo, std::allocator<moose::internal::SolutionInvalidityInfo> > >, moose::internal::SolutionInvalidityInfo&&) /usr/include/c++/12/bits/vector.tcc:474 (libmoose-devel.so.0+0x1724536)
    #11 moose::internal::SolutionInvalidityInfo& std::vector<moose::internal::SolutionInvalidityInfo, std::allocator<moose::internal::SolutionInvalidityInfo> >::emplace_back<moose::internal::SolutionInvalidityInfo>(moose::internal::SolutionInvalidityInfo&&) /usr/include/c++/12/bits/vector.tcc:123 (libmoose-devel.so.0+0x1636510) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #12 registerItem<const moose::internal::SolutionInvalidityRegistry::registerInvalidity(const std::string&, const std::string&)::<lambda(std::size_t)> > /home/lindad/projects/thread-debugging/framework/build/header_symlinks/GeneralRegistry.h:160 (libmoose-devel.so.0+0x1636510)
    #13 moose::internal::SolutionInvalidityRegistry::registerInvalidity(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/lindad/projects/thread-debugging/framework/src/utils/SolutionInvalidityRegistry.C:39 (libmoose-devel.so.0+0x1636510)
    #14 SolutionInvalidInterface::registerInvalidSolutionInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /home/lindad/projects/thread-debugging/framework/src/interfaces/SolutionInvalidInterface.C:37 (libmoose-devel.so.0+0x2db06f9) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #15 NonsafeMaterial::computeQpProperties() /home/lindad/projects/thread-debugging/test/src/materials/NonsafeMaterial.C:59 (libmoose_test-devel.so.0+0xfc04c1) (BuildId: c0ef1972f8270dcf8c16adf82de5474aab51a8b1)
    #16 Material::computeProperties() /home/lindad/projects/thread-debugging/framework/src/materials/Material.C:139 (libmoose-devel.so.0+0x3907432) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #17 void MaterialData::reinit<std::vector<std::shared_ptr<MaterialBase>, std::allocator<std::shared_ptr<MaterialBase> > > >(std::vector<std::shared_ptr<MaterialBase>, std::allocator<std::shared_ptr<MaterialBase> > > const&) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/MaterialData.h:329 (libmoose-devel.so.0+0x2bcd323) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #18 FEProblemBase::reinitMaterials(unsigned short, unsigned int, bool) /home/lindad/projects/thread-debugging/framework/src/problems/FEProblemBase.C:3668 (libmoose-devel.so.0+0x2bcd323)
    #19 ThreadedElementLoop<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> >::prepareElement(libMesh::Elem const*) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/ThreadedElementLoop.h:190 (libmoose-devel.so.0+0x27f1f08) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #20 NonlinearThread::onElement(libMesh::Elem const*) /home/lindad/projects/thread-debugging/framework/src/loops/NonlinearThread.C:127 (libmoose-devel.so.0+0x2798346) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #21 ThreadedElementLoopBase<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> >::operator()(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&, bool) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/ThreadedElementLoopBase.h:256 (libmoose-devel.so.0+0x126589a) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #22 NonlinearThread::operator()(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&, bool) /home/lindad/projects/thread-debugging/framework/src/loops/NonlinearThread.C:60 (libmoose-devel.so.0+0x27a8e7e) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #23 void* libMesh::Threads::run_body<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*>, ComputeResidualThread>(void*) /home/lindad/projects/thread-debugging/scripts/../libmesh/installed/include/libmesh/threads_pthread.h:236 (libmoose-devel.so.0+0x37b205c) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)

  Previous read of size 8 at 0x7b14000eca40 by thread T70 (mutexes: write M2):
    #0 ConsoleStream const& ConsoleStream::operator<< <std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const /home/lindad/projects/thread-debugging/framework/build/header_symlinks/ConsoleStream.h:110 (libmoose-devel.so.0+0x1265175) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #1 SolutionInvalidity::printDebug(unsigned int) const /home/lindad/projects/thread-debugging/framework/src/utils/SolutionInvalidity.C:142 (libmoose-devel.so.0+0x15af1dd) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #2 SolutionInvalidInterface::flagInvalidSolutionInternal(unsigned int) const /home/lindad/projects/thread-debugging/framework/src/interfaces/SolutionInvalidInterface.C:30 (libmoose-devel.so.0+0x2db069c) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #3 NonsafeMaterial::computeQpProperties() /home/lindad/projects/thread-debugging/test/src/materials/NonsafeMaterial.C:57 (libmoose_test-devel.so.0+0xfc03bb) (BuildId: c0ef1972f8270dcf8c16adf82de5474aab51a8b1)
    #4 Material::computeProperties() /home/lindad/projects/thread-debugging/framework/src/materials/Material.C:139 (libmoose-devel.so.0+0x3907432) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #5 void MaterialData::reinit<std::vector<std::shared_ptr<MaterialBase>, std::allocator<std::shared_ptr<MaterialBase> > > >(std::vector<std::shared_ptr<MaterialBase>, std::allocator<std::shared_ptr<MaterialBase> > > const&) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/MaterialData.h:329 (libmoose-devel.so.0+0x2bcd323) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #6 FEProblemBase::reinitMaterials(unsigned short, unsigned int, bool) /home/lindad/projects/thread-debugging/framework/src/problems/FEProblemBase.C:3668 (libmoose-devel.so.0+0x2bcd323)
    #7 ThreadedElementLoop<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> >::prepareElement(libMesh::Elem const*) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/ThreadedElementLoop.h:190 (libmoose-devel.so.0+0x27f1f08) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #8 NonlinearThread::onElement(libMesh::Elem const*) /home/lindad/projects/thread-debugging/framework/src/loops/NonlinearThread.C:127 (libmoose-devel.so.0+0x2798346) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #9 ThreadedElementLoopBase<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> >::operator()(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&, bool) /home/lindad/projects/thread-debugging/framework/build/header_symlinks/ThreadedElementLoopBase.h:256 (libmoose-devel.so.0+0x126589a) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #10 NonlinearThread::operator()(libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*> const&, bool) /home/lindad/projects/thread-debugging/framework/src/loops/NonlinearThread.C:60 (libmoose-devel.so.0+0x27a8e7e) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)
    #11 void* libMesh::Threads::run_body<libMesh::StoredRange<libMesh::MeshBase::const_element_iterator, libMesh::Elem const*>, ComputeResidualThread>(void*) /home/lindad/projects/thread-debugging/scripts/../libmesh/installed/include/libmesh/threads_pthread.h:236 (libmoose-devel.so.0+0x37b205c) (BuildId: 97a2f3d8bd8d7ba36c75fbdc0aa8e8747d7c6a45)

@loganharbour
Copy link
Member

I believe that the above is indicative of another significant problem. I've seen those solution invalidity tests fail in threads on occasion

@lindsayad
Copy link
Member Author

Pretty sure this is what is killing us using std::vector::emplace_back:

If after the operation the new size() is greater than old capacity() a reallocation takes place, in which case all iterators (including the end() iterator) and all references to the elements are invalidated. Otherwise only the end() iterator is invalidated.

@lindsayad
Copy link
Member Author

Yea the stack shows the realloc calls

This way when we release references into the wild we don't have to
worry about them ever being invalidated
@lindsayad lindsayad changed the title Make ConsoleStream thread-safe Make ConsoleStream/GeneralRegistry thread-safe Jun 26, 2024
@roystgnr
Copy link
Contributor

Wait, does std::deque fix the problem here? "If the new size is bigger than the old one - all iterators are invalidated." still applies there, no? You need std::list if you want an arbitrarily-ordered container that can be extended without invalidating.

@lindsayad
Copy link
Member Author

iterators are invalidated but not references. From emplace_back:

Appends a new element to the end of the container. The element is constructed through std::allocator_traits::construct, which typically uses placement-new to construct the element in-place at the location provided by the container. The arguments args... are forwarded to the constructor as std::forward(args)....

All iterators (including the end() iterator) are invalidated. No references are invalidated.

@lindsayad
Copy link
Member Author

Which makes sense because

The storage of a deque is automatically expanded and contracted as needed. Expansion of a deque is cheaper than the expansion of a std::vector because it does not involve copying of the existing elements to a new memory location.

@roystgnr
Copy link
Contributor

Oh that's wild. Expanding a deque can force a reallocation of metadata, so iterators get invalidated, but not of data, so references don't get invalidated. My apologies, carry on.

@lindsayad lindsayad merged commit 49c3751 into idaholab:next Jun 27, 2024
47 checks passed
@lindsayad lindsayad deleted the make-console-stream-thread-safe branch June 27, 2024 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Solution invalidity reporter issue with mpi/threads
4 participants