Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the race condition while calling rcl_shutdown #1353

Open
wants to merge 3 commits into
base: rolling
Choose a base branch
from
Open

Fix the race condition while calling rcl_shutdown #1353

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
54dcc50
Fix the race condition while calling rcl_shutdown
Barry-Xu-2018 Sep 3, 2024
40329f2
Avoid calling rcl_shutdown() multiple times on the same context
Barry-Xu-2018 Sep 3, 2024
0eb7ee5
Multiple calls to Context::shutdown will throw an exception
Barry-Xu-2018 Sep 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 7 additions & 10 deletions rclpy/src/rclpy/context.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,18 +150,15 @@ Context::ok()
void
Context::shutdown()
{
{
std::lock_guard<std::mutex> guard{g_contexts_mutex};
auto iter = std::find(g_contexts.begin(), g_contexts.end(), rcl_context_.get());
if (iter != g_contexts.end()) {
g_contexts.erase(iter);
std::lock_guard<std::mutex> guard{g_contexts_mutex};
auto iter = std::find(g_contexts.begin(), g_contexts.end(), rcl_context_.get());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the g_contexts is the collection of valid context, and this means if it cannot find the context in the g_contexts during this shutdown call, that is the invalid operation? so we can generate the exception without introducing already_shutdown_?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the g_contexts is the collection of valid context, and this means if it cannot find the context in the g_contexts during this shutdown call, that is the invalid operation?

Yes.
rc_context_ is added to g_contexts in constructor of Context. Before calling rcl_shutdown(), rc_context_ is removed from g_contexts.

so we can generate the exception without introducing already_shutdown_?

already_shutdown_ is only used for multiple calls to Context::shutdown() to throw the exception.

About issue reported by #1352, it describes a possible scenario. There are two threads to call rcl_shutdown for the same context. One call is from rclpy::shutdown() and the other from Context::shutdown(). In this situation, we do not want to get an exception in Context::shutdown().

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that is exactly why we have std::lock_guard<std::mutex> guard{g_contexts_mutex}; lock escalated to the this function scope? so that when rclpy::shutdown_contexts is under process, we will not find the iterator with this context, so generate the exception. i really do not understand to have already_shutdown_ flag internally...

if (iter != g_contexts.end()) {
g_contexts.erase(iter);
rcl_ret_t ret = rcl_shutdown(rcl_context_.get());
if (RCL_RET_OK != ret) {
throw RCLError("failed to shutdown");
}
}

rcl_ret_t ret = rcl_shutdown(rcl_context_.get());
if (RCL_RET_OK != ret) {
throw RCLError("failed to shutdown");
}
}

void define_context(py::object module)
Expand Down
9 changes: 0 additions & 9 deletions rclpy/test/test_init_shutdown.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,15 +74,6 @@ def test_double_init():
rclpy.shutdown(context=context)


def test_double_shutdown():
context = rclpy.context.Context()
rclpy.init(context=context)
assert context.ok()
rclpy.shutdown(context=context)
with pytest.raises(RuntimeError):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to remove this test?

I think we can keep this test, but expects RCLError exception here to catch from 2nd rcl_shutdown (internally RCL_RET_ALREADY_SHUTDOWN returns in rcl) ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your comments.

I think we can keep this test, but expects RCLError exception here to catch from 2nd rcl_shutdown (internally RCL_RET_ALREADY_SHUTDOWN returns in rcl) ?

With the current fix, if the context is not in g_contexts, rcl_shutdown will not be called.

According to the issue described in #1352, it is indeed possible for two threads to call rcl_shutdown on the same context. However, one call is from rclpy::shutdown() and the other from Context::shutdown(). In this situation, we do not want to see an exception being thrown.

The test calls Context::shutdown() twice, and it is reasonable to expect an exception on the second call. Therefore, I am considering setting a flag variable in the Context class. If Context::shutdown() has been successfully called, this variable will be set. If Context::shutdown() is called again, it will check if the flag has been set and, if so, will throw an exception CONTEXT_ALREADY_SHUTDOWN (not from RCL). What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Therefore, I am considering setting a flag variable in the Context class. If Context::shutdown() has been successfully called, this variable will be set. If Context::shutdown() is called again, it will check if the flag has been set and, if so, will throw an exception CONTEXT_ALREADY_SHUTDOWN (not from RCL). What do you think?

Please review 0eb7ee5

rclpy.shutdown(context=context)


def test_create_node_without_init():
context = rclpy.context.Context()
with pytest.raises(NotInitializedException):
Expand Down