Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixGwhisperHangOnDescDbStreamClose #147

Merged
merged 5 commits into from
Nov 20, 2023

Conversation

abb3r
Copy link

@abb3r abb3r commented Nov 16, 2023

close reflection stream with a deadline after fetching services description data.

  • Dynamic deadlines are now propagated to the DescDb stream close.

  • The db stream is now closed after 2 (after fetching name and type info from server).

@abb3r abb3r changed the title close refelction stream with a deadline after fetching services descr… fixGwhisperHangOnDescDbStreamClose Nov 16, 2023
Rahman Abber Tahir added 2 commits November 16, 2023 15:52
@abb3r abb3r force-pushed the fixGwhisperHangOnDescDbStreamClose branch from afab6ce to 41810a5 Compare November 16, 2023 14:52
Copy link
Member

@rainerschoe rainerschoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did I understand this right: You are not setting the reflection stream deadline, instead you are trying to close it with a deadline, if this fails you bail out.

Looging good, just the thing about the cache.

@@ -350,12 +350,27 @@ void DescDbProxy::getDescriptors(const std::string &f_hostAddress)
}
}

grpc::Status DescDbProxy::closeDescDbStream(std::optional<std::chrono::time_point<std::chrono::system_clock>> deadline)
{
if( m_disableCache == false )//cache enabled, no reflection stream required.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a slippery slope.
If cache is enabled, we will still use the reflection stream, if the cache entry is outdated or if the cache does not hold data for the host.
see line 330

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I see it. I change the condition to give prefence to m_reflectionDescDb check. if the ptr is valid we close the stream any how.



//before calling the RPC, close the DescDb connection with a timeout.
grpc::Status dbDescStatus = ConnectionManager::getInstance().closeDescDbWithDeadline(serverAddress, deadline);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was about to suggest moving all the closing into destructors and just drop the descriptor database / proxy when we do not need it any more.
However the ConnectionManager singleton makes life hard here. gWhisper does not have the most beautiful design :-(
So I am fine with it as is. what do you think?

Copy link
Author

@abb3r abb3r Nov 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah ConnectionManager actually makes it hard. since we are also using shared ptrs here we would not know how many owners are there for the proxy and when will it actually closes.
Doing explicitly allows us to check the grpc status/result from calling the stream finish() aswell.

@abb3r
Copy link
Author

abb3r commented Nov 17, 2023

Did I understand this right: You are not setting the reflection stream deadline, instead you are trying to close it with a deadline, if this fails you bail out.

Looging good, just the thing about the cache.

Yes we close the reflection stream after fecthing type/service info this we we don't couple the timeouts between reflection & main rpc. We also get the grpc status/result from the stream close by doing it early.

rainerschoe
rainerschoe previously approved these changes Nov 20, 2023
Copy link
Member

@rainerschoe rainerschoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good now! I think it would be easier to just ignore the error when we try to close reflection stream when it was not opened.
wdyt? (feel free to merge as is, or update again)

if ( m_reflectionDescDb == nullptr )
{
if( m_disableCache == false )//cache enabled, no reflection stream required.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just remove the check and return once m_reflectionDescDb == nullptr yields true? (meaning replace whe whole outer if body with a return;)

Sotty github woudn't let me write a suggestion for this.

Copy link
Author

@abb3r abb3r Nov 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, the unique ptr == null garauntees the stream is already closed. that is actually a valid case.

Copy link
Member

@rainerschoe rainerschoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM Thanks for the effort! Let me know if this helps long-term and catches all scenarios.

@rainerschoe rainerschoe merged commit 9d7a9d6 into IBM:master Nov 20, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants