Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CP-48676: Reuse pool sessions on slave logins #6258

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

snwoods
Copy link
Contributor

@snwoods snwoods commented Jan 28, 2025

This was first introduced in #5986 but was later reverted as the mutex it introduced caused a deadlock with the DB mutex. This PR has now been reworked to use an Atomic variable, removing the need for the mutex.

This PR allows pool sessions to be reused (if the flag is turned on, which it is by default). This greatly speeds up communications between hosts which is important for repetitive calls e.g. one customer was running get_servertime frequently on all hosts in a pool which was causing problems due to the length of each call.

When we get this reusable session, we can optionally validate the session before using it but this increases the duration of the call, so this is off by default. Excepting a toolstack restart, the reusable session should always be valid as it is excluded from deletion in destroy_db_session

@snwoods snwoods force-pushed the private/stevenwo/CA-400339 branch from 81c5fd5 to f17fb1b Compare January 28, 2025 12:53
Copy link
Member

@psafont psafont left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor style issues. The general approach seems quite sensible, even if a race can cause xapi to use more than one session (to avoid a loop when getting an invalid resusable session)

if session <> Ref.null && is_valid_session session then (
if
session <> Ref.null
&& ((not !Xapi_globs.validate_reusable_pool_session)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the flag check to is_valid as well

@snwoods snwoods force-pushed the private/stevenwo/CA-400339 branch from f17fb1b to b94335a Compare January 31, 2025 15:52
@snwoods
Copy link
Contributor Author

snwoods commented Feb 3, 2025

The Ring3: BVT + BST suiteruns I have run were all green, both with (211574) and without (211571) additional debug logging. The debug suiterun shows that the pool sessions are being reused as expected.

try
(* Call an API function to check the session is still valid *)
let rpc = Helpers.make_rpc ~__context in
ignore (Client.Pool.get_all ~rpc ~session_id) ;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this call reasonably cheap (since the result is thrown away)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the same call that is used after creating a session to force the time to update (see line 666 above), so I assume it is the cheapest call we can do. It does take some time though, which is why I've disabled calling is_valid_session by default (!Xapi_globs.validate_reusable_pool_session defaults to false)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During testing (BVT + BST etc.) the resuable_pool_session never became invalid because we prevent the resuable_pool_session from being destroyed in destroy_db_session. Something I'm unsure about though is, is there any possible situation where the session does become invalid (excluding a toolstack restart where a new session will be made anyway)?
If that were to happen and Xapi_globs.validate_reusable_pool_session = false, is_valid_session would still return true (as session != null) and whatever the session is used for would fail. I'm unsure if we would then get out of this state.

@snwoods snwoods force-pushed the private/stevenwo/CA-400339 branch from b94335a to 3efb3bb Compare February 4, 2025 11:48
@snwoods snwoods force-pushed the private/stevenwo/CA-400339 branch from 3efb3bb to 151686b Compare February 20, 2025 09:44
@snwoods
Copy link
Contributor Author

snwoods commented Feb 26, 2025

During testing (BVT + BST etc.) the resuable_pool_session never became invalid because we prevent the resuable_pool_session from being destroyed in destroy_db_session. Something I'm unsure about though is, is there any possible situation where the session does become invalid (excluding a toolstack restart where a new session will be made anyway)? If that were to happen and Xapi_globs.validate_reusable_pool_session = false, is_valid_session would still return true (as session != null) and whatever the session is used for would fail. I'm unsure if we would then get out of this state.

I've marked all the comments as resolved where appropriate. This is still an outstanding question for me though.

Prevent this reusable pool session from being destroyed so that it
remains valid. This feature can be toggled with the flag reuse-pool-sessions.

This commit has been reworked to use Atomic instead of a mutex to
prevent a deadlock as shown in CA-400339.

Signed-off-by: Steven Woods <[email protected]>
Add a new flag validate-reusable-pool-session to xapi globs which skips
the reusable pool session validity check if it is false. This saves time
as we are no longer calling pool.get_all for each session.

Signed-off-by: Steven Woods <[email protected]>
@snwoods snwoods force-pushed the private/stevenwo/CA-400339 branch from 151686b to c6da552 Compare February 26, 2025 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants