api_skyportal: no_retry version #442
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ng into concurrency issues when the session retries sending a request to SkyPortal that might still be processing the first one. We use it paired with a longer timeout when sending followup requests
|
@nabeelre just so you know that issue is being addressed. |
|
we might want to keep using a shorter timeout though (maybe 10 s) to try to minimize these concurrently issues when 2 alerts are being triggered on at once (for the same object). |
nabeelre
reviewed
Oct 9, 2024
| "POST", | ||
| "/api/followup_request", | ||
| passed_filter["auto_followup"]["data"], | ||
| timeout=30, |
There was a problem hiding this comment.
Jamie said last night's request took 42 seconds to process. With a 30 second timeout we'd mark the trigger as failed via timeout but it would actually succeed later? Maybe up this to 60 seconds to be conservative?
There was a problem hiding this comment.
I'm honestly not quite sure where his number came from, since we got a response within 5 to 6 seconds, and definitely didn't wait 42 seconds. I would proceed with these numbers while we investigate this further w/ Jamie
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
create a no-retry version of the api_skyportal method, to avoid running into concurrency issues when the session retries sending a request to SkyPortal (because it took longer than the specified time out to respond to the client) when the initial one is still being processed. This is fine and can safely happen for pretty much everything, except follow-up requests.
If we tell SkyPortal to trigger an instrument, after 5 seconds without a response (the default timeout) decide to resent that request but SkyPortal was almost done (and is still) processing it, we might end up sending 2 requests at the same time and creating duplication issues.
With this PR, we can use much longer timeouts (here we try 30 seconds) while avoiding any retries when sending a follow-up request.
PS:
We can still run into a concurrency issue of course (where 2 alerts of the same object get processed at the same time, and worker B tries to trigger on alert 2 at the same time as Worker A is triggering on alert 1) and we already have logic in SkyPortal to avoid that, but if the distant server SkyPortal is sending the request to is taking too long is becomes a risk. In a future set of SkyPortal+Kowalski+Fritz PRs, we can consider posting the request in the DB in a "processing" state as soon as possible so that even before we start waiting for a distance server to answer, other processed can know that we are actively trying to send an identical request and they should cancel sending anything.