Reduce errors in workers #3962
Merged
Reduce errors in workers #3962
+48
−34
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| return basic_retry_wrapper( | ||
| make_slack_api_rate_limited(_make_slack_api_call_logged(call)) | ||
| )(**kwargs) | ||
| return basic_retry_wrapper(make_slack_api_rate_limited(call))(**kwargs) |
There was a problem hiding this comment.
do we want to completely remove the logging? or should we truncate or shorten it? I can see us wanting to have debug logging here occasionally for troubleshooting purposes, but it definitely is not helpful to be at the current level of spam by default.
| # Then we upsert the document's external permissions in postgres | ||
| try: | ||
| # Add the users to the DB if they don't exist | ||
| batch_add_ext_perm_user_if_not_exists( |
There was a problem hiding this comment.
I think it's more appropriate for the function itself to be more resilient to the type of error that is occurring here rather than the caller being responsible. It should be entirely possible for this function to "just work" and it seems better if we write it as such.
Couple of ideas for batch_add_ext_perm_user_if_not_exists:
- Catch the IntegrityError if the batch commit fails and go one by one while deliberately ignoring the failure we know can happen.
- Use Postgres ON CONFLICT DO NOTHING ... could work?
pablonyx
force-pushed
the
monitoring_errorspam
branch
from
5a709b5 to
2c36dd1
Compare
rkuo-danswer
approved these changes
Feb 13, 2025
github-merge-queue bot
pushed a commit
that referenced
this pull request
Feb 13, 2025
* reduce errors in workers * quick nit * update * nit
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is a general pass at reducing the number of unnecessary / spammy error logs in our backgrounds pods. Each exception has a link to a relevant example in Grafana + the relevant change
Primary Worker
Tenant ID exceptions
Fixed miscellaneous tenant ID exceptions
Monitoring worker
Entity exception
Heavy Worker
Dupe external email issue
Error ended up being that we have concurrent tasks creating the same external user email
Some pretty excessive Slack spam https://g-dfa4b7be0c.grafana-workspace.us-east-2.amazonaws.com/goto/Jnc1ujFHR?orgId=1)
Light Worker
View in Grafana
How Has This Been Tested?
Backporting (check the box to trigger backport action)
Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.