Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add label to staleness metrics to distinguish published vs duplicate entries #93

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

eparker-tulip
Copy link
Contributor

@eparker-tulip eparker-tulip commented Jan 21, 2025

Oplogtoredis runs two replicas in parallel which both write to redis, with dedup being done by a lua script. However, both of these replicas report the latency in time to reach redis, regardless of whether it was the first to reach redis or not. This means that is one replica is lagging behind, it will report larger latency values than what’s actually reaching redis, and we don’t have a way to filter these since they will be reported at a later time than the earlier successful write from the other replica.

This adds a return value to the dedup script and uses it as a label for the latency metrics.

Note: for some reason returning true/false from the lua script was causing the performance and fault_injection tests to fail (after an extra long time). Switching to int, returning 1 or 2, worked. Testing showed that returning false was causing issues, and a search found this which seems to align with this issue, where the lua script returning false is treated as an error, which then caused OTR to retry (until the dedupe token expired). I opened an issue with go-redis.

@eparker-tulip eparker-tulip requested review from alex-goodisman and torywheelwright and removed request for alex-goodisman January 22, 2025 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant