Skip to content
This repository has been archived by the owner on May 3, 2022. It is now read-only.

Add double timeout guard to lazy out req #172

Open
Raynos opened this issue Nov 3, 2015 · 1 comment
Open

Add double timeout guard to lazy out req #172

Raynos opened this issue Nov 3, 2015 · 1 comment

Comments

@Raynos
Copy link
Contributor

Raynos commented Nov 3, 2015

        if (self.operations) {
            self.operations.checkLastTimeoutTime(now);
            self.operations.popOutReq(self.id);
        }

We had an issue where on all affinity nodes we had about 0.3 timeouts per second to an individual peer for service FOO.

In the eager out_request path we have checkLastTimeoutTime() to kill connections that are timing out.

We need to add this check back to the lazy out request path.

cc @blampe @jcorbin

@Raynos
Copy link
Contributor Author

Raynos commented Nov 3, 2015

Other solutions to consider when a single service peer for service FOO is unhealthy:

  • Ship partial affinity, less peers === better least pending ?
  • Reduce kValue, less affinity nodes === better local decisions ?
  • Penalize timeout error frames in least pending ?
  • Bring back per-peer health state machine ?

@jcorbin jcorbin changed the title Add double timeout gaurd to lazy out req Add double timeout guard to lazy out req Nov 4, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant