Retry mismatched Rpc requests with error response #804
Conversation
| c.log.Debugf("current ration retried against total %v/%v", c.numberOfRetriedRequests, c.totalNumberOfRequests) | ||
| c.numberOfRetriedRequests++ | ||
| state.Data.StateDB.IsRecovered = true | ||
| compareErr = retryRequest(state) |
There was a problem hiding this comment.
If I understand it correctly. If RPC request fails(/result is different), then the code attempts just 1 more time the same request? Wouldn't multiple retries be useful?
There was a problem hiding this comment.
My stance on this is to have our own opera archive running. Having multiple attempts doesn't guarantee stable result. If an error comes from the recording, a single retry connecting to a stable opera client should produce a correct result.
There was a problem hiding this comment.
If I understand it correctly. If RPC request fails(/result is different), then the code attempts just 1 more time the same request? Wouldn't multiple retries be useful?
Yes and no. If comparison fails, request is send to Rpc Gateway to obtain new response. Then it is compared again. From what I have tried, this has 100% recovery rate.
| @@ -108,6 +140,11 @@ func (c *rpcComparator) PostTransaction(state executor.State[*rpc.RequestAndResu | |||
|
|
|||
| // PostRun prints all caught errors. | |||
| func (c *rpcComparator) PostRun(executor.State[*rpc.RequestAndResults], *executor.Context, error) error { | |||
| // log only if continue on failure is enabled | |||
There was a problem hiding this comment.
I presume if error occurred and it got raised in Process then the PostRun isn't called at all. If so, does it even make call then to check in here?
There was a problem hiding this comment.
There is no way Process causes error. It can possibly cause a panic and in that case this would not be called because of how executor handles panics.
Co-authored-by: matejmlejnek <[email protected]>
cabrador
force-pushed
the
petr/retry-rpc-requests
branch
from
82205ac to
13257dd
Compare
Description
This PR adds feature that retries any missmatch in the
Rpc. This only works for records with error response. Valid response does not make sense to replay.