CBG-4067: do not release doc sequence upon timeout error

[gregns1]

CBG-4067

• Added check to only release doc sequences when a write error is not a timeout error.
• Added some functionality to leaky bucket to force return a timeout error upon a successful write (open to ideas of better forcing a timeout here)
• Test also covers us correctly releasing a doc sequence upon a different et error at write time by casuing conflict uszing leaky bucket functionality

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

• GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2617/

[torcolvin]

This LGTM, just some questions about the tests, and some code nits you can take or leave.

[torcolvin]

This is interesting because this definitely means the write occurred. However, in the case of this ticket, we don't know whether the write went through or not? Is there a reason to actually have this error after it writes and not before?

[gregns1]

I want this behaviour as this is what the customer saw. See ticket linked to this for more info but they had write, server under load and operation times out, but write eventually succeeds but we've timeout and release the document sequence. Then then the successful write is ignored at cache due to duplicate sequence with the released sequence.

It certainly seems odd to do it this way but its the only way I could think of mocking/forcing this situation reliably in a test. We cannot be sure when we hit timeout that the write will eventually go through or not but the changes this PR makes will protect against docs being missed in replications if the write does succeed.