25% signature generator failed with signature failed to be produced: AssertionFailed("signature failed to verify")

[ppca]

gcp log showing signature failed to be produced, the error message is in the err field in jsonpayload.
this is part of investigating why dev signature success rate is <95%.
Each hour there are 50-60 signature generators failed because of this error.
Each hour there are 110-130 total signature generators failed.
Each hour there are ~110-130 new signature generators started, and 100-130 restarted signature generators.
So basically AssertionFailed("signature failed to verify") constitutes 50% of all failed signature generators, and 25% of all signature generators.

[volovyks]

Here is the occurrence of this error during the last 30 days:

It happens at the same time we have issues with triple generation when the system is under load. See #923

Maybe the reason is not at triple or presignature levels.