Retry failed transactions and adjust the cached nonce

[iFrostizz]

Context

awm-relayer/vms/evm/destination_client.go

Lines 185 to 197 in f7093cb

	if err := c.client.SendTransaction(context.Background(), signedTx); err != nil {
	c.logger.Error(
	"Failed to send transaction",
	zap.Error(err),
	)
	return common.Hash{}, err
	}
	c.logger.Info(
	"Sent transaction",
	zap.String("txID", signedTx.Hash().String()),
	zap.Uint64("nonce", c.currentNonce),
	)
	c.currentNonce++

If the transaction sending returns an error, or if the transaction is dropped, the transaction does not appear do be sent again. It would happen though as pointed by @geoff-vball when restarting the node and it would catch up those transactions.

Description
Rather than treating a dropped transaction as fatal, we could add sent transactions to a pending queue and retry them in this scenario. If they are accepted (i.e. they produce a receipt), then they can be safely removed from the pending queue.

Expected behavior
If the transaction failed (and return an error) because of a revert, mark the transaction as "executed" and don't try to execute it again.
If it was dropped, these should have been placed in a queue and retried later while reverting the nonce or fetch it.

[geoff-vball]

Just a note: c.currentNonce is not incremented in the error case.

[iFrostizz]

Just a note: c.currentNonce is not incremented in the error case.

Oops, phrased this incorrectly, thanks! So one case that may trigger this code path is if the transaction was sent correctly but was dropped

[cam-schultz]

This is not a bug. This case is intentionally treated as fatal. The block that contained the message on the sending chain will not be considered processed if the tx on the destination is dropped, and will be re-processed on the next startup.

That said, gracefully handling dropped transactions would improve overall throughput, and could make a sizable difference for certain applications where the rate of dropped transactions is high, and end-to-end latency is important. I'm going to convert this ticket to a feature request, and edit the fields of the issue description accordingly.