retry only if processing completed for a packet

[agouin]

relayer/relayer/processor/message_processor.go

Line 323 in be5a4cf

retries := dst.trackProcessingMessage(t)

here we track the height where we are "processing" a message. This is prior to attempting the broadcas

relayer/relayer/processor/path_end_runtime.go

Line 580 in be5a4cf

if blocksSinceLastProcessed < blocksToRetrySendAfter {

here, we check if the number of blocks since the lastProcessedHeight is at least 5, and will retry if so.

So, if we broadcast and don't see the observed event within 5 blocks, it will retry. This is especially an issue on faster chains such as sei and dydx where tx propagation through the p2p network can take more than 5 blocks

This modifies the behavior to be: if the last attempted broadcast completed, then retry if it has been more than 5 blocks since the previous broadcast attempt.

Additionally:
removePacketRetention was using the incorrect key for clearing caches on the counterparty, which for acks can lead to counterparty recvpackets not being cleared.

Related: #1359

[jtieri]

great catch on these! had two questions for ya but other than that this looks good

[jtieri]

huge performance boost right here, great sleuthing!