Bridge: A refund transaction present in both pending transactions and executed transactions which shouldn't happen.

[sameh-farouk]

On devnet, I noticed that a RefundTransaction is present in both RefundTransactions and ExecutedRefundTransactions. This is unexpected because we remove the transaction from the RefundTransactions storage map before inserting it into the ExecutedRefundTransactions storage map.

tfchain/substrate-node/pallets/pallet-tft-bridge/src/tft_bridge.rs

Lines 419 to 422 in 1da1c15

	let tx = RefundTransactions::<T>::get(&tx_id);
	RefundTransactions::<T>::remove(&tx_id);
	ExecutedRefundTransactions::<T>::insert(tx_id.clone(), &tx);

This could be a sign of a race condition since we have multiple running instances of the bridge daemons.
One daemon adds this transaction to the pending list after another daemon removed it from the pending list and set it as executed.

This results in tfchain emitting ExpiredRefundTransaction events for this transaction continuously because it exists in the pending transactions map. However, bridge validators keep skipping this event since the transaction exists in the executed transactions.

Perhaps we should investigate how this can happen and how the flow can be further improved to prevent such occurrences.