fix(UTXO): calc of txfee with change, min relay fee #2316

dimxy · 2025-01-09T14:50:12Z

Fixes for utxo tx fee calculation in UtxoTxBuilder::build() fn:

include change output in txfee
fix min relay tx fee calc (to multiply by tx_size).

UtxoTxBuilder::build() code was refactored to implement fixed txfee calc: made it similar to daemon code (with a loop), also broke it into functions.
Existing tests updated for the fixed txfee.

Fixes issue: #2313 (a test added to validate txfee from the issue).
Should also fix #1567 issue.

TODO: fix input size for segwit

@cipig

fix use gas_fee in build(); fix min_relay_tx_fee calc; refactor break build() into functions rename fn get_tx_fee to get_fee_per_kb; fix utxos tests for recalculated tx fee

* dev: feat(tendermint): validators RPC (#2310) chore(CI): validate Cargo lock file (#2309) test(P2P): add test for peer time sync validation (#2304)

fix refactored UtxoTxBuilder::build(): return only txfee (w/o gas fee) with tx as it used to be

mariocynicys · 2025-01-09T17:09:37Z

Is this ready for review?

dimxy · 2025-01-09T17:18:38Z

Is this ready for review?

I am doing final checks and will change the status for ready after that

borngraced · 2025-01-10T01:44:57Z

Thank you for this PR. Covers most of what I was working on here #2083.

I will close mine when this is approved.

dimxy · 2025-01-10T09:13:41Z

Thank you for this PR. Covers most of what I was working on here #2083.

I will close mine when this is approved.

I thought #2083 was about fee priority (although this PR and #2083 definitely have some interception in txfee calc fixes)

onur-ozkan · 2025-01-22T07:53:03Z

mm2src/coins/utxo/utxo_common.rs

            let total_fee = if tx.outputs.len() == outputs_count {
                // take into account the change output
-                data.fee_amount + (dynamic_fee * P2PKH_OUTPUT_LEN) / KILO_BYTE
+                data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)


Suggested change

data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)

data.fee_amount + actual_tx_fee.get_tx_fee_for_change(0)

onur-ozkan · 2025-01-22T07:53:31Z

mm2src/coins/utxo/utxo_common.rs

-                    data.fee_amount
-                }
+                // take into account the change output
+                data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))


Suggested change

data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))

data.fee_amount + fee_per_kb.get_tx_fee_for_change(tx_bytes.len() as u64)

onur-ozkan · 2025-01-22T07:54:15Z

mm2src/coins/utxo.rs

+    }
+
+    /// Return extra tx fee for the change output as p2pkh
+    fn get_tx_fee_for_change(&self, tx_size: Option<u64>) -> u64 {


I see that tx_size can be used as u64 (see the other suggestions)

removed Option from get_tx_fee_for_change in 3f709fc

onur-ozkan · 2025-01-22T08:15:47Z

mm2src/coins/utxo/utxo_common.rs

+        for utxo in self.required_inputs.clone() {
+            self.tx.inputs.push(UnsignedTransactionInput {
+                previous_output: utxo.outpoint,
+                prev_script: utxo.script,
+                sequence: SEQUENCE_FINAL,
+                amount: utxo.value,
+            });
+            total += utxo.value;
+        }
+
+        for utxo in self.available_inputs.clone() {
+            if total >= amount {
+                break;
+            }
+            self.tx.inputs.push(UnsignedTransactionInput {
+                previous_output: utxo.outpoint,
+                prev_script: utxo.script,
+                sequence: SEQUENCE_FINAL,
+                amount: utxo.value,
+            });
+            total += utxo.value;
+        }


This should be slightly cheaper:

Suggested change

for utxo in self.required_inputs.clone() {

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script,

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in self.available_inputs.clone() {

if total >= amount {

break;

}

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script,

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in &self.required_inputs {

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script.clone(),

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in &self.available_inputs {

if total >= amount {

break;

}

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script.clone(),

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

done in 50a38f8

onur-ozkan · 2025-01-22T08:18:58Z

mm2src/coins/utxo/utxo_common.rs

+            if total >= amount {
+                break;
+            }


We should be able to add this check above the loop, so we don't start iterating it for no reason if it's already true.

fixed in 50a38f8

mariocynicys

Thanks! 1st iteration.
went over all the changes but skipped most of my comments on the tx builder stuff since i couldn't fully comprehend it and don't wanna cause confusion.
will need a couple more iters as this part of the code is a lil risky and confusing.

mariocynicys · 2025-01-22T11:12:14Z

mm2src/coins/utxo.rs

@@ -279,11 +279,43 @@ pub enum TxFee {
 pub enum ActualTxFee {
    /// fee amount per Kbyte received from coin RPC
    Dynamic(u64),
-    /// Use specified amount per each 1 kb of transaction and also per each output less than amount.
+    /// Use specified fee amount per each 1 kb of transaction and also per each output less than the fee amount.


could you explain more about this fee scheme. the per each output part isn't really clear (understood it as 1 fee_rate per 1 output, which i doubt is a correct understanding).

Hmm, actually I have the same question:
this fee calc is not fully clear to me. The comment refers to DOGE, I took a look at its daemon code but did not find where this logic is.

mariocynicys · 2025-01-22T11:14:32Z

mm2src/coins/utxo.rs

+impl ActualTxFee {
+    fn get_tx_fee(&self, tx_size: u64) -> u64 {
+        match self {
+            ActualTxFee::Dynamic(fee_per_kb) => (fee_per_kb * tx_size) / KILO_BYTE,


couldn't this wrongfully floor to zero?

I guess it could.
If it goes too low we would use min relay fee (if this is set though)

mariocynicys · 2025-01-22T11:16:53Z

mm2src/coins/utxo.rs

@@ -279,11 +279,43 @@ pub enum TxFee {
 pub enum ActualTxFee {


since we corrected the confusing namings, we can fix this one as well to TxFeeRate.

renamed vars in the whole code to use always 'FeeRate' or 'fee_rate' 3f709fc

mariocynicys · 2025-01-22T15:13:12Z

mm2src/coins/utxo/utxo_common.rs

+        return_err_if_false!(
            !self.available_inputs.is_empty() || !self.tx.inputs.is_empty(),
            GenerateTxError::EmptyUtxoSet {
-                required: self.sum_outputs_value
+                required: self.required_amount()
            }
        );


what if we refactor this macro to make it return_err_if!. i find it a very challenging mind game to compute the value in there and negate it then. esp that the main action is erroring while the passive one is doing nothing.

return_err_if!(self.available.is_empty() && self.tx.is_empty())

refactored in 615ff3e

mariocynicys · 2025-01-22T15:14:52Z

mm2src/coins/utxo/utxo_common.rs

-                if self.update_fee_and_check_completeness(from.addr_format(), &actual_tx_fee) {
-                    break;
-                }
+        let mut unused_change;


looks very unusual. let's init the value?

yes I agree,
but when I added init to 0u64 clippy issues a 'value is never read' warning. I guess I'd better add init and disable the warning here.

(done)

mariocynicys · 2025-01-22T16:24:26Z

mm2src/coins/utxo/utxo_common.rs

+            if self.sum_inputs >= self.sum_outputs + self.total_tx_fee() {
+                break;
+            }


shouldn't this be exactly equal (or equal + unused change)?

I believe, not.
At this step tx_fee has been updated by the tx_size * fee_rate formula (not like difference of inputs - outputs). So if we have added sufficient inputs, we are finished

and the change will have been adjusted to inputs - outputs - fees, which brings the equation above to equality.

I think this is not necessarily so.
Within an iteration, we are getting next input set and change for the total outputs + previous-txfee.
But due to the input selection algo, we may get this input set with the length less than it was at the previous step (rare case, ofc). So the updated txfee may be less than the previous-txfee (but we want to quit now as we have added enough inputs to cover the txfee. As I can see, kmd works like that: it quits the loop when nFeeRet >= nFeeNeeded)

So the updated txfee may be less than the previous-txfee (but we want to quit now as we have added enough inputs to cover the txfee. As I can see, kmd works like that: it quits the loop when nFeeRet >= nFeeNeeded)

I am not following here. Do you mean that the new tx fee is less than the previous iteration but we willingly and intentionally don't update/reduce it? why?

or we can make this argument simpler: if sum_inputs > sum_outputs + fees + unused_change where do these wasted sats (extra fees?) go? and what is the higher bound on the amount of these wasted sats?

Do you mean that the new tx fee is less than the previous iteration but we willingly and intentionally don't update/reduce it? why?

I could imagine an example:
Previous txfee was calculated with presence of the change output, but on the next iteration we select inputs in a way that no change is required. So the updated txfee would be of lesser value. But it's better to quit here, because if we try to do a new iteration for new txfee we may get a new input set with the change required (and bigger txfee). This may lead to endless loop.

aha gotcha. so we can sometimes over spend on the fees but that's bounded to a really small number. that number is the difference of fee for this tx assuming there is change and there is no change (same tx same inputs same outputs, just playing with the change). that's indeed a pretty small number.

BUT: isn't this number represented (or supposed to be represented) by the unused_change? and then if we plug that in we can check for equality rather than >=.

my issue with a >= is that if we have a bug in tx creation code, we might end up creating a tx burning a lot of money.
if >= is necessary then we can check that the difference sum_inputs - (sum_outputs + total_tx_fee) is bounded to a small number as a last validation check before returning such a tx.

mariocynicys · 2025-01-22T16:25:36Z

mm2src/coins/utxo/utxo_common.rs

        let data = AdditionalTxData {
-            fee_amount: self.tx_fee,
-            received_by_me,
+            fee_amount: self.tx_fee, // we return only txfee here (w/o gas_fee)
+            received_by_me: self.sum_received_by_me(&change_script_pubkey),
            spent_by_me: self.sum_inputs,
            unused_change,
            // will be changed if the ticker is KMD


Q: why don't we add this unused/dust change to the fee amount? i mean it's part of the fee at the end of the day.

As I can see in the code, unused_change is reserved until the kmd interest is calculated (for the kmd coin). And if unused_change + interest becomes greater than dust the change output would be added.

mariocynicys · 2025-01-22T16:40:16Z

mm2src/coins/utxo/utxo_common.rs

+        let mut one_time_fee_update = false;
+        loop {
+            let required_amount_0 = self.required_amount();


im not really sure about the convergence/termination of this loop. it's not trivial to see.
we can do a 10 rounds or something using a for loop and give up.
and maybe use this for counter to simulate one_time_fee_update.

This logic I believe matches to the loop daemon wallet code.
I think it should work like that:

pass 1: no change yet, we add inputs w/o it. We add change and txfee increases. If inputs not enough we go to pass 2.

pass 2: we add inputs for the increased txfee. New inputs set may be bigger (if we added small input(s) to cover new txfee) and txfee again increases. Again, if inputs not enough go to pass 3.

pass 3: same logic.
and so on.

I guess, if we have many inputs of small enough value, to be added on each pass to cover the previous txfee (but insufficient to cover the next txfee), increasing the input set so adding more value to next txfee, then we may have many iterations.

thx so much for the explanation..that helps a lot.

I was suggesting a for loop because the logic inside is complicated and one can't trivially see that it is guaranteed to terminate (either by erroring or successfully matching inputs and outputs).

when I have more confidence and understanding of the code flow inside the loop and the edge cases all being covered I will probably be for a loop instead of for for simplicity 😂

borngraced

Really great work! Some minor notes from my side

mm2src/coins/utxo/utxo_common.rs

refactor (borngraced): add_change simplified, remove tx_fee mut, use checked_sub

…true

dimxy added 4 commits January 8, 2025 23:52

fix UtxoTxBuilder::build() tx fee calculation to include change output;

cea1b2f

fix use gas_fee in build(); fix min_relay_tx_fee calc; refactor break build() into functions rename fn get_tx_fee to get_fee_per_kb; fix utxos tests for recalculated tx fee

Merge branch 'dev' into fix-utxo-tx-fee-for-change

455403c

* dev: feat(tendermint): validators RPC (#2310) chore(CI): validate Cargo lock file (#2309) test(P2P): add test for peer time sync validation (#2304)

fix qrc20 gas fee calc in trade preimage

8d2fe79

revert qtum gas_fee calc in trade preimage,

bc69a42

fix refactored UtxoTxBuilder::build(): return only txfee (w/o gas fee) with tx as it used to be

dimxy added status: in progress priority: medium Moderately important tasks that should be completed but are not urgent. bug: API labels Jan 9, 2025

dimxy added status: pending review and removed status: in progress labels Jan 13, 2025

dimxy requested review from borngraced and mariocynicys January 21, 2025 17:13

onur-ozkan reviewed Jan 22, 2025

View reviewed changes

mariocynicys reviewed Jan 22, 2025

View reviewed changes

borngraced requested changes Jan 23, 2025

View reviewed changes

dimxy added 4 commits January 23, 2025 14:26

fix: err if change < dust (was <=)

40cfbda

refactor (borngraced): add_change simplified, remove tx_fee mut, use checked_sub

refactor (onur-ozkan): use refs instead of clone in add_tx_inputs

50a38f8

refactor (mariocynicys): fix return_err_if macro to check if cond is …

615ff3e

…true

refactor: rename vars to use always 'fee rate' root word

3f709fc

	data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)
	data.fee_amount + actual_tx_fee.get_tx_fee_for_change(0)

	data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))
	data.fee_amount + fee_per_kb.get_tx_fee_for_change(tx_bytes.len() as u64)

		@@ -279,11 +279,43 @@ pub enum TxFee {
		pub enum ActualTxFee {

fix(UTXO): calc of txfee with change, min relay fee #2316

Are you sure you want to change the base?

fix(UTXO): calc of txfee with change, min relay fee #2316

Conversation

dimxy commented Jan 9, 2025 • edited Loading

mariocynicys commented Jan 9, 2025

dimxy commented Jan 9, 2025

borngraced commented Jan 10, 2025

dimxy commented Jan 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariocynicys left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimxy Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

borngraced left a comment

Choose a reason for hiding this comment

dimxy commented Jan 9, 2025 •

edited

Loading

dimxy Jan 23, 2025 •

edited

Loading