XCM delivery fee breaks XCM compatibility

[xlc]

A failed transfer: https://karura.subscan.io/xcm_message/kusama-024c52268887c3fb756eeea47242a7d2d831b6ba

Since the new Kusama runtime upgrade, which includes #1234, KSM transfer between parachains stops working.

This is due to the XCM execution on Kusama unable to charge deliver fee from holding register and therefore refuse to send DMP to dest chain.

This also highlights a main issue with the current deliver fee mechanism. It is infeasible to query and calculate the delivery fee pure from onchain runtime. It is a tx version breaking change to ask user to specify an extra delivery fee parameter. It is not possible to enable jit_withdraw mode in this particular case. It just doesn't work.

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/polkadot-digest-26-jan-2024/5848/1

[albertov19]

Similar problem here -> https://kusama.subscan.io/extrinsic/21573927-7

[acatangiu]

@xlc do you have a chopstix test or similar to use to test out ideas for quick fixes?

I believe something like this 371d5b0 could get us running again until we come up with a comprehensive solution for:

This also highlights a main issue with the current deliver fee mechanism. It is infeasible to query and calculate the delivery fee pure from onchain runtime.

[acatangiu]

Also, does no one have these scenarios running on testnets?

#1234 is live on Rococo relay and System Chains since 3-4 months now - not cool getting burned directly on Kusama 😢

[xlc]

To reproduce this with Chopsticks

To run the networks
KARURA_BLOCK_NUMBER=6136909 KUSAMA_BLOCK_NUMBER=21594465 BIFROST_BLOCK_NUMBER=5895485 npx @acala-network/chopsticks@latest xcm --relaychain kusama --parachain karura --parachain bifrost

Open Karura and send the extrinsic:
https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A8000#/rpc

submit author.submitExtrinsic with data:
0x95028400842745b99b8042d28a7c677d9469332bfc24aa5266c7ec57c43c7af125a0c16c016ed23bfa0338e2c971e6e9c1e375b56c6791de705fee18e43d2f0d7c1e06f864546e9b62179b9691e364cb9aa39773f1b8b090980cbd66a2b4e551b70cb0c28da4008981003600008200743ba40b000000000000000000000003010200451f0100842745b99b8042d28a7c677d9469332bfc24aa5266c7ec57c43c7af125a0c16c00

Submit it and wait for few minutes to produce blocks. Check terminal logs for output.

Check the new block produced on Kusama: https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A8002#/explorer

If the issue is fixed, check the block on Bifrost: https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A8001#/explorer

To test the fix or print XCM logs, we need to do wasm override for Kusama:
copy the kusama config: https://github.com/AcalaNetwork/chopsticks/blob/master/configs/kusama.yml
specify wasm-override point to the new Kusama runtime
repeat from step 1 but use --relaychain path/to/kuama.yml

[xlc]

I can create something similar to https://github.com/xlc/chopsticks-ksm-xcm tomorrow. but it should be trivial enough to modify it to test for this bug

[itsbirdo]

Linked issue: #3075

[adaOctopus]

Same issue for me. Bridging KSM to KAR seems fine in the explorer, but KSM funds never arrived to the KAR wallet.
https://kusama.subscan.io/extrinsic/21602766-2
Any ideas when this is fixed and if funds have been lost?

[albertov19]

FYI - I've created two scripts to calculate the worst-case scenario for Kusama -> Parachain KSM transfers and from AssetHub -> Parachain Asset Transfers:

Kusama -> https://github.com/albertov19/xcmTools/blob/main/calculateKusamaDeliveryFees.ts
AssetHub -> https://github.com/albertov19/xcmTools/blob/main/calculateAHXcmDeliveryFees.ts

These scripts assume the worst-case scenario by applying U128::MAX to the variable input of the elements that result in the "longest" XCM possible. This check should be done in the origin chain (either Kusama or AssetHub) by dApps, and add this value to the tx fee of paymentInfo endpoint.

[rageruun]

Tokens stuck in parachain bridge when transmitted from the Kusama network to the Moonriver (xcKSM).

[acatangiu]

update:

fix is merged to patch-release branch: #3085 and released to crates.io: https://crates.io/crates/staging-xcm-executor/4.0.1

another fix is also being backported #3113 and 4.0.2 needs to be published to crates.io

next, we'll have to include it in Polkadot & Kusama runtimes and do a patch release there

[rageruun]

thanks вт, 30 янв. 2024 г. в 14:22, Adrian Catangiu ***@***.***>:

…

update: fix is merged to patch-release branch: #3085 <#3085> and released to crates.io: https://crates.io/crates/staging-xcm-executor/4.0.1 another fix is also being backported #3113 <#3113> and 4.0.2 needs to be published to crates.io next, we'll have to include it in Polkadot & Kusama runtimes and do a patch release there — Reply to this email directly, view it on GitHub <#3070 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWYFUHTIER5QJB4ACQ2VPP3YRDQXHAVCNFSM6AAAAABCLU37ACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJWG4ZDONZSGM> . You are receiving this because you commented.Message ID: ***@***.***>

[Hawkarhf92]

Please someone help, I sent some KSM 5 days ago but still not received, from kusama network to Sora mainnet, but xcm transfer not created, here is my extrinsic hash, (0x78540d5b0b3831468c7c4a70ef5a81bccb3c7cb96e44ad81e39c96e73a9ebcff ) please help ,

[albertov19]

FYI - For people that sent XCM transfers to Moonriver -> https://forum.moonbeam.network/t/kusama-assethub-rt-1001000-xcm-failed-transactions/1542

[acatangiu]

should be fixed in runtimes patch release https://github.com/polkadot-fellows/runtimes/releases/tag/v1.1.2

[adaOctopus]

@acatangiu do you know funds bridged from Kusama to Karura will be unlocked and available again to us? It has been already a week. There are dozens of people by the way asking this same question in the discord server. Any inputs from your end in terms of ETAs will be much appreciated. Super thanks !!!

[acatangiu]

@acatangiu do you know funds bridged from Kusama to Karura will be unlocked and available again to us? It has been already a week. There are dozens of people by the way asking this same question in the discord server. Any inputs from your end in terms of ETAs will be much appreciated. Super thanks !!!

Once Kusama upgrade is enacted, the root cause issue will be fixed - aka it should not happen again (further transfers should work well).
Regarding already locked funds of past transfers, these can be unlocked by the specific parachains involved at any time.

In your case, Karura needs to manually unlock the funds, but bear in mind that this is probably a somewhat manual process so it might take a bit of time on their side.

[adaOctopus]

@acatangiu thanks for the reply. Appreciate it. But how can we enact this manual process? is there someone from Karura we need to talk to? And what does "a bit of time" mean? I know Karura probably is not your team to know more stuff about it, but any guidance would help.

[acatangiu]

Please move this discussion to a more relevant place.

I know Karura probably is not your team to know more stuff about it, but any guidance would help.

On their homepage I can see links at the bottom to their telegram, twitter, discord, etc, try one of those.

[jonathanpdunne]

I'm aware of at least one case where there are locked funds of past transfers stuck on Kusama, where the recovery process is likely a referenda via OpenGov.

Would it appropriate for someone familiar with Substrate to take a look at the history of the chain, and propose a referenda which retrieves all stuck KSM that can be attributed to this deliveryFee error?

[xlc]

There are two different cases. For KSM transfer between parachains, it is the dest chain need referenda to credit the missing fund. For Karura, we plan to make it happen after Kusama have fixed the issue and then try to identify all the failed transfer to Karura and then create a referendum to credit the fund.

For KSM from Kusama to parachain that goes missing, it can only be returned by Kusama referendum.

[albertov19]

@xlc for KSM from Kusama to para chains, it can also be returned through a referendum in the parachain itself.

We indexed all transactions in Kusama that transferred KSM to Moonriver's Sovereign account but the XCM Attempted failed. Then we proposed a referendum that sends an XCM message to Kusama doing [WithdrawAsset, ClearOrigin, BuyExecution, DepossitAsset(1), DepossitAsset(2)....,DepositAsset(Wild(AllCounted)....] where 1,2,3.. are the addresses of the affected users that we got from the indexer.

https://forum.moonbeam.network/t/kusama-assethub-rt-1001000-xcm-failed-transactions/1542/2?u=albertov19

The only KSMs that need to be returned through a Kusama referenda are those that were teleported from Kusama to AssetHub, as they are locked in a system account in Kusama (some XCM related account)

[sebastianmontero]

Has this issue been fixed in rococo?

[acatangiu]

Has this issue been fixed in rococo?

yes, this is fixed on Rococo, Westend, Kusama and Polkadot.

[Pescador1551]

@albertov19 @acatangiu : I lost 76 KSM funds in January during an XCM teleport from Kusama to Assethub due to the bug. Do I understand correctly that I can only recover this by initiating a referendum on Kusama? Or is there another way to regain control over my funds? Thanks in advance

[albertov19]

Hey @Pescador1551 to my understanding, yes.

Unsure if the fellowship is doing anything to help recover these funds. Or the wallet team or dApp that you used to transfer the funds

[acatangiu]

@albertov19 @acatangiu : I lost 76 KSM funds in January during an XCM teleport from Kusama to Assethub due to the bug. Do I understand correctly that I can only recover this by initiating a referendum on Kusama? Or is there another way to regain control over my funds? Thanks in advance

Hi @Pescador1551, you should be able to recover them yourself, by using new claim_assets() extrinsic. This will be available in the upcoming runtimes v1.2.0 release happening next week (plus a couple of weeks for gov to upgrade).

Alternatively, you can go the referendum route to unlock the funds which may or may not be faster in the end.

[Pescador1551]

Hi @acatangiu, I would like to reclaim my assets by using the function you mentioned in your previous message ('new claim_asset() extrinsic'). However, I'm not an expert in this field and honestly don't know where to start. I kindly ask if you have any information on how I can execute this action. I highly appreciate it. Thank you in advance!

[franciscoaguirre]

Hello @Pescador1551, I made a small guide you can follow here: https://hackmd.io/@franciscoaguirre/Byahn6wZR.
Here if you have any other questions 😄

[Pescador1551]

Hello @franciscoaguirre, thank you so much for this explanation. I've made progress, but before I proceed with the action, I'd like to verify with you if I've set up all the variables correctly so I don't end up losing my assets after all. It was actually about a transfer from the native KSM chain to the Statemine chain where I lost my assets. It was this transaction. I have added a screenshot of the action I want to perform. I have two questions, how much KSM should be available in my account to perform the action? As far as you can see, the variables are defined correctly (XCM version: V3, interior: Here (does that mean it will be settled on my own account?).

BTW: If this is not the right place to continue this conversation, we can certainly move to another platform. Do you have any suggestions where we could potentially continue?

[franciscoaguirre]

Hi @Pescador1551, thank you for clarifying with me first.

Two things:

• The assets should be exactly the assets that were trapped, so both the version and the id and amount must be correct. So you need to click on "add item" and put the assets that were trapped.
• Don't use (parents: 0, interior: Here), that's the chain origin! You should use your own account, so (parents: 0, interior: X1(AccountId32 { })). You can just click on your account's avatar to copy it and then paste it there.

I'll update the guide with this. Thank you for letting me know!

I don't have any problem continuing the conversation here. It's useful for anyone who might have this issue.

[Pescador1551]

Thank you for your prompt response and your updated guide. I have set up the action again and would like to ask if you find what I entered now logical, given the extrinsic (which I already mentioned in my previous message)?

For the ID, I have now entered my Kusama address, is that what you meant? I see in the example that it should be a 0x address.

Thank you in advance!

[Pescador1551]

Hi @franciscoaguirre, I forgot to tag you in my previous message so you missed it probably! My bad, sorry for that!

[franciscoaguirre]

@Pescador1551 that looks okay now, but taking a look at your transaction, I don't see the AssetsTrapped event.
It might be that we have to make a governance proposal to get them back.
We are investigating to see if there are more affected accounts, then we'll create proposals to get all the funds back.

[Pescador1551]

Hi @franciscoaguirre, should I attempt to carry out the action, or am I risking the assets disappearing from sight? Additionally, I lack experience in creating a governance proposal if that becomes necessary. I hope you can assist me.

[Pescador1551]

Hi @franciscoaguirre, I found a proposal on Kusama that addresses the same issue, but from an individual address. You can find it here: https://kusama.subsquare.io/referenda/356. Could this help with a new proposal for addresses that have encountered the same issue?

[franciscoaguirre]

@Pescador1551 We will assist you and everyone affected by this bug. Thanks for sharing that proposal.

I'll let you know when the proposal is created

[Pescador1551]

Hi @franciscoaguirre, that would be very nice. Hopefully the proposal will be approved and I can support your work a little bit with a small portion of lost funds. Thank you in advance. I really appreciate your help.

[Pescador1551]

Hi @franciscoaguirre, I am curious if there have been any developments regarding the drafting of a proposal for the lost funds. Is there anything I can contribute to this?

[Pescador1551]

Hi @franciscoaguirre, I hope this message reaches you well. Could you provide an update on the proposal being drafted to recover the lost funds? Thank you in advance!