[bug]: Channel Becomes Inactive after InvRef Error

[carlaKC]

Full honestly that I have not done much digging to understand whether this is a known bug that has been fixed, so please feel free to close out if this has been fixed.

Background

We're running two signet nodes with a channel between them lnd0 --- lnd1 for an education exercise. There are a few hundred invoices on lnd0, and around 400 people are doing a scavenger hunt where they have to get their invoice paid over the channel.

The channel is inactive on lnd0 and active on lnd1. Looking at lnd0's logs I saw the following:

2025-01-31 16:34:46.804 [DBG] INVC: Invoice(pay_hash=2d11b2e50b9ed00ebe37cea1f372982c36498b2f3a56a63d1814a99f6af54096, pay_addr=a174738478a9e81451cdf784662ea44d3eba25215eb313c3d2bb5eb579f1d062): inv ref matches multiple invoices, amt=3080000 mSAT, expiry=5425, circuit=(Chan ID=1824:1:0, HTLC ID=141), mpp=total=3080000 mSAT, addr=a174738478a9e81451cdf784662ea44d3eba25215eb313c3d2bb5eb579f1d062, amp=<nil>, metadata=<nil>
2025-01-31 16:34:46.804 [ERR] HSWC: ChannelLink(c695cb1405377d7f4c41fd4aee1565f55fd54cc31bb353c707881659039922f9:0): failing link: inv ref matches multiple invoices with error: internal error
2025-01-31 16:34:46.804 [WRN] HSWC: ChannelLink(c695cb1405377d7f4c41fd4aee1565f55fd54cc31bb353c707881659039922f9:0): ignoring link failure (unable to resolve fwd pkgs: link failed while processing remote adds), as link already failed
2025-01-31 16:34:46.804 [INF] HSWC: ChannelLink(c695cb1405377d7f4c41fd4aee1565f55fd54cc31bb353c707881659039922f9:0): exited
2025-01-31 16:34:46.804 [INF] HSWC: ChannelLink(c695cb1405377d7f4c41fd4aee1565f55fd54cc31bb353c707881659039922f9:0): stopping
2025-01-31 16:34:46.804 [INF] HSWC: Removing channel link with ChannelID(f922990359168807c753b31bc34cd55ff56515ee4afd414c7f7d370514cb95c6)
2025-01-31 16:34:46.804 [DBG] PEER: Peer(037c3565e3c335886e44082b1cf14e081e1a030605b5b7925680954c2bc6ea1394): Sending Error(chan_id=f922990359168807c753b31bc34cd55ff56515ee4afd414c7f7d370514cb95c6, err=internal error) to 037c3565e3c335886e44082b1cf14e081e1a030605b5b7925680954c2bc6ea1394@127.0.0.1:54992
2025-01-31 16:34:46.805 [DBG] PEER: Peer(037c3565e3c335886e44082b1cf14e081e1a030605b5b7925680954c2bc6ea1394): Received Error(chan_id=f922990359168807c753b31bc34cd55ff56515ee4afd414c7f7d370514cb95c6, err=remote error) from 037c3565e3c335886e44082b1cf14e081e1a030605b5b7925680954c2bc6ea1394@127.0.0.1:54992

Seems like lnd0 has some issue with the invoice, and it's erroring out and disconnection, and lnd1 keeps trying to reconnect. I've restarted both nodes and the issue did not resolve, so force closed the channel and spun up a new one for our tests.

Happy to share full logs because it's just a test env.

Your environment

• Version of LND: 0.18.4-beta (pulled straight from release page)
• Version of Bitcoind: 28.0
• Default DB

[Roasbeef]

How did you make the invoices? Did any of the invoices set a custom payment hash, rather than letting lnd generate one itself?

[carlaKC]

How did you make the invoices? Did any of the invoices set a custom payment hash, rather than letting lnd generate one itself?

Totally vanilla invoices, generated around 1000: lncli --network=signet addinvoice --amt="$random_amt" --expiry=4838400 --memo="$wallet_name"

Full script to create the batch here. Attaching logs, server IP is public so no private info:

• lnd0.txt
• lnd1.txt

[carlaKC]

Nothing special about the invoice:

{
    "memo":  "wallet_543",
    "r_preimage":  "4434468082f1f714f9e6d34e76020f2a305d467951a191a40ba2f48f77b5e19f",
    "r_hash":  "",
    "value":  "4518",
    "value_msat":  "4518000",
    "settled":  false,
    "creation_date":  "1736267022",
    "settle_date":  "0",
    "payment_request":  "lntbs45180n1pnh64gwpp5nj7c6t6373huz47vdm69m49at4j6ax7q74a2s0jgc8gd4sws33csdqswaskcmr9w30n2dpncqzzsxq9yn4qqsp5ys2qtfnpymdnvq9acmw5dldmaj0akyf52fs8eumac9uapne9rjgq9qxpqysgqcf0tk5u63c6r5jrd7q4uvmmsd7kq2dvz8qaxapnpy59f9ldclrprxg6c6396z7ehn5hs2zyjpq6hs6sd8zns7r5tlyq88fe7027juxsq03epql",
    "description_hash":  "",
    "expiry":  "4838400",
    "fallback_addr":  "",
    "cltv_expiry":  "80",
    "route_hints":  [],
    "private":  false,
    "add_index":  "1993",
    "settle_index":  "0",
    "amt_paid":  "0",
    "amt_paid_sat":  "0",
    "amt_paid_msat":  "0",
    "state":  "OPEN",
    "htlcs":  [],
    "features":  {
        "8":  {
            "name":  "tlv-onion",
            "is_required":  true,
            "is_known":  true
        },
        "14":  {
            "name":  "payment-addr",
            "is_required":  true,
            "is_known":  true
        },
        "17":  {
            "name":  "multi-path-payments",
            "is_required":  false,
            "is_known":  true
        },
        "25":  {
            "name":  "route-blinding",
            "is_required":  false,
            "is_known":  true
        }
    },
    "is_keysend":  false,
    "payment_addr":  "241405a66126db3600bdc6dd46fdbbec9fdb113452607cf37dc179d0cf251c90",
    "is_amp":  false,
    "amp_invoice_state":  {},
    "is_blinded":  false,
    "blinded_path_config":  null
}

Was paid with an incorrect payment address (part of the challenge is paying over a custom route):

2025-01-31 19:36:25.349 [DBG] HSWC: ChannelLink(fe4953aa9187cb4d4b2452ff786e65c6f2b6ff0142c794eeb72fb7017dfc313d:0): received cancel resolution for (Chan ID=5414:2:0, HTLC ID=8) with outcome: payment address mismatch

[carlaKC]

Tried to band-aid our test setup by opening up another channel and it ended up borked as well. Attempted to reproduce on a different node by paying an invoice with the wrong payment address and couldn't get the same behavior.

Ended up just killing the nodes and downgrading to 18.3 for our purposes (since it's just a test env). Attaching full LND dirs for the nodes involved (LND0 is the problematic one).

lnd0_files.zip
lnd1_files.zip