Failed payment to invoices with route hints

[Darth-Coin]

Describe the bug

For embedded nodes, payments to invoices created in Zeus with route hints are bounced back with error "no route found".
This is a follow up of #2489 were was fixed the creation of invoice but still the invoice cannot be paid.

Reproduce

Open Zeus and create a new LN invoice, selecting any existing channel as route hint. Tried with each channel, including an Olympus LSP channel.
Invoice is created correctly, node is persistent mode and well synced.
Trying to pay the invoice from various different sources (custodial, self-custodial public node, other private nodes) is getting the error "no route found".

I tried also to switch from Zeus EGS to Blixt EGS, but same result.
I tried resetting the EGS too.
Other Zeus user also test ti and have the same result: https://t.me/zeusLN/87192

Maybe is also linked to this one too #2169

ZEUS version

v0.9.3

Node interface

Embedded LND

Network

Clearnet

Device

Moto G9 and others

Device operating system

Android 11 and 12

Log output

Nothing weird in logs, just that payer cannot find the correct route.

[kaloudis]

I think we might have an issue still with the individual hint selector

@Darth-Coin can you see if you have the same issue with automatic selection

@myxmaster we should test against LND REST and LNC and see if this is just isolated to the embedded node

[Darth-Coin]

Yes, if I select automated route hints it worked perfectly fine. But is always using Olympus. And if I have the Olympus channel full I want to avoid that by selecting a specific channel.
Can you make a process logic that if the LSP option is off and Olympus channel is full or do not have enough inbound for that invoice will switch automatically for next available channel ?
Something like Blixt have.
Otherwise yes, user must choose manually the inbound channel.

[kaloudis]

Yes, if I select automated route hints it worked perfectly fine. But is always using Olympus. And if I have the Olympus channel full I want to avoid that by selecting a specific channel. Can you make a process logic that if the LSP option is off and Olympus channel is full or do not have enough inbound for that invoice will switch automatically for next available channel ? Something like Blixt have. Otherwise yes, user must choose manually the inbound channel.

Haven't seen that feature before.

I think for addressing this bug the next step is to decode an automatic invoice and a manually selected invoice and see if there's a difference in how the channels are encoded

[Darth-Coin]

In Blixt you do not have the option to receive on specific route channel, nor wrapped invoice.
But if you have multiple channels, is trying first with Blixt LSP if you have one. If that channel is filled out is going to the next available. I don't know the technical details, but this is how I experiment it in Blixt.
Maybe ask @hsjoberg how is done in Blixt?

[kaloudis]

The problem is how we're receiving the format of the chan_id from the backend. Some bytes are being lost in the callback. Still investigating.

[Darth-Coin]

OK thanks for looking into this issue.
This issue could affect many users with embedded nodes using multiple channels with various peers.
Practically it makes all other non-LSP channels unusable and after depletion you have to close them.
OK the LSP channel can be used with wrapped invoice, but not all users will want / know to use that option for all payments.

[kaloudis]

OK thanks for looking into this issue. This issue could affect many users with embedded nodes using multiple channels with various peers. Practically it makes all other non-LSP channels unusable and after depletion you have to close them. OK the LSP channel can be used with wrapped invoice, but not all users will want / know to use that option for all payments.

Bit of an overstatement. Doesn't affect public channels + users can still use automatic route hint selection.

We'll try to find a solution though.

[Darth-Coin]

I've tried automatic route hint. That works but is always taking the LSP channel.
If I have the LSP channel full and others with lots of inbound available, the payment fails with "no route found".

[kaloudis]

1c2afd3