Too many failed HTLCs

Support and Troubleshooting
#1

Good morning umbrel community,

I just realised that I have tons of failed HTLCs with the reason being „Fee Insufficient“. My node runs perfectly otherwise with routings going through every now and then.

What am I doing wrong here?

Cheers
Guido

My configuration:
Raspberry 4
1TB SSD
umbrel 0.4.17
pi-hole

P.S. I have added the following lines to my lnd.conf:

accept-keysend=true
gc-canceled-invoices-on-startup=true
gc-canceled-invoices-on-the-fly=true
ignore-historical-gossip-filters=true
allow-circular-route=true
sync-freelist=true

#3

Hey Orville,

thanks for the quick answer. If that is really the solution I feel ashamed to have asked such a stupid question. :crazy_face:
As my fees were low (1 - 199 ppm) already, well at least I thought they were low, I thought that there would be some kind of gossip / wrong settings problem…

I just lowered my fees to 1 ppm for most of my channels and will hopefully not have any channel failures any more.

Ciao
Guido

#4

Please read my other guide about fees here, where I explained how low fees can help more healthy routing, but also if you manage your max HTLC, you will have less failed routed payments.

1 Like
#5

Hey @DarthCoin,
I already read your guide and found it so good, that I directly changed a few things with my node:

  1. I closed some channels that were unable to contact (and had strange fee behaiviours)
  2. I set my base fee to 0 (#ZeroBaseFee) and the fee rate to 1 ppm
  3. I started to update the max HTLC values on a regular basis

Since then there have been hardly any HTLC failures and the routing numbers have gone through the roof. :grinning:

Thank you for the effort you have put into the guide (and into this forum)!

I feel totally enlightened. :+1:

1 Like
#6

There’s also this
https://plebnet.wiki/wiki/Using_Charge-Lnd_to_Adjust_HTLC_Limits
But I didn’t try it yet. I prefer to do manually, only when is needed.

#7

As I don‘t have too many channels I will continue to do it by hand and wait until it is implemented into the „official“ version.

So far I have found 3 nodes (including yours) that follow this approach. :-]

#8

Many nodes from this community of ZeroBaseFee are doing that

#9

Thanks for the link @DarthCoin , I am already a member of that honorable ZeroBaseFee club. :sunglasses:

BTW I had to stop the max HTLC experiment because it always takes 8-24 h (yes, sometimes a whole day) until the changes are communicated to all of my peers. The amount of HTLC errors stays the same, it just changes from ‚Insufficient Balance‘ to ‚HTLC Exceeds Max‘ and back.

On amboss.space it sometimes takes even longer (2 days or more) before the fees are updated.

Any idea why this takes so long?

#10

Don’t play with changing HTLC every time you see a movement from your channels!
Change them only once/day or even once/2-3 days.
Every change in your channel it have to be broadcast into gossip and also it increase your channel.db size.

max HTLC is also referring to a max size of a tx routed, but you can have for example a MPP (multi-part payment) so from a 500k payment, you got 10 txs x 50k sats.

I would strongly encourage all users to use MPP on any payment they do over 50k sats. That increase a lot the path finding and faster payments.

You should also change it only for those channels where you do not have enough sats on your side, when the channel is drained. All payments FROM that peer can come easily, no worry, your intention is to not redirect to that channel the payments that GO TO that peer, because you do not have enough sats.

What I was trying to do, is to observe first what is the average size of a tx routed through my channels and adjust that to max HTLC for each channel.

Let’s say I have a 999k sats max HTLC policy default for all channels, of course when most of the balance is on MY side.
But when the balance get’s lower than 1M sats on my side, I start splitting that max HTLC into small pieces. Let’s say I go for 200k. If i see that in bext days, that channel is still draining, I lower the max HTLC to 50k. Once is drained I set a max 1k sats and wait. In few days that channel start moving the other way around the sats, FROM the peer towards my side, through other channels, so gradually I increase also the max HTLC.

All depends of your peers habits, how they do the payments through you. Observe them.

The important key is that users get used to do MPP payments.

#11

O.K. That‘s what I did wrong. I changed the Max HTLC value every time something happened in the channel. Next I will try your approach, sounds very logical to me.

Still don‘t know why it takes soooooo long for my changes to be communicated to the other peers. Is it because my node is so small (28M)? Or is it slow because I only use Tor?

And how do I activate MPP in my Bluewallet or in my node?

#12

You are not the only one. I saw this behavior to many other Tor nodes. Is all about the gossip propagation over Tor and this is a bottleneck.
Solutions for you:

  • run your node in hybrid mode
  • wait for a better release of Tor protocol (is coming soon).

Blue wallet is not using MPP. Bluewallet is NOT for managing your channels! Bluewallet use lndhub.
Zeus, Zap are mobile apps that connect to your LN node and use directly your channels and can set MPP.
Thunderhub also uses MPP.

#13

Regarding MPP:

So far I had a different approach with my wallets/apps.

Zeus and Zap are only used to see if my node is o.k. when I‘m not at home.
My Bluewallet is connected via LNDhub and uses the nodes‘s channels and liquidity.

So no MPP with this configuration, right?

Regarding hybrid:

Without a proper firewall installed I am not brave enough to open a port on my router. Maybe it is better for me to wait for the next Tor update.

#14

What is so special in entering into your router and open a damn port and forward it to your node internal IP?
This is not about having a super complicated firewall that doesn’t serve at all in this case.
Follow the instructions from this guide are very simple and clear, for each case.

#15

O.K. I will try to go hybrid this weekend.

#16

Yes, my node is running on hybrid mode. :metal: Thanks @DarthCoin for the gentle nudge. :wink:

Well, at least there is a new clearnet address shown in amboss.space now. To be 100% sure I need to wait until someone opens a channel to me via the clearnet address.

#17

No, those who wants to open channels using the clearnet will choose it. Also the Tor nodes can use it.
Tor nodes can communicate with clearnet, but clearnet only nodes can’t communicate with Tor nodes.
Once a channel is open, your node will announce via both Tor and clearnet.
From now on, when a request through gossip`will ask for your channels, it can respond on both Tor and clearnet, having a balancing and faster response.
It will just make your node more “visible”.

#18

Hey guys how do I check for failed HTLCs?

#19

the easy way is to install the Lightning Shell app in Umbrel inside that you run lntop
There will show you all pending and in transit HTLCs and the reason why are failed.

Another way is to use LNDg app also from Umbrel suite. Is more advanced.

1 Like