Umbrel 0.4.4 stuck, then 0.4.5 stuck and no sync at all

1

Hi i’m running Umbrel on raspi 4 4GB.
was doing great until 0.4.4 : stuck and neverending loading of differents apps… i rebooted, then installed thje 0.4.5 thinking it would fix the issue… now it’s worse no sync at all and neverending loading apps.
I tried to uninstall the apps to see if sync was starting again but not.

Some logs i have collected.

Blockquote
bitcoin | 2021-10-24T17:10:53Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:10:56Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:10:59Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:02Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:05Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:08Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:11Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:14Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:17Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:20Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:23Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)
bitcoin | 2021-10-24T17:11:26Z connect() to 10.21.21.11:9050 failed after wait: Host is unreachable (113)

Blockquote
A look at iptables :

Chain DOCKER (2 references)
target prot opt source destination
ACCEPT tcp – anywhere 10.21.21.2 tcp dpt:http
ACCEPT tcp – anywhere 10.21.21.8 tcp dpt:8333
ACCEPT tcp – anywhere 10.21.21.30 tcp dpt:3008

Blockquote
| Starting electrs 0.9.1 on aarch64 linux with Config { network: Bitcoin, db_path: “/data/db/bitcoin”, daemon_dir: “/data/.bitcoin”, daemon_auth: CookieFile(“/data/.bitcoin/.cookie”), daemon_rpc_addr: V4(10.21.21.8:8332), daemon_p2p_addr: V4(10.21.21.8:8333), electrum_rpc_addr: V4(0.0.0.0:50001), monitoring_addr: V4(127.0.0.1:4224), wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(200), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, disable_electrum_rpc: false, server_banner: “Umbrel v0.4.5”, args: }
electrs | [2021-10-24T17:00:43.107Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
electrs | [2021-10-24T17:00:43.241Z INFO electrs::db] “/data/db/bitcoin”: 2564 SST files, 4.292819957 GB, 0.354060605 Grows
electrs | [2021-10-24T17:00:45.117Z INFO electrs::chain] loading 346010 headers, tip=00000000000000000b945acaf2118d500b988c49b4c1fc87f9e49bf7737e98c3
electrs | [2021-10-24T17:00:47.021Z INFO electrs::chain] chain updated: tip=00000000000000000b945acaf2118d500b988c49b4c1fc87f9e49bf7737e98c3, height=346010
electrs | [2021-10-24T17:00:47.088Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
electrs | [2021-10-24T17:00:47.188Z INFO electrs::index] indexing 2000 blocks: [346011…348010]
electrs | thread ‘p2p_recv’ panicked at ‘invalid message: UnsupportedSegwitFlag(0)’, src/libcore/result.rs:1188:5
electrs | note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
electrs | [2021-10-24T17:00:47.273Z INFO electrs::db] closing DB at /data/db/bitcoin
electrs | Error: electrs failed
electrs |
electrs | Caused by:
electrs | 0: sync failed
electrs | 1: failed to get block 00000000000000000a1668e84c7dd5d4bee16bbd36060d6f56c3e60c60d57f3e
electrs | 2: receiving on an empty and disconnected channel

if anyone has an idea to fix that…

2

Looks like your Tor container has an issue. Can you share its logs?

1 Like
3 
tor                  | ============================================================ T= 1635102542

tor | Tor 0.4.5.7 died: Caught signal 4
tor | tor(+0xebb04)[0x557b3dbb04]
tor | linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7f95e6c7c0]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x142000)[0x7f95af9000]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x157914)[0x7f95b0e914]
tor | /lib/aarch64-linux-gnu/libpthread.so.0(+0xf250)[0x7f95997250]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0xc)[0x7f95b57a94]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(OPENSSL_init_crypto+0x1f0)[0x7f95b0ee08]
tor | /usr/lib/aarch64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x3c)[0x7f95c91bb4]
tor | tor(crypto_openssl_early_init+0x30)[0x557b3d09c0]
tor | tor(crypto_early_init+0x6c)[0x557b3d01dc]
tor | tor(+0xe0244)[0x557b3d0244]
tor | tor(subsystems_init_upto+0x80)[0x557b3b3220]
tor | tor(tor_run_main+0x4c)[0x557b351bc4]
tor | tor(tor_main+0x48)[0x557b3505a8]
tor | tor(main+0x24)[0x557b3500d4]
tor | /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0xe4)[0x7f95822d24]
tor | tor(+0x60130)[0x557b350130]
tor |
tor | ============================================================ T= 1635102603
tor | Tor 0.4.5.7 died: Caught signal 4
tor | tor(+0xebb04)[0x5590a4bb04]
tor | linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7f9f16c7c0]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x142000)[0x7f9edf9000]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x157914)[0x7f9ee0e914]
tor | /lib/aarch64-linux-gnu/libpthread.so.0(+0xf250)[0x7f9ec97250]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0xc)[0x7f9ee57a94]
tor | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(OPENSSL_init_crypto+0x1f0)[0x7f9ee0ee08]
tor | /usr/lib/aarch64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x3c)[0x7f9ef91bb4]
tor | tor(crypto_openssl_early_init+0x30)[0x5590a409c0]
tor | tor(crypto_early_init+0x6c)[0x5590a401dc]
tor | tor(+0xe0244)[0x5590a40244]
tor | tor(subsystems_init_upto+0x80)[0x5590a23220]
tor | tor(tor_run_main+0x4c)[0x55909c1bc4]
tor | tor(tor_main+0x48)[0x55909c05a8]
tor | tor(main+0x24)[0x55909c00d4]
tor | /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0xe4)[0x7f9eb22d24]
tor | tor(+0x60130)[0x55909c0130]
umbrel@umbrel:~/umbrel $

got this with docker-compose logs --tail=300 tor
not sure if there is other place to check the TOR logs ?

4

lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host
lnd | dial tcp 10.21.21.11:9050: connect: no route to host

also this log from LND makes me think the update went bad and the docker interface is not well defined : ghost IP ?
if this is the case what could i do fo fixing this ‘network’ issue ?

thanks.

5

Any idea for fixing the network issue ? now the Umbrel nginx seems down too, only access with SSH.
Not always a good idea to update to lastest version :smiley:

6

Updated to the latest 0.4.6 version and tor still does not work, i rebooted several times but it doesn’t fix anything.

docker-compose logs --tail=100 tor

   tor                  | ============================================================ T= 1635325011
tor                  | Tor 0.4.5.7 died: Caught signal 4
tor                  | tor(+0xebb04)[0x558451bb04]
tor                  | linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7fa67f47c0]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x141f00)[0x7fa6480f00]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x157914)[0x7fa6496914]
tor                  | /lib/aarch64-linux-gnu/libpthread.so.0(+0xf250)[0x7fa631f250]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0xc)[0x7fa64dfa94]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(OPENSSL_init_crypto+0x1f0)[0x7fa6496e08]
tor                  | /usr/lib/aarch64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x3c)[0x7fa6619bb4]
tor                  | tor(crypto_openssl_early_init+0x30)[0x55845109c0]
tor                  | tor(crypto_early_init+0x6c)[0x55845101dc]
tor                  | tor(+0xe0244)[0x5584510244]
tor                  | tor(subsystems_init_upto+0x80)[0x55844f3220]
tor                  | tor(tor_run_main+0x4c)[0x5584491bc4]
tor                  | tor(tor_main+0x48)[0x55844905a8]
tor                  | tor(main+0x24)[0x55844900d4]
tor                  | /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0xe4)[0x7fa61aad24]
tor                  | tor(+0x60130)[0x5584490130]
tor                  |
tor                  | ============================================================ T= 1635325072
tor                  | Tor 0.4.5.7 died: Caught signal 4
tor                  | tor(+0xebb04)[0x555a79bb04]
tor                  | linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7f9bef17c0]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x141f00)[0x7f9bb7df00]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(+0x157914)[0x7f9bb93914]
tor                  | /lib/aarch64-linux-gnu/libpthread.so.0(+0xf250)[0x7f9ba1c250]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0xc)[0x7f9bbdca94]
tor                  | /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1(OPENSSL_init_crypto+0x1f0)[0x7f9bb93e08]
tor                  | /usr/lib/aarch64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x3c)[0x7f9bd16bb4]
tor                  | tor(crypto_openssl_early_init+0x30)[0x555a7909c0]
tor                  | tor(crypto_early_init+0x6c)[0x555a7901dc]
tor                  | tor(+0xe0244)[0x555a790244]
tor                  | tor(subsystems_init_upto+0x80)[0x555a773220]
tor                  | tor(tor_run_main+0x4c)[0x555a711bc4]
tor                  | tor(tor_main+0x48)[0x555a7105a8]
tor                  | tor(main+0x24)[0x555a7100d4]
tor                  | /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0xe4)[0x7f9b8a7d24]
tor                  | tor(+0x60130)[0x555a710130]
umbrel@umbrel:~/umbrel $

What can i do ?

A friend who is running a small shop was thinking about using umbrel with BTCPayServer but i think for production it’s not mature enough ?

7

I reverted to the 0.4.3 version and i have another issue with LND
look like the DB corrupted ? any idea how to fix ? maybe related to the fact that the node is DOWN since at least 1 week and LND channel is dead / locked / corrupted ?

cd ~/umbrel && sudo ./scripts/update/update --repo getumbrel/umbrel#v0.4.3

nd                  | goroutine 56 [running]:
lnd                  | go.etcd.io/bbolt.(*DB).freepages.func2(0x400011efc0)
lnd                  |  go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:1003 +0xc4
lnd                  | created by go.etcd.io/bbolt.(*DB).freepages
lnd                  |  go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:1001 +0x134
lnd                  | 2021-10-27 12:22:50.544 [INF] LTND: Version: 0.13.3-beta commit=v0.13.3-beta, build=production, logging=default, debuglevel=info
lnd                  | 2021-10-27 12:22:50.544 [INF] LTND: Active chain: Bitcoin (network=mainnet)
lnd                  | 2021-10-27 12:22:50.581 [INF] RPCS: RPC server listening on 0.0.0.0:10009
lnd                  | 2021-10-27 12:22:50.583 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080
lnd                  | 2021-10-27 12:22:50.583 [INF] LTND: Opening the main database, this might take a few minutes...
lnd                  | 2021-10-27 12:22:50.583 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
lnd                  | 2021-10-27 12:22:51.016 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:22:51.209 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:22:51.210 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:22:51.213 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:22:51.217 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: waiting to start, RPC services not available
lnd                  | panic: freepages: failed to get all reachable pages (page 549755826386: out of bounds: 50852)
lnd                  |
lnd                  | goroutine 106 [running]:
lnd                  | go.etcd.io/bbolt.(*DB).freepages.func2(0x40002c8900)
lnd                  |  go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:1003 +0xc4
lnd                  | created by go.etcd.io/bbolt.(*DB).freepages
lnd                  |  go.etcd.io/bbolt@v1.3.5-0.20200615073812-232d8fc87f50/db.go:1001 +0x134
lnd                  | 2021-10-27 12:23:06.573 [INF] LTND: Version: 0.13.3-beta commit=v0.13.3-beta, build=production, logging=default, debuglevel=info
lnd                  | 2021-10-27 12:23:06.574 [INF] LTND: Active chain: Bitcoin (network=mainnet)
lnd                  | 2021-10-27 12:23:06.624 [INF] RPCS: RPC server listening on 0.0.0.0:10009
lnd                  | 2021-10-27 12:23:06.627 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080
lnd                  | 2021-10-27 12:23:06.627 [INF] LTND: Opening the main database, this might take a few minutes...
lnd                  | 2021-10-27 12:23:06.627 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
lnd                  | 2021-10-27 12:23:06.850 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:06.853 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:06.855 [ERR] RPCS: [/lnrpc.Lightning/ListPayments]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:06.859 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:06.879 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:06.880 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:06.930 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:06.930 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:07.333 [ERR] RPCS: [/lnrpc.Lightning/GetTransactions]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:07.335 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:07.355 [ERR] RPCS: [/lnrpc.Lightning/ListInvoices]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:07.355 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:07.389 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:07.390 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available
lnd                  | 2021-10-27 12:23:07.490 [ERR] RPCS: [/lnrpc.Lightning/ListPayments]: waiting to start, RPC services not available
lnd                  | panic: freepages: failed to get all reachable pages (page 549755826386: out of bounds: 50852)
lnd                  |
9

Hi DanOnthemoon i don’t see what you wrote…

Regarding my issue the umbrel is fully STUCK. After more than 10 days trying to get it working again i think i will stop and take my loss (Lightning channel dead) and look for another more stable solution for running my Bitcoin node… From my experience Umbrel is not enough stable yet to put some real money on it .
The docker thing break everything, tor seem lost because of some config issues and LND become stuck too…

lnd | panic: freepages: failed to get all reachable pages (page 140771848108375: out of bounds: 50852)

10

Sorry, I deleted my earlier comment because after re-reading your issue I didn’t think my response would have helped. I had upgraded to 0.4.4 after which the blockchain stopped syncing and kept repeating previous blocks after getting to within 5 or so blocks of the latest block. My solution was to fully resync the blockchain and then perform a manual installation of 0.4.6. In your case it looks like you have a different issue that doesn’t match my experience.

11

One thing I can comment on is that 4GB is probably not sufficient to run an Umbrel node with a suite of apps. I kept having crashes when I was on a 4GB Pi and my problems stopped when I swapped out the hardware for one with 8GB.

12

Hi Dan, Thanks for the feedback, i will check for upgrading the hardware to 8GB RAM if the other solutions keep crashing…

13

I’m glad i didn’t put a lot of money on it … i already lost around 500K sats because of the LND backup corrupted. Fingers crossed the 24 words wallet is recoverable somewhere else.
I hope other newbies won’t lose too much money because of theses weird random crashs after upgrades.

The docker things must be done in a more solid way, the 3 millions $ should help to have more people on the project to be in production mode.

I’m going to try other Bitcoin solutions around, if anyone have recommendations ?