IBD never ends, keeps reverting back

My SATA controller failed, and I swapped a new SATA controller and the node reboots correctly and proceeded to IBD for the blocks that I missed.

I realized that the node keeps trying to do IBD, even after it says it is 100% it would freeze, and then when I rebooted it goes back to 99.95%.

And then I caught the log as follow:
electrs | [2022-02-18T20:45:31.693Z INFO electrs::daemon] waiting for 36 blocks to download
electrs | [2022-02-18T20:45:32.697Z INFO electrs::daemon] waiting for 36 blocks to download
electrs | [2022-02-18T20:45:33.701Z INFO electrs::daemon] waiting for 36 blocks to download
electrs | [2022-02-18T20:45:51.020Z INFO electrs::db] closing DB at /data/db/bitcoin
electrs | Error: electrs failed
electrs |
electrs | Caused by:
electrs | 0: bitcoind RPC polling failed
electrs | 1: daemon not available
electrs | 2: JSON-RPC error: transport error: Didn’t receive response data in time, timed out.
electrs | Starting electrs 0.9.4 on aarch64 linux with Config { network: Bitcoin, db_path: “/data/db/bitcoin”, daemon_dir: “/data/.bitcoin”, daemon_auth: CookieFile("/data/.bitcoin/.cookie"), daemon_rpc_addr: 10.21.21.8:8332, daemon_p2p_addr: 10.21.21.8:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: None, reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, disable_electrum_rpc: false, server_banner: “Umbrel v0.4.14”, args: [] }
electrs | [2022-02-18T20:45:54.820Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
electrs | [2022-02-18T20:45:54.820Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
electrs | [2022-02-18T20:45:55.676Z INFO electrs::db] “/data/db/bitcoin”: 10145 SST files, 4.815686268 GB, 0.398189233 Grows
electrs | [2022-02-18T20:45:58.278Z INFO electrs::chain] loading 355760 headers, tip=00000000000000000531b8bd110cde244c4692da66b863005b1c2cc8cddfc0ae

Not sure if this is with any issues with heat, this dood is running at 73C, probably too high.

Does any one have any idea?

You shared only a small part of your logs, so is hard to have a diagnostic.
If you have a bad internet connection, I suggest you to take in consideration this guide, for a hybrid mode clearnet + Tor. At least for the IBD.

It’s not about bad connection I am sure, my node was sync’d and functional for quite a while until the SATA driver failed.

Here is the full log from umbrel:

Startup service logs

Feb 18 20:08:19 umbrel umbrel startup[1015]: Creating nginx … done
Feb 18 20:08:19 umbrel umbrel startup[1015]: Creating bitcoin …
Feb 18 20:08:19 umbrel umbrel startup[1015]: Creating lnd … done
Feb 18 20:08:20 umbrel umbrel startup[1015]: Creating bitcoin … done
Feb 18 20:08:20 umbrel umbrel startup[1015]: Creating middleware …
Feb 18 20:08:20 umbrel umbrel startup[1015]: Creating neutrino-switcher …
Feb 18 20:08:37 umbrel umbrel startup[1015]: Creating neutrino-switcher … done
Feb 18 20:08:38 umbrel umbrel startup[1015]: Creating middleware … done
Feb 18 20:08:38 umbrel umbrel startup[1015]: Removing status server iptables entry…
Feb 18 20:08:38 umbrel umbrel startup[1015]: Removed existing iptables entry.
Feb 18 20:08:38 umbrel umbrel startup[1015]: Starting installed apps…
Feb 18 20:08:38 umbrel umbrel startup[1015]: Starting app ride-the-lightning…
Feb 18 20:08:38 umbrel umbrel startup[1015]: Starting app btc-rpc-explorer…
Feb 18 20:08:38 umbrel umbrel startup[1015]: Starting app thunderhub…
Feb 18 20:08:44 umbrel umbrel startup[1015]: Creating thunderhub_web_1 …
Feb 18 20:08:44 umbrel umbrel startup[1015]: Creating btc-rpc-explorer_web_1 …
Feb 18 20:08:45 umbrel umbrel startup[1015]: Creating ride-the-lightning_loop_1 …
Feb 18 20:08:45 umbrel umbrel startup[1015]: Creating ride-the-lightning_boltz_1 …
Feb 18 20:08:45 umbrel umbrel startup[1015]: Creating ride-the-lightning_web_1 …
Feb 18 20:08:48 umbrel umbrel startup[1015]: Creating btc-rpc-explorer_web_1 … done
Feb 18 20:08:48 umbrel umbrel startup[1015]: Creating thunderhub_web_1 … done
Feb 18 20:08:50 umbrel umbrel startup[1015]: Creating ride-the-lightning_loop_1 … done
Feb 18 20:08:50 umbrel umbrel startup[1015]: Creating ride-the-lightning_boltz_1 … done
Feb 18 20:08:50 umbrel umbrel startup[1015]: Creating ride-the-lightning_web_1 … done
Feb 18 20:08:50 umbrel umbrel startup[1015]: Umbrel is now accessible at
Feb 18 20:08:50 umbrel umbrel startup[1015]: http://///umbrel.local
Feb 18 20:08:50 umbrel umbrel startup[1015]: http://///192.168.0.103
Feb 18 20:08:50 umbrel systemd[1]: Started Umbrel Startup Service.
Feb 18 20:09:23 umbrel passwd[5076]: pam_unix(passwd:chauthtok): password changed for umbrel

External storage SD card update service logs

– Logs begin at Fri 2022-02-18 20:07:48 UTC, end at Fri 2022-02-18 21:10:07 UTC. –
Feb 18 20:08:08 umbrel systemd[1]: Starting External Storage SDcard Updater…
Feb 18 20:08:08 umbrel external storage updater[934]: Checking if SD card Umbrel is newer than external storage…
Feb 18 20:08:09 umbrel external storage updater[934]: No, SD version is not newer, exiting.
Feb 18 20:08:09 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs

ERROR: for dashboard removal of container 4abed75de3917b84613098648d2cd3a62667d31d772a42e28bd53a36e9f5d5d3 is already in progress

ERROR: for umbrel_tor_server_1 No such container: 8ed70954719058c6b987e12eacdbc799a9a6b523f425389e153fbf9505e18b85
Removing network umbrel_main_network
Removing nginx … done
Removing lnd … error
Removing dashboard … error

ERROR: for tor removal of container 0b9f40be8bb77906049921e34d398121e109cd93a80b03f0b90e8f93523778f7 is already in progress

ERROR: for electrs removal of container dec33ac15dae8754e30641a1dee66f38eb6a8779c804b89a39a48a9e44cfb3e3 is already in progress

ERROR: for bitcoin removal of container 11876d502a7ef5fdc34c2076c52ca9942a83511990eca2174ceeca91f9db7859 is already in progress

ERROR: for middleware removal of container 8f5404122b614d5d0d0649ca6aa81388f6c317eee28b14864156e6793ea581cc is already in progress

ERROR: for umbrel_tor_server_1 removal of container 8ed70954719058c6b987e12eacdbc799a9a6b523f425389e153fbf9505e18b85 is already in progress

ERROR: for manager No such container: c72bf740e13c7b11799f02d891d2c3648224fb6df97d259a0a113b9b5a8c027f

ERROR: for neutrino-switcher removal of container 1e7a2f214b00bce3350682aa52c215ab2a7cbc9392273e4d77d5e12333e10667 is already in progress

ERROR: for lnd No such container: 13ddae3caf4220cf0fc39c84469c3838c8c655c47996b362cff91bd189cd9d7b

ERROR: for dashboard No such container: 4abed75de3917b84613098648d2cd3a62667d31d772a42e28bd53a36e9f5d5d3
Removing network umbrel_main_network
Network umbrel_main_network not found.
Network umbrel_main_network not found.
Network umbrel_main_network not found.
karen is running in /home/umbrel/umbrel/events
Got signal:
karen is getting triggered!
./karen: line 68: /home/umbrel/umbrel/events/triggers/: Is a directory
Got signal:
karen is getting triggered!
./karen: line 68: /home/umbrel/umbrel/events/triggers/: Is a directory
Got signal: change-password
karen is getting triggered!
New password: Retype new password: passwd: password updated successfully
Got signal: debug
karen is getting triggered!
Debug result file generated
Got signal: debug
karen is getting triggered!
Debug result file generated
Got signal: change-password
karen is getting triggered!
New password: Retype new password: passwd: password updated successfully
Got signal: debug
karen is getting triggered!

Docker containers

NAMES STATUS
ride-the-lightning_boltz_1 Up About an hour
ride-the-lightning_web_1 Up About an hour
ride-the-lightning_loop_1 Up 10 minutes
btc-rpc-explorer_web_1 Up About an hour
thunderhub_web_1 Up About an hour
middleware Up About an hour
bitcoin Up 11 minutes
lnd Up 11 minutes
nginx Up About an hour
manager Up About an hour
umbrel_tor_server_1 Up About an hour
tor Up About an hour
dashboard Up About an hour
electrs Up 5 minutes

Umbrel logs

Attaching to middleware, manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/system/get-update HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/system/storage HTTP/1.0” 200 206 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
middleware | at Object.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:1210:9)
middleware | at InterceptingListener._callNext (/app/node_modules/grpc/src/client_interceptors.js:568:42)
middleware | at InterceptingListener.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:618:8)
middleware | at callback (/app/node_modules/grpc/src/client_interceptors.js:847:24)
middleware | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/lnd/wallet/btc HTTP/1.0” 500 103 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
middleware |
middleware | umbrel-middleware
middleware | Unable to get payments, cannot retrieve macaroon: cannot get macaroon: macaroon store is locked
middleware | /v1/lnd/lightning/payments
middleware | LndError: Unable to get payments
middleware | at /app/services/lnd.js:69:18
middleware | at Object.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:1210:9)
middleware | at InterceptingListener._callNext (/app/node_modules/grpc/src/client_interceptors.js:568:42)
middleware | at InterceptingListener.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:618:8)
middleware | at callback (/app/node_modules/grpc/src/client_interceptors.js:847:24)
middleware | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/lnd/lightning/payments HTTP/1.0” 500 97 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
middleware |
middleware | umbrel-middleware
middleware | Unable to list channels, database not open
middleware | /v1/lnd/transaction
middleware | LndError: Unable to list channels
middleware | at /app/services/lnd.js:69:18
middleware | at Object.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:1210:9)
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/system/temperature HTTP/1.0” 200 2 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/system/debug-result HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/system/update-status HTTP/1.0” 200 102 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:56 GMT] “GET /v1/system/debug-result HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:56 GMT] “GET /v1/external/price HTTP/1.0” 200 13 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:57 GMT] “GET /v1/system/debug-result HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:58 GMT] “GET /v1/system/debug-result HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:59 GMT] “GET /v1/system/debug-result HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
manager |
manager | umbrel-manager
middleware | at InterceptingListener._callNext (/app/node_modules/grpc/src/client_interceptors.js:568:42)
middleware | at InterceptingListener.onReceiveStatus (/app/node_modules/grpc/src/client_interceptors.js:618:8)
middleware | at callback (/app/node_modules/grpc/src/client_interceptors.js:847:24)
middleware | ::ffff:10.21.21.2 - - [Fri, 18 Feb 2022 21:10:55 GMT] “GET /v1/lnd/transaction HTTP/1.0” 500 44 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36”
middleware |
middleware | umbrel-middleware
middleware | LndUnlocker: Wallet failed to unlock!

Bitcoin Core logs

Attaching to bitcoin
bitcoin | 2022-02-18T21:10:59Z Config file arg: zmqpubrawblock=“tcp://0.0.0.0:28332”
bitcoin | 2022-02-18T21:10:59Z Config file arg: zmqpubrawtx=“tcp://0.0.0.0:28333”
bitcoin | 2022-02-18T21:10:59Z Command-line arg: zmqpubrawblock=“tcp://0.0.0.0:28332”
bitcoin | 2022-02-18T21:10:59Z Command-line arg: zmqpubrawtx=“tcp://0.0.0.0:28333”
bitcoin | 2022-02-18T21:10:59Z Using at most 125 automatic connections (1048576 file descriptors available)
bitcoin | 2022-02-18T21:10:59Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
bitcoin | 2022-02-18T21:10:59Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
bitcoin | 2022-02-18T21:10:59Z Script verification uses 3 additional threads
bitcoin | 2022-02-18T21:10:59Z scheduler thread start
bitcoin | 2022-02-18T21:10:59Z HTTP: creating work queue of depth 64
bitcoin | 2022-02-18T21:10:59Z Using random cookie authentication.
bitcoin | 2022-02-18T21:11:00Z Generated RPC authentication cookie /data/.bitcoin/.cookie
bitcoin | 2022-02-18T21:11:00Z Using rpcauth authentication.
bitcoin | 2022-02-18T21:11:00Z HTTP: starting 4 worker threads
bitcoin | 2022-02-18T21:11:00Z Using wallet directory /data/.bitcoin
bitcoin | 2022-02-18T21:11:00Z init message: Verifying wallet(s)…
bitcoin | 2022-02-18T21:11:00Z init message: Loading banlist…
bitcoin | 2022-02-18T21:11:00Z SetNetworkActive: true
bitcoin | 2022-02-18T21:11:00Z Using /16 prefix for IP bucketing
bitcoin | 2022-02-18T21:11:00Z Cache configuration:
bitcoin | 2022-02-18T21:11:00Z * Using 2.0 MiB for block index database
bitcoin | 2022-02-18T21:11:00Z * Using 24.8 MiB for transaction index database
bitcoin | 2022-02-18T21:11:00Z * Using 21.7 MiB for basic block filter index database
bitcoin | 2022-02-18T21:11:00Z * Using 8.0 MiB for chain state database
bitcoin | 2022-02-18T21:11:00Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
bitcoin | 2022-02-18T21:11:00Z init message: Loading block index…
bitcoin | 2022-02-18T21:11:00Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
bitcoin | 2022-02-18T21:11:00Z Opening LevelDB in /data/.bitcoin/blocks/index
bitcoin | 2022-02-18T21:11:00Z Opened LevelDB successfully
bitcoin | 2022-02-18T21:11:00Z Using obfuscation key for /data/.bitcoin/blocks/index: 0000000000000000

LND logs

Attaching to lnd
lnd | 2022-02-18 21:10:55.394 [INF] RPCS: Stopping InvoicesRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.407 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: unable to get best block info: Post “http://///10.21.21.8:8332”: dial tcp 10.21.21.8:8332: connect: connection refused
lnd | 2022-02-18 21:10:55.386 [ERR] LNWL: Unable to complete chain rescan: Post “http://///10.21.21.8:8332”: read tcp 10.21.21.9:43500->10.21.21.8:8332: read: connection reset by peer
lnd | 2022-02-18 21:10:55.432 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: unable to get best block info: Post “http://///10.21.21.8:8332”: dial tcp 10.21.21.8:8332: connect: connection refused
lnd | 2022-02-18 21:10:55.446 [INF] RPCS: Stopping WatchtowerRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.452 [INF] RPCS: Stopping WalletKitRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.452 [INF] RPCS: Stopping RouterRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.454 [INF] RPCS: Stopping AutopilotRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.488 [INF] RPCS: Stopping ChainRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.488 [INF] RPCS: Stopping VersionRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.488 [INF] RPCS: Stopping WatchtowerClientRPC Sub-RPC Server
lnd | 2022-02-18 21:10:55.489 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: unable to get best block info: Post “http://///10.21.21.8:8332”: dial tcp 10.21.21.8:8332: connect: connection refused
lnd | 2022-02-18 21:10:55.576 [INF] TORC: Stopping tor controller
lnd | 2022-02-18 21:10:55.595 [ERR] TORC: DEL_ONION got error: invalid arguments: unexpected code
lnd | 2022-02-18 21:10:55.611 [ERR] LTND: error stopping tor controller: invalid arguments: unexpected code
lnd | 2022-02-18 21:10:55.659 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: cannot retrieve macaroon: cannot get macaroon: macaroon store is locked
lnd | 2022-02-18 21:10:55.683 [ERR] RPCS: [/lnrpc.Lightning/ListPayments]: cannot retrieve macaroon: cannot get macaroon: macaroon store is locked
lnd | 2022-02-18 21:10:55.889 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: database not open
lnd | 2022-02-18 21:10:55.910 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: unable to get retrieve pending channels: database not open
lnd | 2022-02-18 21:10:56.064 [INF] LTND: Shutdown complete
lnd |
lnd | unable to determine if wallet is synced: Post “http://///10.21.21.8:8332”: dial tcp 10.21.21.8:8332: connect: connection refused
lnd | 2022-02-18 21:11:00.121 [INF] LTND: Version: 0.14.2-beta commit=v0.14.2-beta, build=production, logging=default, debuglevel=info
lnd | 2022-02-18 21:11:00.122 [INF] LTND: Active chain: Bitcoin (network=mainnet)
lnd | 2022-02-18 21:11:00.128 [INF] RPCS: RPC server listening on 0.0.0.0:10009
lnd | 2022-02-18 21:11:00.178 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080
lnd | 2022-02-18 21:11:00.194 [INF] LTND: Opening the main database, this might take a few minutes…
lnd | 2022-02-18 21:11:00.194 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
lnd | 2022-02-18 21:11:00.943 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available
lnd | 2022-02-18 21:11:03.444 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available

electrs logs

Attaching to electrs
electrs | [2022-02-18T21:09:38.758Z INFO electrs::daemon] waiting for 24 blocks to download
electrs | [2022-02-18T21:09:39.763Z INFO electrs::daemon] waiting for 24 blocks to download
electrs | [2022-02-18T21:09:44.975Z INFO electrs::daemon] waiting for 21 blocks to download
electrs | [2022-02-18T21:09:46.997Z INFO electrs::daemon] waiting for 20 blocks to download
electrs | [2022-02-18T21:09:52.187Z INFO electrs::daemon] waiting for 17 blocks to download
electrs | [2022-02-18T21:09:53.191Z INFO electrs::daemon] waiting for 17 blocks to download
electrs | [2022-02-18T21:09:54.197Z INFO electrs::daemon] waiting for 17 blocks to download
electrs | [2022-02-18T21:09:56.455Z INFO electrs::daemon] waiting for 15 blocks to download
electrs | [2022-02-18T21:10:04.036Z INFO electrs::daemon] waiting for 10 blocks to download
electrs | [2022-02-18T21:10:05.040Z INFO electrs::daemon] waiting for 10 blocks to download
electrs | [2022-02-18T21:10:13.259Z INFO electrs::daemon] waiting for 4 blocks to download
electrs | [2022-02-18T21:10:14.281Z INFO electrs::daemon] waiting for 4 blocks to download
electrs | [2022-02-18T21:10:15.341Z INFO electrs::daemon] waiting for 4 blocks to download
electrs | [2022-02-18T21:10:23.985Z INFO electrs::index] indexing 2000 blocks: [355761…357760]
electrs | [2022-02-18T21:10:55.384Z INFO electrs::db] closing DB at /data/db/bitcoin
electrs | Error: electrs failed
electrs |
electrs | Caused by:
electrs | 0: sync failed
electrs | 1: failed to get block 000000000000000004deab52ef437a4dc5e5c94324b9981bd124987e9b8684a2
electrs | 2: receiving on an empty and disconnected channel
electrs | Starting electrs 0.9.4 on aarch64 linux with Config { network: Bitcoin, db_path: “/data/db/bitcoin”, daemon_dir: “/data/.bitcoin”, daemon_auth: CookieFile("/data/.bitcoin/.cookie"), daemon_rpc_addr: 10.21.21.8:8332, daemon_p2p_addr: 10.21.21.8:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: None, reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, disable_electrum_rpc: false, server_banner: “Umbrel v0.4.14”, args: [] }
electrs | [2022-02-18T21:10:59.698Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
electrs | [2022-02-18T21:10:59.698Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
electrs | [2022-02-18T21:11:00.785Z INFO electrs::db] “/data/db/bitcoin”: 10150 SST files, 4.815690293 GB, 0.398189233 Grows
electrs | [2022-02-18T21:11:03.779Z INFO electrs::chain] loading 355760 headers, tip=00000000000000000531b8bd110cde244c4692da66b863005b1c2cc8cddfc0ae
electrs | [2022-02-18T21:11:06.923Z INFO electrs::chain] chain updated: tip=00000000000000000531b8bd110cde244c4692da66b863005b1c2cc8cddfc0ae, height=355760
electrs | [2022-02-18T21:11:06.994Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2022-02-18T21:11:07.999Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2022-02-18T21:11:09.002Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…

Tor logs

tor_server_1 | Feb 18 20:09:36.000 [warn] Guard TheEpTicZeus ($77A56CB237740E24AEA2D61C8C8936232AFC1BD8) is failing a very large amount of circuits. Most likely this means the Tor network is overloaded, but it could also mean an attack against you or potentially the guard itself. Success counts are 96/193. Use counts are 61/61. 96 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out. For reference, your timeout cutoff is 60 seconds.
tor_server_1 | Feb 18 20:09:36.000 [warn] Guard TheEpTicZeus ($77A56CB237740E24AEA2D61C8C8936232AFC1BD8) is failing an extremely large amount of circuits. This could indicate a route manipulation attack, extreme network overload, or a bug. Success counts are 95/317. Use counts are 61/61. 95 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out. For reference, your timeout cutoff is 60 seconds.
tor_server_1 | Feb 18 20:09:38.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 103 buildtimes.
tor_server_1 | Feb 18 20:09:39.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 103 buildtimes.
tor_server_1 | Feb 18 20:09:40.000 [notice] Guard eclipse02 ($69B53A5AE5025DA8947F7C32F69CA90375ED2A4E) is failing more circuits than usual. Most likely this means the Tor network is overloaded. Success counts are 102/151. Use counts are 63/63. 102 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out. For reference, your timeout cutoff is 60 seconds.
tor_server_1 | Feb 18 20:09:41.000 [warn] Guard eclipse02 ($69B53A5AE5025DA8947F7C32F69CA90375ED2A4E) is failing a very large amount of circuits. Most likely this means the Tor network is overloaded, but it could also mean an attack against you or potentially the guard itself. Success counts are 102/205. Use counts are 63/63. 102 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out. For reference, your timeout cutoff is 60 seconds.
tor_server_1 | Feb 18 20:09:42.000 [notice] Guard wardsback ($B143D439B72D239A419F8DCE07B8A8EB1B486FA7) is failing more circuits than usual. Most likely this means the Tor network is overloaded. Success counts are 102/151. Use counts are 60/60. 102 circuits completed, 0 were unusable, 0 collapsed, and 1 timed out. For reference, your timeout cutoff is 60 seconds.
tor_server_1 | Feb 18 20:09:43.000 [warn] Guard wardsback ($B143D439B72D239A419F8DCE07B8A8EB1B486FA7) is failing a very large amount of circuits. Most likely this means the Tor network is overloaded, but it could also mean an attack against you or potentially the guard itself. Success counts are 102/205. Use counts are 60/60. 102 circuits completed, 0 were unusable, 0 collapsed, and 1 timed out. For reference, your timeout cutoff is 60 seconds.
tor_server_1 | Feb 18 20:09:44.000 [warn] Guard wardsback ($B143D439B72D239A419F8DCE07B8A8EB1B486FA7) is failing an extremely large amount of circuits. This could indicate a route manipulation attack, extreme network overload, or a bug. Success counts are 98/327. Use counts are 62/62. 98 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out. For reference, your timeout cutoff is 60 seconds.

App logs

btc-rpc-explorer

web_1 | 2022-02-18T20:40:05.740Z btcexp:app Skipping performance-intensive task: fetch UTXO set summary. This is skipped due to the flag ‘slowDeviceMode’ which defaults to ‘true’ to protect slow nodes. Set this flag to ‘false’ to enjoy UTXO set summary details.
web_1 | 2022-02-18T20:45:51.350Z btcexp:error Error 937gf47dsyde: Error: close connect, json: {}, userData: [object Object] (json: {“host”:“10.21.21.10”,“port”:50001,“protocol”:“tcp”})
web_1 | 2022-02-18T20:45:51.363Z btcexp:errorVerbose Stack: Error: close connect
web_1 | at /data/node_modules/electrum-client/lib/client.js:192:37
web_1 | at Array.forEach ()
web_1 | at ElectrumClient.onClose (/data/node_modules/electrum-client/lib/client.js:191:44)
web_1 | at ElectrumClient.onClose (/data/node_modules/electrum-client/index.js:69:9)
web_1 | at Socket. (/data/node_modules/electrum-client/lib/client.js:68:9)
web_1 | at Socket.emit (events.js:314:20)
web_1 | at TCP. (net.js:675:12)
web_1 | 2022-02-18T21:04:28.539Z btcexp:error Error 937gf47dsyde: Error: close connect, json: {}, userData: [object Object] (json: {“host”:“10.21.21.10”,“port”:50001,“protocol”:“tcp”})
web_1 | 2022-02-18T21:04:28.555Z btcexp:errorVerbose Stack: Error: close connect
web_1 | at /data/node_modules/electrum-client/lib/client.js:192:37
web_1 | at Array.forEach ()
web_1 | at ElectrumClient.onClose (/data/node_modules/electrum-client/lib/client.js:191:44)
web_1 | at ElectrumClient.onClose (/data/node_modules/electrum-client/index.js:69:9)
web_1 | at Socket. (/data/node_modules/electrum-client/lib/client.js:68:9)
web_1 | at Socket.emit (events.js:314:20)
web_1 | at TCP. (net.js:675:12)
web_1 | 2022-02-18T21:05:02.924Z btcexp:error Error 937gf47dsyde: Error: close connect, json: {}, userData: [object Object] (json: {“host”:“10.21.21.10”,“port”:50001,“protocol”:“tcp”})
web_1 | 2022-02-18T21:05:02.924Z btcexp:errorVerbose Stack: Error: close connect
web_1 | at /data/node_modules/electrum-client/lib/client.js:192:37
web_1 | at Array.forEach ()
web_1 | at ElectrumClient.onClose (/data/node_modules/electrum-client/lib/client.js:191:44)
web_1 | at ElectrumClient.onClose (/data/node_modules/electrum-client/index.js:69:9)
web_1 | at Socket. (/data/node_modules/electrum-client/lib/client.js:68:9)
web_1 | at Socket.emit (events.js:314:20)
web_1 | at TCP. (net.js:675:12)
web_1 | 2022-02-18T21:10:05.725Z btcexp:app Skipping performance-intensive task: fetch last 24 hrs of blockstats to calculate transaction volume. This is skipped due to the flag ‘slowDeviceMode’ which defaults to ‘true’ to protect slow nodes. Set this flag to ‘false’ to enjoy UTXO set summary details.
web_1 | 2022-02-18T21:10:05.743Z btcexp:app Skipping performance-intensive task: fetch UTXO set summary. This is skipped due to the flag ‘slowDeviceMode’ which defaults to ‘true’ to protect slow nodes. Set this flag to ‘false’ to enjoy UTXO set summary details.

ride-the-lightning

loop_1 | 2022-02-18 21:00:22.701 [INF] LNDC: lnd version: v0.14.2-beta, build tags ‘autopilotrpc,signrpc,walletrpc,chainrpc,invoicesrpc,watchtowerrpc,monitoring,kvdb_postgres,kvdb_etcd’
loop_1 | 2022-02-18 21:00:22.702 [INF] LNDC: Using network mainnet
loop_1 | 2022-02-18 21:00:22.702 [INF] LNDC: Waiting for lnd to be fully synced to its chain backend, this might take a while
loop_1 | loopd exited with an error: error waiting for chain to be synced: error in GetInfo call: rpc error: code = Unknown desc = unable to get best block info: Post “http://///10.21.21.8:8332”: dial tcp 10.21.21.8:8332: connect: connection refused
loop_1 | 2022-02-18 21:10:59.825 [INF] LOOPD: Version: 0.12.1-beta commit=
loop_1 | 2022-02-18 21:10:59.827 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009
loop_1 | 2022-02-18 21:10:59.831 [INF] LNDC: Connected to lnd
loop_1 | 2022-02-18 21:10:59.831 [INF] LNDC: Waiting for lnd to unlock
loop_1 | loopd exited with an error: rpc error: code = Unknown desc = waiting to start, RPC services not available
loop_1 | 2022-02-18 21:11:03.349 [INF] LOOPD: Version: 0.12.1-beta commit=
loop_1 | 2022-02-18 21:11:03.351 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009
loop_1 | 2022-02-18 21:11:03.353 [INF] LNDC: Connected to lnd
loop_1 | 2022-02-18 21:11:03.354 [INF] LNDC: Waiting for lnd to unlock
loop_1 | loopd exited with an error: rpc error: code = Unknown desc = waiting to start, RPC services not available
loop_1 | 2022-02-18 21:11:06.145 [INF] LOOPD: Version: 0.12.1-beta commit=
loop_1 | 2022-02-18 21:11:06.152 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009
loop_1 | 2022-02-18 21:11:06.153 [INF] LNDC: Connected to lnd
loop_1 | 2022-02-18 21:11:06.154 [INF] LNDC: Waiting for lnd to unlock
loop_1 | loopd exited with an error: rpc error: code = Unknown desc = waiting to start, RPC services not available
loop_1 | 2022-02-18 21:11:09.196 [INF] LOOPD: Version: 0.12.1-beta commit=
loop_1 | 2022-02-18 21:11:09.197 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009
loop_1 | 2022-02-18 21:11:09.198 [INF] LNDC: Connected to lnd
loop_1 | 2022-02-18 21:11:09.199 [INF] LNDC: Waiting for lnd to unlock
loop_1 | loopd exited with an error: rpc error: code = Unknown desc = waiting to start, RPC services not available
loop_1 | 2022-02-18 21:11:12.622 [INF] LOOPD: Version: 0.12.1-beta commit=
loop_1 | 2022-02-18 21:11:12.623 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009
loop_1 | 2022-02-18 21:11:12.624 [INF] LNDC: Connected to lnd
web_1 | Server is up and running, please open the UI at http://///localhost:3001
loop_1 | 2022-02-18 21:11:12.626 [INF] LNDC: Waiting for lnd to unlock
loop_1 | loopd exited with an error: rpc error: code = Unknown desc = waiting to start, RPC services not available

Let’s clear some aspects:

  1. Please don’t post directly here all the logs, copy/paste. Is terrible to read like that. Use instead 0bin.net or pastebin. 0bin is better because it keeps formatted the text. Post here only the link to that 0bin log. Is nothing that can compromise your privacy in those logs.
  2. Read for yourself the logs. Try to learn them and understand them. Will help you a lot for future use. Is not so hard to read and see what is happening. You don’t need to be a coder to read that. Just pay attention.
  3. sometimes only the excerpt from the general debug log is NOT enough. So start running separate logs for each main modules of your node: bitcoin, lnd, electrs, tor So you will obtain more details.
    See the Umbrel troubleshooting manual about those commands and many more items that can help you identify and diagnose your issue.
  4. Most of the issues of not sync comes from BAD HARDWARE. This is very serious and many users do not want to accept it. But in 99% cases is from that.
  5. Consider also assigning a static IP to your node. That will fix a lot of connectivity issues. You can find the guide how to do that in the Guides section of this forum and many other good guides.
2 Likes