Problem: Umbrel disaster recovery

[el_revengero]

Hello support,
I will describe the accident happened, so my umbrel runs on ubuntu server docker instance, this server is part of 3 servers cluster at office and runs under firewall.
Yesterday at 6 UTC a fuse cutted off electricity and ups died at 8. This seems to be related to the failure of a RAID controller.

At 16 UTC the sysop put online my server, but strangly umbrel was already running (restored from a memory snapshot? maybe) after the sync of the missing blocks I couldn’t access to wallets, some “APPS” didn’t start and thunderhub won’t let me login.

i runned sudo ./stop and start and waited the night.

This morning the situation was the same, so i stopped and checked htop , surprisingly i had containers running so with “docker container ls” i discovered it and stopped it (not kill).

started umbrel again not risolved , but i’ve noticed LND and others keep restarting, downloaded the log file, opened it and for what i can understood there are multiple reference to the mariadb and wallet it’s locked.
Log it’s below, i’ve replace my internal IPs!

debug - Pastebin.com

So, how can i fix this?
Thanks in advance

[DarthCoin]

Did you checked in the Guides section of this forum the Troubleshooting manual for Umbrel ?

[el_revengero]

yes, but there is no single entry that match 100% what happened, the nearest matches are " some dockers components fails to start" and “After a bad shutdown the node it’s not syncing” .
So what do you suggest? thxs

[DarthCoin]

Apply the manual update.
See the instruction in that same manual.
Update first the OS and after that umbrel to latest version.
If the blocks data and LN channels db files are intact, nothing to worry.
Just apply the fresh update to fix the corrupted files (if there are) and start your node.

[el_revengero]

ehy thanks for the prompt reply!
Just for double checking you refer to this right?
" B. For normal Linux OS users:

• Make a channels backup. Go to Umbrel dashboard - Lightning and on the right top side, 3 dots, save backup on your local PC.
• Stop Umbrel node: sudo ./scripts/stop
• Save your lnd.conf file if you edited (add color, name or specific features to your node). How to edit lnd.conf in Umbrel. More details about what you can add to a lnd.conf file here . After version 0.4.0 this file is not overwritten anymore with a default one, but just in case, make a backup.
• Update system: sudo apt update and then sudo apt upgrade
• Restart system
• Update Umbrel (replace version with the latest release):
  cd ~/umbrel && sudo ./scripts/update/update --repo getumbrel/umbrel#v0.4.2
• Replace lnd.conf file with the one you save it before update (After version 0.4.0 this file is not overwritten anymore with a default one)
• Start Umbrel: sudo ./scripts/start
• Leave the node to catch up with blocks and logs, usually takes several minutes, be patient. Now you can enter into dashboard Umbrel

C. Just re-flash the mSD card with the latest Umbrel OS version, like you did when you install it first time.
" ?

[DarthCoin]

B is for you. As I understan you have your node in a rack with RAID disks and linux machine not a rasp pi.
reminder: we are at v0.4.4 now

[el_revengero]

yes thxs, the instance was already updated

[DarthCoin]

doesn’t matter, by re-applying the update it can fix broken files in docker repository.
Also sudo apt update && upgrade is bringing more fixes for docker. All the time you should do those updates before any update of umbrel.

[el_revengero]

OK, i’ve updated, still the same errors, I’ve the BTC seed words, tested on bluwallet working… and the channels backup made on the 0.4.4 update.
Do you suggest to erase everything and start with a fresh install? Any way to backup the blockchain and the btc wallet without typing the seed words?

Or at least prune the blockchain (not a big issue , it will take 2 days downloading from the start, still worth save some time)

[DarthCoin]

You can keep the blockchain no problem.
There’s a way to reset only user data not the whole blockchain data, but not sure if that will fix your problem. See Umbrel troubleshooting manual about how to reset user data.

Here is also a guide with 6 ways how to restore your funds node in case of SHTF. Be careful how you do it.
But anyway, if you have the 24 words seed and a latest backup channel file (from Umbrel - Lightning menu, not from RTL or Thunderhub) you are safe, with those you can restore your funds easily.

[el_revengero]

sorry… but restoring the btc wallet i 've got the error message “user already exist” guessing the are more things to erase or not?

[el_revengero]

Opened another browser and login, now my username it’s my LN name, weird…
On chain wallet not showing up, blockchain synced
Still not working, should i go for a full reset?

[el_revengero]

I reinstalled from scratch, now the balance on the upper left shows $0 and Nan.
I’m stopped to step 1 in recovery with a fresh node, should i proceed forward?
the node it’s synced at block 705644, so it’s really synced
note now the balance it’s zero without Nan

[DarthCoin]

After blockchain is synced, you still have to wait until electrum server is indexing the blocks.
That can take up to 72h.
Check the electrs logs to see at what height it is.

[el_revengero]

where to check the logs?
any cli command?

[DarthCoin]

https://community.getumbrel.com/t/umbrel-troubleshooting-guide/3632/2

[el_revengero]

indent preformatted text by 4 spaces Attaching to electrs

electrs | Config { network: Bitcoin, db_path: “/data/db/bitcoin”, daemon_dir: “/data/.bitcoin”, daemon_auth: CookieFile("/data/.bitcoin/.cookie"), daemon_rpc_addr: V4(10.21.21.8:8332), daemon_p2p_addr: V4(10.21.21.8:8333), electrum_rpc_addr: V4(0.0.0.0:50001), monitoring_addr: V4(127.0.0.1:4224), wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(200), auto_reindex: true, ignore_mempool: false, sync_once: false, server_banner: “Umbrel v0.4.4”, args: [] }
electrs | [2021-10-19T06:55:08.059Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
electrs | [2021-10-19T06:55:09.006Z INFO electrs::db] “/data/db/bitcoin”: 394 SST files, 24.350481048 GB, 2.010645002 Grows
electrs | [2021-10-19T06:55:10.994Z INFO electrs::chain] loading 540840 headers, tip=0000000000000000000b045030a622c9f0b0fc51122a4475010088fb20a8bc25
electrs | [2021-10-19T06:55:13.096Z INFO electrs::chain] chain updated: tip=0000000000000000000b045030a622c9f0b0fc51122a4475010088fb20a8bc25, height=540840
electrs | [2021-10-19T06:55:13.112Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2021-10-19T06:55:14.113Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2021-10-19T06:55:15.114Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2021-10-19T06:55:16.115Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2021-10-19T06:55:17.116Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2021-10-19T06:55:18.117Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2021-10-19T06:55:19.118Z INFO electrs::daemon] waiting for RPC warmup: Loading block index…
electrs | [2021-10-19T06:55:20.119Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…

[el_revengero]

electrs | [2021-10-19T09:11:33.180Z INFO electrs::index] indexing 2000 blocks: [698841…700840]
electrs | [2021-10-19T09:13:05.622Z INFO electrs::chain] chain updated: tip=0000000000000000000bbbc7fc14c75488066307c040d47c18fe7bbc4cf070f7, height=700840
electrs | [2021-10-19T09:13:05.633Z INFO electrs::index] indexing 2000 blocks: [700841…702840]
electrs | [2021-10-19T09:14:36.094Z INFO electrs::chain] chain updated: tip=000000000000000000011ef06268ab708dce37d230e29a5ab28f68bbb552aeb8, height=702840
electrs | [2021-10-19T09:14:36.109Z INFO electrs::index] indexing 2000 blocks: [702841…704840]
electrs | [2021-10-19T09:16:15.382Z INFO electrs::chain] chain updated: tip=0000000000000000000e1dfde8329b2477c362732e468ac46d97e221a8b535be, height=704840
electrs | [2021-10-19T09:16:15.392Z INFO electrs::index] indexing 819 blocks: [704841…705659]
electrs | [2021-10-19T09:16:56.324Z INFO electrs::chain] chain updated: tip=0000000000000000000816b79022817e6e966859bfcc2b6c192c9bb0dfb1f654, height=705659
electrs | [2021-10-19T09:17:02.958Z INFO electrs::db] starting config compaction
electrs | [2021-10-19T09:17:03.080Z INFO electrs::db] starting headers compaction
electrs | [2021-10-19T09:17:05.403Z INFO electrs::db] starting txid compaction

[DarthCoin]

so is still long way to finish. Just wait until it reaches last block and after that will appear a message like “compacting database” wait 1-2h more until is finishing.
Patience, don’t do anything until then.

[el_revengero]

is it finished?

electrs | [2021-10-19T09:17:02.958Z INFO electrs::db] starting config compaction
electrs | [2021-10-19T09:17:03.080Z INFO electrs::db] starting headers compaction
electrs | [2021-10-19T09:17:05.403Z INFO electrs::db] starting txid compaction
electrs | [2021-10-19T09:29:09.723Z INFO electrs::db] starting funding compaction
electrs | [2021-10-19T09:57:27.895Z INFO electrs::db] starting spending compaction
electrs | [2021-10-19T10:25:53.777Z INFO electrs::db] finished full compaction
electrs | [2021-10-19T10:25:59.328Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001
electrs | [2021-10-19T10:25:59.338Z INFO electrs::index] indexing 9 blocks: [705660…705668]
electrs | [2021-10-19T10:25:59.819Z INFO electrs::chain] chain updated: tip=0000000000000000000c42baebcf8fc4a63a9d6ad6e5ab83cf3213eef28f8405, height=705668
electrs | [2021-10-19T10:26:54.428Z INFO electrs::index] indexing 1 blocks: [705669…705669]
electrs | [2021-10-19T10:26:54.568Z INFO electrs::chain] chain updated: tip=0000000000000000000204abe15f8ef5ea227afe4880ac244a24b9ae41e77852, height=705669