Umbrel Keep Crashing

Support and Troubleshooting
1

I’m hesitant to ask for help in here, but here it goes.

I have been trying to run an Umbrel node for several months now. Yes, the setup is very simple and user friendly but my Umbrel crashes basically every other week. Out of the months that I have had it, the longest uptime it has had was about 1 week before crashing again.

Each time it crashes I dig through logs and read through the community forum to find out what has gone wrong. After spending hours trying to solve the problem, I never find a solution other than starting over from scratch.

I have restarted from scratch 4 times already and would like to know what I am doing wrong and how I can move forward without having to start over again.

I ran ~/umbrel/scripts/debug --upload , but it did not give me a link to share in the group (not that I can find at least) … So please don’t roast me for posting the logs here directly in the forum as I know that it is not proper etiquette.

=====================
= Umbrel debug info =

Umbrel version

0.4.18

Flashed OS version

v0.4.18

Raspberry Pi Model

Revision : d03115
Serial : 1000000015dd0624
Model : Raspberry Pi 4 Model B Rev 1.5

Firmware

Dec 1 2021 15:01:54
Copyright © 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

Temperature

temp=36.5’C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 7.8G 135M 7.1G 8.0M 581M 7.6G
Swap: 0B 0B 0B

total: 1.7%
system: 1.7%
tor: 0%
lnd: 0%
electrs: 0%
bitcoin: 0%

Memory monitor logs

tail: cannot open ‘/home/umbrel/umbrel/scripts/…/logs/memory-monitor.log’ for reading: No such file or directory

Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/root 118G 3.2G 110G 3% /
/dev/root 118G 3.2G 110G 3% /

Startup service logs

– Logs begin at Fri 2022-05-13 16:44:08 UTC, end at Sun 2022-05-22 16:58:52 UTC. –
May 13 16:44:24 umbrel systemd[1]: Dependency failed for Umbrel Startup Service.
May 13 16:44:24 umbrel systemd[1]: umbrel-startup.service: Job umbrel-startup.service/start failed with result ‘dependency’.

External storage service logs

– Logs begin at Fri 2022-05-13 16:44:08 UTC, end at Sun 2022-05-22 16:58:52 UTC. –
May 13 16:44:16 umbrel systemd[1]: Starting External Storage Mounter…
May 13 16:44:16 umbrel external storage mounter[509]: Running external storage mount script…
May 13 16:44:18 umbrel external storage mounter[509]: Found device “Samsung PSSD T7”
May 13 16:44:18 umbrel external storage mounter[509]: Blacklisting USB device IDs against UAS driver…
May 13 16:44:18 umbrel external storage mounter[509]: Rebinding USB drivers…
May 13 16:44:19 umbrel external storage mounter[509]: Checking USB devices are back…
May 13 16:44:19 umbrel external storage mounter[509]: Waiting for USB devices…
May 13 16:44:20 umbrel external storage mounter[509]: Waiting for USB devices…
May 13 16:44:21 umbrel external storage mounter[509]: Waiting for USB devices…
May 13 16:44:22 umbrel external storage mounter[509]: Checking if the device is ext4…
May 13 16:44:22 umbrel external storage mounter[509]: Yes, it is ext4
May 13 16:44:24 umbrel external storage mounter[509]: mount: /mnt/data: can’t read superblock on /dev/sda1.
May 13 16:44:24 umbrel systemd[1]: umbrel-external-storage.service: Main process exited, code=exited, status=32/n/a
May 13 16:44:24 umbrel systemd[1]: umbrel-external-storage.service: Failed with result ‘exit-code’.
May 13 16:44:24 umbrel systemd[1]: Failed to start External Storage Mounter.

External storage SD card update service logs

– Logs begin at Fri 2022-05-13 16:44:08 UTC, end at Sun 2022-05-22 16:58:52 UTC. –
May 13 16:44:24 umbrel systemd[1]: Dependency failed for External Storage SDcard Updater.
May 13 16:44:24 umbrel systemd[1]: umbrel-external-storage-sdcard-update.service: Job umbrel-external-storage-sdcard-update.service/start failed with result ‘dependency’.

Karen logs

tail: cannot open ‘logs/karen.log’ for reading: No such file or directory

Docker containers

NAMES STATUS

Umbrel logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

Bitcoin Core logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

LND logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

electrs logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

Tor logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

==== Result ====

The debug script did not automatically detect any issues with your Umbrel.

1 Like
2

How big is your SSD? From what I see in the logs it appears to be 128GB? It has to be 1TB or bigger.

3

The SSD is 1T. It is brand new and was purchased specifically to run Umbrel.

The micro SD card that Umbrel is flashed on is 128GB

4

What kind of SSD did you get? It doesn’t seem it was properly detected by Umbrel, as it doesn’t show up in the Filesystem information list under the SD card at /dev/root

5

The SSD is a Samsung T7 with 1TB of storage.

6

To my understanding, most of the issues that Umbrel operators experience is from hardware configuration.

Is there anyone that can tell me what hardware combination I can purchase that will not have any issues running Umbrel?

I really just want to get this thing up and running without worrying about it crashing again. Every time I have gotten it to run, I’m hesitant to open any channels because I have had so many issues just getting it to work properly.

Is there better options other than Umbrel for people that simply want to run a node and experiment with lightning?

I hate to ask that in this forum, but I’m kind of just at a loss and want something that works. I’d say that I have been quite patient through this process, but anytime I have an issue I can never find a clear solution.

7

Well, I am in the process of starting over from scratch for the 5th time.

I’m going to let it do it’s thing and check to see if it is still up and running a month from now. If it runs fine after a month with no issues, I will start opening up channels again.

If someone that has been running Umbrel uninterrupted for several months could please let me know what hardware you are using and what I can do to also have a properly working node it will be much appreciated.

8

I just started from scratch and am in the process of syncing the blockchain.

I ran ~/umbrel/scripts/debug --upload so that I could compare the File System Information

Filesystem information before starting over

Filesystem Size Used Avail Use% Mounted on
/dev/root 118G 3.2G 110G 3% /
/dev/root 118G 3.2G 110G 3% /

Filesystem information after reflashing micro SD and reformatting SSD

Filesystem Size Used Avail Use% Mounted on
/dev/root 118G 3.2G 110G 3% /
/dev/sda1 916G 5.8G 864G 1% /home/umbrel/umbrel

Any ideas on what could have been going on?

9

You have ssd files system corruption. Can be power fault or some piece issued. Test carefully every piece of external storage system, change one each time and test performance of system. Do a write speed test. I use Windows to it:
1 - plug SSD system to a desktop or a laptop USB3 port (blue color inside conector)
2- exclude all partitions, then format SSD in any file system you like
3- write a large file file to ssd (greater than 1 GB)
4- copy the written large file from SSD to itself (in same or another directory, no matter)
5- if parts are health, copy speed should be steady and in range 100 to 160MB/second
6- if not pass copy test, change cable or case or SSD and do test again to determine the issued part.
If you have bad energy where Umbrel is installed, then you need a UPS.
Bad energy issues also can burn the mSD card, so change it if pass the ssd tests

10

How long have you had Umbrel running without any issue?

I just don’t understand how it can be working fine for a week then go down.

All of the hardware I purchased is brand new. I don’t want to keep buying new hardware to try and troubleshoot whatever problem I’m having. It would be awesome if there was a detailed list of hardware to use that can run Umbrel without fail rather than trying to find the right combination of hardware that will work.

11

I have a UPS to my modem router and RasPis, so I don’t have problems long time.
But you are wrong to believe in perfect hardware, all hardware will fail, some first use, some few days using and some long time using.

12

I know that all hardware will fail at some point in time. I just would assume that once the SSD “fails”, I would not be able to reformat it be able to get umbrel running on it at all (which I have several times already).

I don’t know much about hardware failure, so I guess that maybe failed hardware can run fine for a few weeks and then stop working. Honestly not sure.

Just going through this form the answer always seems to be to purchase new hardware, even when people have already purchased multiple SSD, mSD, USB cords, ethernet cords, etc…, then start from scratch.

It just seems like all of this would be a million times easier if there was a clear list of hardware that is guaranteed to work.

I’ve already created a new instance and am currently syncing the blockchain, so don’t really want to stop to test the SSD at this point.

If it goes down again and I can’t figure out what went wrong, I’m probably just going to try out MyNode or something.

13

Umbrel is a software, hardware issues are out of control of developers. If a ligtning strikes your electrical lines, bad things may happen, and is all natural thing.

14

The list of parts you are looking for is on the Umbrel homepage with links to the Amazon listing—assuming you are in the US.

Do not deviate on the SSD, SSD enclosure, or power supply.

Good luck.

15

I had very similar issues. The first instance of Umbrel that I set up wouldn’t stay online for more than a few days at a time. I think the issue was my power would flicker from time to time. Not enough to hardly notice but enough to force the Umbrel into a hard shutdown. Enough of these apparently corrupted the SSD. I ended up buying a new SSD and a UPS. I’ve been up for over a month now with no issues. I did have channels open at the time. Be very glad you don’t have to go through that recovery process. I ultimately got it working but it was a challenge.