Previously working, now getting Error: System service failed

dhuyvetter · August 17, 2022, 5:15pm

Hi!

[I have edited my original post to add information I forgot when first posting, I’ve marked my edits with cursif]

I had my Umbrel set up and running for a few months but recently moved it from my home, where it was connected to the router via Ethernet cable to a setup where I only have an LTE dingle without Ethernet ports.

Before moving it from the ethernet connected network I stopped being able to connect via Tor or Tailscale as I had before. Since I wasn’t in the same physical location as my Umbrel Pi I didn’t try logging in via umbrel.local

I Googled how to connect to WiFi and took the SD card out, added a wpa_supplicant.conf file in the root and tried to connect. This was unsuccessful, the Umbrel Raspberry Pi didn’t show up on my network and I was unable to connect to umbrel.local.

I then tried another method: connecting the Ethernet cable to another Raspberry Pi running Raspberian and I was able to get umbrel.local up but it was now showing the “http://umbrel.local/” screen. I tried taking the SD card out again and removing the wpa_supplicant.conf file but am still getting the same screen.

I also tried to SSH into my Umbrel Pi from the Raspberian device but the standard moneyprintergobrrr password does not work.

What else can I try to get my Umbrel Raspberry Pi back up?

I now have physical access to the Umbrel Pi and can connect it via ethernet to another Pi running Raspberian, but I do not know the password for the SSH connection. I might have changed the default password when setting up originally but I cannot remember.

dhuyvetter · September 14, 2022, 10:44am

Since I suspected a faulty SD card could be the issue, I bought a new card and first tried cloning the old card to the new one using ‘gdd’ command on my Mac. That resulted in the same error.
After that I tried installing the latest version of Umbrel on the new SD card, but then got the “Error: Failed to start containers” error.
Not sure what else I can do at this point.
I would hate to have to download the whole blockchain again as my limited data plan wouldn’t allow it, apart from the time it would take.
Any suggestions what else I could try?

dhuyvetter · September 14, 2022, 12:40pm

Here is my debug output:

=====================
= Umbrel debug info =
=====================

Umbrel version
--------------
0.5.1

Flashed OS version
-----------------
v0.4.17

Raspberry Pi Model
------------------
Revision	: c03114
Serial		: 10000000ee8d698e
Model		: Raspberry Pi 4 Model B Rev 1.4

Firmware
--------
Dec  1 2021 15:01:54 
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

Temperature
-----------
temp=36.5'C

Throttling
----------
throttled=0x0

Memory usage
------------
              total        used        free      shared  buff/cache   available
Mem:           3.8G        110M        3.1G        174M        545M        3.4G
Swap:          4.1G          0B        4.1G

And at the very end:

================
==== Result ====
================
The debug script did not automatically detect any issues with your Umbrel.

I realise there is a whole middle part missing from the logs, but between that first screen and the end bit I am getting a bunch of errors: https://pastebin.com/LBnWzunn (this is only as much as I could see on terminal screen)

One line I saw repeated in there a lot is this:

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

The way I am connecting to my Umbrell box is via SSH from another Raspberry Pi (running Raspberian) via network cable (connecting network ports of both Raspberry Pies.
My Raspberian box is connected to the internet via WiFi but I assume my Umbrel Pi isn’t and I am unsure how to connect it to the wider internet. I assume that is where the above error is coming from.

Not sure how to further debug.

Or if I would need to reinstall from scratch, how to clear my hard drive. Although that would be the very last resort, not just because I already had quite a few Lightning channels open, but also because re-downloading the whole blockchain would be a lengthy and costly affair.

MaxUmbrel · September 14, 2022, 1:50pm

Hi Dhuyvetter,

Your SSH password matches the exact password of the Umbrel system password. So whatever you use to log into Umbrel on boot is what you’ll use for SSH.

Thanks,
Max

dhuyvetter · September 15, 2022, 10:41am

Hi Max! Thanks! Yeah I figured that out, so I can SSH into my Umbrel Raspberry Pi now, just need to figure out why it isn’t working.

dhuyvetter · September 15, 2022, 11:14am

I had a look at /umbrel/logs/status-monitor.log and the most recent errors are all write error: No space left on device
So I think I figured out what the issue is.

What would be safe to delete to free up some space on my SD card?

And how do I go about migrating to a bigger SD card? I have one, but how do I safely copy all the content to the new card?

dhuyvetter · September 15, 2022, 2:52pm

I switched to a larger SD card and now running ~/umbrel/scripts/debug --upload gives less errors and more information:

=====================
= Umbrel debug info =
=====================

Umbrel version
--------------
0.5.1

Flashed OS version
-----------------
v0.4.17

Raspberry Pi Model
------------------
Revision	: c03114
Serial		: 10000000ee8d698e
Model		: Raspberry Pi 4 Model B Rev 1.4

Firmware
--------
Dec  1 2021 15:01:54 
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

Temperature
-----------
temp=35.5'C

Throttling
----------
throttled=0x0

Memory usage
------------
              total        used        free      shared  buff/cache   available
Mem:           3.8G        103M        3.5G        8.0M        206M        3.6G
Swap:            0B          0B          0B

total: 2.7%
system: 2.7%
tor: 0%
lnd: 0%
electrs: 0%
bitcoin: 0%

Memory monitor logs
-------------------
tail: cannot open '/home/umbrel/umbrel/scripts/../logs/memory-monitor.log' for reading: No such file or directory

Filesystem information
----------------------
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G   59G     0 100% /
/dev/root        59G   59G     0 100% /

Startup service logs
--------------------
-- Logs begin at Thu 2019-02-14 10:11:58 UTC, end at Thu 2019-02-14 10:23:50 UTC. --
Feb 14 10:12:03 umbrel systemd[1]: Dependency failed for Umbrel Startup Service.
Feb 14 10:12:03 umbrel systemd[1]: umbrel-startup.service: Job umbrel-startup.service/start failed with result 'dependency'.

External storage service logs
-----------------------------
-- Logs begin at Thu 2019-02-14 10:11:58 UTC, end at Thu 2019-02-14 10:23:50 UTC. --
Feb 14 10:12:03 umbrel systemd[1]: Starting External Storage Mounter...
Feb 14 10:12:03 umbrel external storage mounter[472]: /status-server/set-status: line 31: echo: write error: No space left on device
Feb 14 10:12:03 umbrel systemd[1]: umbrel-external-storage.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 10:12:03 umbrel systemd[1]: umbrel-external-storage.service: Failed with result 'exit-code'.
Feb 14 10:12:03 umbrel systemd[1]: Failed to start External Storage Mounter.

External storage SD card update service logs
--------------------------------------------
-- Logs begin at Thu 2019-02-14 10:11:58 UTC, end at Thu 2019-02-14 10:23:50 UTC. --
Feb 14 10:12:03 umbrel systemd[1]: Dependency failed for External Storage SDcard Updater.
Feb 14 10:12:03 umbrel systemd[1]: umbrel-external-storage-sdcard-update.service: Job umbrel-external-storage-sdcard-update.service/start failed with result 'dependency'.

Karen logs
----------

tail: cannot open 'logs/karen.log' for reading: No such file or directory

Docker containers
-----------------
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Umbrel logs
-----------

ERROR: The Compose file './docker-compose.yml' is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object

Tor logs
--------

ERROR: The Compose file './docker-compose.yml' is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
================
==== Result ====
================
The debug script did not automatically detect any issues with your Umbrel.

lukechilds · September 16, 2022, 9:17am

Hey @dhuyvetter,

Sorry to hear about your issue, good detective skills figuring out the filesystem out of space issue!

Would be great if we could get to the bottom of what caused the issue so we can prevent it happening to other users. Did you flash a fresh larger SD card or image your old SD card over to a new one?

Just wondering if we can run some commands in the original environment that ran out of space to figure out where the large files are.

You can run:

sudo du -h -a / 2>/dev/null | sort -h -r | head -n 100

To list 100 largest files on the filesystem (this might take a while to complete) if you still have access to the original environment.

CryptoMistro · September 25, 2022, 8:15pm

I’m having this same problem with the Error: System service failed. Previously had the No Space left on device in debug logs. My Umbrel was been up and running on a RPi for some time and then something happened while I was out of town on vacation which led to my not being able to bootup. I’m able to login to the device via SSH. I’ve combed through the community Support and Troubleshooting posts, ran all kinds of commands that have led to no success. Only now I do not see the No Space Left on Device error in the debug logs from reboot Re-flashed the existing mSD card (not a new one). Not sure what is filling up the hard drive and causing the problem. Any help or suggestions on what to do moving forward would be greatly appreciated. I may have to reformat the SSD and start fresh and recover the node? I don’t know what else to try.

Below are the results from the debug report and when I ran the command @lukechilds recommended to @dhuyvetter above:

sudo du -h -a / 2>/dev/null | sort -h -r | head -n 100

1.8T /
916G /mnt/data
916G /mnt
517G /mnt/data/umbrel
517G /home/umbrel/umbrel
517G /home/umbrel
517G /home
482G /mnt/data/umbrel/bitcoin
482G /home/umbrel/umbrel/bitcoin
434G /mnt/data/umbrel/bitcoin/blocks
434G /home/umbrel/umbrel/bitcoin/blocks
396G /var/lib
396G /var
396G /mnt/data/docker
395G /var/lib/docker
376G /var/lib/docker/containers
376G /mnt/data/docker/containers
375G /var/lib/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6-json.log
375G /var/lib/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6
375G /mnt/data/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6-json.log
375G /mnt/data/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6
43G /mnt/data/umbrel/bitcoin/indexes
43G /home/umbrel/umbrel/bitcoin/indexes
36G /mnt/data/umbrel/bitcoin/indexes/txindex
36G /home/umbrel/umbrel/bitcoin/indexes/txindex
34G /mnt/data/umbrel/electrs/db/bitcoin
34G /mnt/data/umbrel/electrs/db
34G /mnt/data/umbrel/electrs
34G /home/umbrel/umbrel/electrs/db/bitcoin
34G /home/umbrel/umbrel/electrs/db
34G /home/umbrel/umbrel/electrs
20G /var/lib/docker/overlay2
20G /mnt/data/docker/overlay2
7.6G /mnt/data/umbrel/bitcoin/indexes/blockfilter/basic
7.6G /mnt/data/umbrel/bitcoin/indexes/blockfilter
7.6G /home/umbrel/umbrel/bitcoin/indexes/blockfilter/basic
7.6G /home/umbrel/umbrel/bitcoin/indexes/blockfilter
4.7G /mnt/data/umbrel/bitcoin/chainstate
4.7G /home/umbrel/umbrel/bitcoin/chainstate
4.1G /swap/swapfile
4.1G /swap
4.1G /mnt/data/swap/swapfile
4.1G /mnt/data/swap
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay/core
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay/core
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a
1.6G /mnt/data/umbrel/app-data
1.6G /home/umbrel/umbrel/app-data
1.3G /usr
984M /var/lib/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c-json.log
984M /var/lib/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c
984M /mnt/data/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c-json.log
984M /mnt/data/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030
896M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9/site-packages
896M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9
896M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib
896M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9/site-packages
896M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9
896M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib
866M /mnt/data/umbrel/app-data/nextcloud/data
866M /mnt/data/umbrel/app-data/nextcloud
866M /home/umbrel/umbrel/app-data/nextcloud/data
866M /home/umbrel/umbrel/app-data/nextcloud
718M /var/log
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL/node_modules
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL/node_modules
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e
664M /mnt/data/umbrel/app-data/nextcloud/data/nextcloud
664M /home/umbrel/umbrel/app-data/nextcloud/data/nextcloud
625M /mnt/data/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff
625M /mnt/data/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4
609M /mnt/data/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff/usr
601M /var/lib/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff
601M /var/lib/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4
584M /var/lib/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff/usr
583M /mnt/data/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff/build
583M /mnt/data/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff
583M /mnt/data/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d
576M /var/lib/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff/build
576M /var/lib/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff
576M /var/lib/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d
529M /var/log/daemon.log.1

~/umbrel $ ./scripts/debug

= Umbrel debug info =

Umbrel version

0.4.18

Flashed OS version

v0.5.0

Raspberry Pi Model

Revision : d03114
Serial : 10000000e1ca6660
Model : Raspberry Pi 4 Model B Rev 1.4

Firmware

Dec 1 2021 15:01:54
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

Temperature

temp=42.3’C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 7.8G 114M 7.1G 170M 611M 7.4G
Swap: 4.1G 0B 4.1G

total: 1.5%
system: 1.5%
vaultwarden: 0%
urbit: 0%
uptime-kuma: 0%
tor: 0%
thunderhub: 0%
syncthing: 0%
synapse: 0%
sphinx-relay: 0%
simple-torrent: 0%
ride-the-lightning: 0%
pi-hole: 0%
photoprism: 0%
node-red: 0%
nextcloud: 0%
mempool: 0%
lnmarkets: 0%
lndg: 0%
lnd: 0%
lnbits: 0%
lightning-terminal: 0%
krystal-bull: 0%
home-assistant: 0%
element: 0%
electrs: 0%
btc-rpc-explorer: 0%
btcpay-server: 0%
bluewallet: 0%
bitfeed: 0%
bitcoin: 0%

Memory monitor logs

2022-07-04 19:45:20 Memory monitor running!
2022-07-04 19:45:32 Memory monitor running!
2022-07-04 19:45:49 Memory monitor running!
2022-07-04 19:46:01 Memory monitor running!
2022-07-04 19:46:13 Memory monitor running!
2022-07-04 19:46:25 Memory monitor running!
2022-07-04 19:46:38 Memory monitor running!
2022-07-04 19:46:55 Memory monitor running!
2022-07-04 19:47:07 Memory monitor running!
2022-07-04 19:47:19 Memory mon
Filesystem information

Filesystem Size Used Avail Use% Mounted on
/dev/root 59G 3.9G 53G 7% /
/dev/sda1 916G 916G 0 100% /home/umbrel/umbrel

Startup service logs

Sep 25 20:08:39 umbrel umbrel startup[32365]: The NGINX_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The NGINX_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_P2P_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The LND_REST_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The LND_GRPC_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The LND_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The DASHBOARD_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The MIDDLEWARE_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_RPC_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_RPC_USER variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_RPC_PASS variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_NETWORK variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The MANAGER_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The NEUTRINO_SWITCHER_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The ELECTRUM_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The ELECTRUM_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The NETWORK_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The Compose file ‘./docker-compose.yml’ is invalid because:
Sep 25 20:08:39 umbrel umbrel startup[32365]: networks.default.ipam.config.subnet is invalid: should use the CIDR format
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.tor_proxy.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.bitcoin.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.electrs.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.lnd.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.nginx.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
Sep 25 20:08:39 umbrel umbrel startup[32365]: Failed to start containers
Sep 25 20:08:39 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Sep 25 20:08:39 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Sep 25 20:08:39 umbrel systemd[1]: Failed to start Umbrel Startup Service.

External storage service logs

– Logs begin at Sun 2022-09-25 08:17:57 UTC, end at Sun 2022-09-25 20:08:39 UTC. –
Sep 25 08:18:02 umbrel systemd[1]: Starting External Storage Mounter…
Sep 25 08:18:02 umbrel external storage mounter[485]: Running external storage mount script…
Sep 25 08:18:02 umbrel external storage mounter[485]: Found device “External USB3.0”
Sep 25 08:18:02 umbrel external storage mounter[485]: Blacklisting USB device IDs against UAS driver…
Sep 25 08:18:02 umbrel external storage mounter[485]: Rebinding USB drivers…
Sep 25 08:18:02 umbrel external storage mounter[485]: Checking USB devices are back…
Sep 25 08:18:02 umbrel external storage mounter[485]: Waiting for USB devices…
Sep 25 08:18:03 umbrel external storage mounter[485]: Waiting for USB devices…
Sep 25 08:18:04 umbrel external storage mounter[485]: Waiting for USB devices…
Sep 25 08:18:05 umbrel external storage mounter[485]: Checking if the device is ext4…
Sep 25 08:18:05 umbrel external storage mounter[485]: Yes, it is ext4
Sep 25 08:18:05 umbrel external storage mounter[485]: Checking if device contains an Umbrel install…
Sep 25 08:18:05 umbrel external storage mounter[485]: Yes, it contains an Umbrel install
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting external storage over local Umbrel installation…
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting external storage over local Docker data dir…
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting external storage to /swap
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting SD card root at /sd-card…
Sep 25 08:18:05 umbrel external storage mounter[485]: Checking Umbrel root is now on external storage…
Sep 25 08:18:06 umbrel external storage mounter[485]: Checking /var/lib/docker is now on external storage…
Sep 25 08:18:06 umbrel external storage mounter[485]: Checking /swap is now on external storage…
Sep 25 08:18:06 umbrel external storage mounter[485]: Setting up swapfile
Sep 25 08:18:07 umbrel external storage mounter[485]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Sep 25 08:18:07 umbrel external storage mounter[485]: no label, UUID=79029681-e317-4a7c-b90d-8fd0982ff0ad
Sep 25 08:18:07 umbrel external storage mounter[485]: Checking SD Card root is bind mounted at /sd-root…
Sep 25 08:18:07 umbrel external storage mounter[485]: Starting external drive mount monitor…
Sep 25 08:18:07 umbrel external storage mounter[485]: Mount script completed successfully!
Sep 25 08:18:07 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs

– Logs begin at Sun 2022-09-25 08:17:57 UTC, end at Sun 2022-09-25 20:08:39 UTC. –
Sep 25 08:18:14 umbrel systemd[1]: Starting External Storage SDcard Updater…
Sep 25 08:18:14 umbrel external storage updater[811]: Checking if SD card Umbrel is newer than external storage…
Sep 25 08:18:14 umbrel external storage updater[811]: Yes, SD version is newer.
Sep 25 08:18:14 umbrel external storage updater[811]: Checking if the external storage version “0.4.18” satisfies update requirement “>=0.4.18”…
Sep 25 08:18:14 umbrel external storage updater[811]: Yes, it does, attempting an automatic update…
Sep 25 08:18:14 umbrel external storage updater[811]: =======================================
Sep 25 08:18:14 umbrel external storage updater[811]: =============== UPDATE ================
Sep 25 08:18:14 umbrel external storage updater[811]: =======================================
Sep 25 08:18:14 umbrel external storage updater[811]: ========== Stage: Download ============
Sep 25 08:18:14 umbrel external storage updater[811]: =======================================
Sep 25 08:18:14 umbrel external storage updater[811]: Creating lock
Sep 25 08:18:14 umbrel external storage updater[811]: Cleaning up any previous mess
Sep 25 08:18:14 umbrel external storage updater[811]: Copying Umbrel 0.5.0 from /sd-root/home/umbrel/umbrel
Sep 25 08:18:14 umbrel external storage updater[811]: mkdir: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.5.0’: No space left on device
Sep 25 08:18:14 umbrel external storage updater[811]: cp: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.5.0’: No space left on device
Sep 25 08:18:14 umbrel external storage updater[811]: /home/umbrel/umbrel/scripts/update/update: line 113: cd: /home/umbrel/umbrel/.umbrel-0.5.0: No such file or directory
Sep 25 08:18:14 umbrel external storage updater[811]: Running update install scripts of the new release
Sep 25 08:18:14 umbrel external storage updater[811]: /home/umbrel/umbrel/scripts/update/update: line 118: cd: scripts/update: No such file or directory
Sep 25 08:18:14 umbrel external storage updater[811]: ls: cannot access ‘*-run.sh’: No such file or directory
Sep 25 08:18:14 umbrel external storage updater[811]: Deleting cloned repository
Sep 25 08:18:14 umbrel external storage updater[811]: Removing lock
Sep 25 08:18:22 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs

karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen i
Docker containers

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Umbrel logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

Bitcoin Core logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

LND logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

electrs logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements

Tor logs

ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements