'no disk space' but there's disk space available

Support and Troubleshooting
1

My umbrel won’t boot to the web server. I’m able to ssh in, and can run the debug script. It ends saying “the debug script did not automatically detect any issues with your Umbrel.”

I see in systemctl that two services, “systemd-hostnamed” and “systemd-timesyncd” have failed. I check them with jouranlctl and can see this:

umbrel@umbrel:~ $ journalctl -u systemd-hostnamed
– Logs begin at Thu 2019-02-14 10:11:59 UTC, end at Thu 2019-02-14 10:51:53 UTC. –
Feb 14 10:12:09 umbrel systemd[1]: systemd-hostnamed.service: Failed to run ‘start’ task: No space left on device
Feb 14 10:12:09 umbrel systemd[1]: systemd-hostnamed.service: Failed with result ‘resources’.
Feb 14 10:12:09 umbrel systemd[1]: Failed to start Hostname Service.

– Logs begin at Thu 2019-02-14 10:11:59 UTC, end at Thu 2019-02-14 10:52:24 UTC. –
Feb 14 10:12:02 umbrel systemd[1]: systemd-timesyncd.service: Failed to run ‘start’ task: No space left on device
Feb 14 10:12:02 umbrel systemd[1]: systemd-timesyncd.service: Failed with result ‘resources’.
Feb 14 10:12:02 umbrel systemd[1]: Failed to start Network Time Synchronization.

df says i still have space, but one partition is full:

umbrel@umbrel:~ $ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 950976 169743 781233 18% /
devtmpfs 444339 428 443911 1% /dev
tmpfs 485763 1 485762 1% /dev/shm
tmpfs 485763 571 485192 1% /run
tmpfs 485763 3 485760 1% /run/lock
tmpfs 485763 15 485748 1% /sys/fs/cgroup
/dev/mmcblk0p1 0 0 0 - /boot
/dev/sda1 61046784 354660 60692124 1% /mnt/data
tmpfs 485763 11 485752 1% /run/user/1000

umbrel@umbrel:~ $ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 14987608 14971224 0 100% /
devtmpfs 1777356 0 1777356 0% /dev
tmpfs 1943052 0 1943052 0% /dev/shm
tmpfs 1943052 97624 1845428 6% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 1943052 0 1943052 0% /sys/fs/cgroup
/dev/mmcblk0p1 258095 30416 227679 12% /boot
/dev/sda1 960322016 604008444 307462148 67% /mnt/data
tmpfs 388608 0 388608 0% /run/user/1000
umbrel@umbrel:~ $

Any help here? Also, is there a different password to use ‘sudo’ as the umbrel user? I can’t seem to make that work.

2

OK, i think i’ve confirmed that the ‘sudo’ password is the password you use to log into the web terminal, not the default ssh password.

3

Annnnnd it loooks like the cultprit is some log message like this:

Feb 11 17:25:17 umbrel dockerd[14157]: time=“2023-02-11T17:25:17.683726866Z” level=error msg=“Failed to log msg “” for logger json-file: error writing lo
g entry: write /var/lib/docker/containers/60d1b98dc5b3d5f342b80136f0823c06bfc78ac88abaab3647cf0e3400686ac1/60d1b98dc5b3d5f342b80136f0823c06bfc78ac88abaab36
47cf0e3400686ac1-json.log: read-only file system”

Something keeps logging this over and over and over and it’s breaking my /dev/root partition where all the logs go.

Is there a way to turn this off?