Bitcoind restarts once per minute

My bitcoin restarts once per minute, but I can’t see any errors in the log file. The log file is the following lines repeated over and over again.

2021-11-22T22:03:31Z Bitcoin Core version v22.0.0 (release build)
2021-11-22T22:03:31Z InitParameterInteraction: parameter interaction: -proxy set -> setting -upnp=0
2021-11-22T22:03:31Z InitParameterInteraction: parameter interaction: -proxy set -> setting -natpmp=0
2021-11-22T22:03:31Z InitParameterInteraction: parameter interaction: -proxy set -> setting -discover=0
2021-11-22T22:03:31Z Assuming ancestors of block 00000000000000000008a89e854d57e5667df88f1cdef6fde2fbca1de5b639ad have valid signatures.
2021-11-22T22:03:31Z Setting nMinimumChainWork=00000000000000000000000000000000000000001fa4663bbbe19f82de910280
2021-11-22T22:03:31Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2021-11-22T22:03:31Z Using RdRand as an additional entropy source
2021-11-22T22:03:31Z Default data directory /data/.bitcoin
2021-11-22T22:03:31Z Using data directory /data/.bitcoin
2021-11-22T22:03:31Z Config file: /data/.bitcoin/bitcoin.conf
2021-11-22T22:03:31Z Config file arg: bind="10.21.21.8"
2021-11-22T22:03:31Z Config file arg: blockfilterindex="1"
2021-11-22T22:03:31Z Config file arg: dbcache="200"
2021-11-22T22:03:31Z Config file arg: deprecatedrpc="addresses"
2021-11-22T22:03:31Z Config file arg: listen="1"
2021-11-22T22:03:31Z Config file arg: maxmempool="300"
2021-11-22T22:03:31Z Config file arg: peerblockfilters="1"
2021-11-22T22:03:31Z Config file arg: peerbloomfilters="1"
2021-11-22T22:03:31Z Config file arg: port="8333"
2021-11-22T22:03:31Z Config file arg: proxy="10.21.21.11:9050"
2021-11-22T22:03:31Z Config file arg: rpcallowip="10.21.21.1/16"
2021-11-22T22:03:31Z Config file arg: rpcallowip="127.0.0.1"
2021-11-22T22:03:31Z Config file arg: rpcauth=****
2021-11-22T22:03:31Z Config file arg: rpcbind=****
2021-11-22T22:03:31Z Config file arg: rpcbind=****
2021-11-22T22:03:31Z Config file arg: rpcport="8332"
2021-11-22T22:03:31Z Config file arg: rpcworkqueue="64"
2021-11-22T22:03:31Z Config file arg: txindex="1"
2021-11-22T22:03:31Z Config file arg: zmqpubhashblock="tcp://0.0.0.0:28334"
2021-11-22T22:03:31Z Config file arg: zmqpubrawblock="tcp://0.0.0.0:28332"
2021-11-22T22:03:31Z Config file arg: zmqpubrawtx="tcp://0.0.0.0:28333"
2021-11-22T22:03:31Z Command-line arg: zmqpubrawblock="tcp://0.0.0.0:28332"
2021-11-22T22:03:31Z Command-line arg: zmqpubrawtx="tcp://0.0.0.0:28333"
2021-11-22T22:03:31Z Using at most 125 automatic connections (1048576 file descriptors available)
2021-11-22T22:03:31Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2021-11-22T22:03:32Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2021-11-22T22:03:32Z Script verification uses 7 additional threads
2021-11-22T22:03:32Z scheduler thread start
2021-11-22T22:03:32Z HTTP: creating work queue of depth 64
2021-11-22T22:03:32Z Using random cookie authentication.
2021-11-22T22:03:32Z Generated RPC authentication cookie /data/.bitcoin/.cookie
2021-11-22T22:03:32Z Using rpcauth authentication.
2021-11-22T22:03:32Z HTTP: starting 4 worker threads
2021-11-22T22:03:32Z Using wallet directory /data/.bitcoin
2021-11-22T22:03:32Z init message: Verifying wallet(s)…
2021-11-22T22:03:32Z init message: Loading banlist…
2021-11-22T22:03:32Z SetNetworkActive: true
2021-11-22T22:03:32Z Using /16 prefix for IP bucketing
2021-11-22T22:03:32Z Cache configuration:
2021-11-22T22:03:32Z * Using 2.0 MiB for block index database
2021-11-22T22:03:32Z * Using 24.8 MiB for transaction index database
2021-11-22T22:03:32Z * Using 21.7 MiB for basic block filter index database
2021-11-22T22:03:32Z * Using 8.0 MiB for chain state database
2021-11-22T22:03:32Z * Using 143.6 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2021-11-22T22:03:32Z init message: Loading block index…
2021-11-22T22:03:32Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2021-11-22T22:03:32Z Opening LevelDB in /data/.bitcoin/blocks/index
2021-11-22T22:03:32Z Opened LevelDB successfully
2021-11-22T22:03:32Z Using obfuscation key for /data/.bitcoin/blocks/index: 0000000000000000
2021-11-22T22:03:35Z LoadBlockIndexDB: last block file = 2813
2021-11-22T22:03:35Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=32, size=39389710, heights=710563...710594, time=2021-11-20...2021-11-20)
2021-11-22T22:03:35Z Checking all blk files are present...
2021-11-22T22:03:35Z Opening LevelDB in /data/.bitcoin/chainstate
2021-11-22T22:03:35Z Opened LevelDB successfully
2021-11-22T22:03:35Z Using obfuscation key for /data/.bitcoin/chainstate: 4930b71f0c03eaca
2021-11-22T22:03:35Z Loaded best chain: hashBestChain=0000000000000000000713d07eb7ed5dfa89d78f8248a9ac62f1a635261b8582 height=710466 date=2021-11-19T20:13:21Z progress=0.999064
2021-11-22T22:03:35Z init message: Verifying blocks…
2021-11-22T22:03:35Z Verifying last 6 blocks at level 3
2021-11-22T22:03:35Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2021-11-22T22:03:37Z No coin database inconsistencies in last 6 blocks (12997 transactions)
2021-11-22T22:03:37Z  block index            4998ms
2021-11-22T22:03:37Z Opening LevelDB in /data/.bitcoin/indexes/txindex
2021-11-22T22:03:37Z Opened LevelDB successfully
2021-11-22T22:03:37Z Using obfuscation key for /data/.bitcoin/indexes/txindex: 0000000000000000
2021-11-22T22:03:37Z Opening LevelDB in /data/.bitcoin/indexes/blockfilter/basic/db
2021-11-22T22:03:37Z txindex thread start
2021-11-22T22:03:37Z txindex is enabled at height 710466
2021-11-22T22:03:37Z txindex thread exit
2021-11-22T22:03:37Z Opened LevelDB successfully
2021-11-22T22:03:37Z Using obfuscation key for /data/.bitcoin/indexes/blockfilter/basic/db: 0000000000000000
2021-11-22T22:03:37Z basic block filter index thread start
2021-11-22T22:03:37Z basic block filter index is enabled at height 710466
2021-11-22T22:03:37Z basic block filter index thread exit
2021-11-22T22:03:37Z block tree size = 710596
2021-11-22T22:03:37Z nBestHeight = 710466
2021-11-22T22:03:37Z loadblk thread start
2021-11-22T22:03:37Z torcontrol thread start
2021-11-22T22:03:37Z Bound to 10.21.21.8:8333
2021-11-22T22:03:37Z Bound to 127.0.0.1:8334
2021-11-22T22:03:37Z init message: Loading P2P addresses…

I’ve set docker-compose to debug level and it just keeps saying

bitcoin exited with code 135
  • Device = Ubuntu Desktop 21.10
  • Umbrel connected to the Internet via ethernet
  • Drive = SSD
  • Docker version 20.10.11, build dea9396

This started happening with umbrel 0.4.8 but I have downgraded to 0.4.7 and it still restarts with exit code 135.

I’m currently re-indexing to see if that makes a difference.

Every time the bitcoin container starts up, it runs to a point and exits with the following error

bitcoin              | 2021-11-25T15:37:28Z Loaded 33 blocks from external file in 733ms
bitcoin              | 2021-11-25T15:37:28Z Reindexing finished
bitcoin              | 2021-11-25T15:37:29Z
bitcoin              |
bitcoin              | ************************
bitcoin              | EXCEPTION: NSt8ios_base7failureB5cxx11E
bitcoin              | CAutoFile::write: write failed: iostream error
bitcoin              | bitcoin in loadblk
bitcoin              |
bitcoin              |
bitcoin              |
bitcoin              | ************************
bitcoin              | EXCEPTION: NSt8ios_base7failureB5cxx11E
bitcoin              | CAutoFile::write: write failed: iostream error
bitcoin              | bitcoin in loadblk
bitcoin              |
bitcoin              | terminate called after throwing an instance of 'std::ios_base::failure[abi:cxx11]'
bitcoin              |   what():  CAutoFile::write: write failed: iostream error

I have tried everything I can think of now.

I have downgraded umbrel to 0.4.7. Same problem. Upgraded back to 0.4.8. Same problem. Downgraded docker to the previous version. Same problem.

Does anybody know how to reset the bitcoin data?

Can I just delete the contents of the bitcoin directory and restart?

I fixed the problem. I think it was bad sectors on the disk, although I did also re-route the SSD sata cables. I booted ubuntu from a USB stick and ran fsck on the root directory. It reported 46 bad sectors.

I had to re-download the blockchain by stopping umbrel and:

rm -rf ./bitcoin/blocks/*
rm -rf ./bitcoin/chainstate/*
rm -rf ./bitcoin/indexes/blockfilter/basic/*
rm -rf ./bitcoin/txindex/*

and restarting umbrel. It re-downloaded the blockchain and everything seem ok. My sats are safe.

So what have I learned?

  1. A brand new Samsung SSD 870 EVO, 2 TB is not guaranteed to be flawless
  2. A disk write error can be persistent and cause the bitcoind to constantly re-start
  3. It is probably best to run umbrel on a Raspery Pi because you will be using the same hardware as everyone else and you wont be so alone with your problems