Mempool app not usable anymore

Hey Max,

the first command didn’t seem to fix it.

Here is the output of the second command:

Blockquote
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 1 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
mariadb_1 | 2022-09-13 19:37:11+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.12+maria~focal started.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] mysqld (mysqld 10.5.12-MariaDB-1:10 .5.12+maria~focal) starting as process 1 …
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Uses event mutexes
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Number of pools: 1
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Using ARMv8 crc32 instructi ons
mariadb_1 | 2022-09-13 19:37:12 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Using Linux native AIO
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Initializing buffer pool, t otal size = 134217728, chunk size = 134217728
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: 128 rollback segments are a ctive.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Setting file ‘./ibtmp1’ siz e to 12 MB. Physically writing the file full; Please wait …
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: 10.5.12 started; log sequen ce number 45118; transaction id 20
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Plugin ‘FEEDBACK’ is disabled.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Server socket created on IP: ‘::’.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Buffer pool(s) load complet ed at 220913 19:37:12
mariadb_1 | 2022-09-13 19:37:12 0 [Warning] ‘proxies_priv’ entry ‘@% root@89 7a03be7cc2’ ignored in --skip-name-resolve mode.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Reading of all Master_info entries succeeded
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Added new Master_info ‘’ to hash ta ble
mariadb_1 | 2022-09-13 19:37:12 0 [Note] mysqld: ready for connections.
mariadb_1 | Version: ‘10.5.12-MariaDB-1:10.5.12+maria~focal’ socket: ‘/run/ mysqld/mysqld.sock’ port: 3306 mariadb.org binary distribution
mariadb_1 | 2022-09-13 19:37:12 3 [Warning] Aborted connection 3 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:13 4 [Warning] Aborted connection 4 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.26’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:14 5 [Warning] Aborted connection 5 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:16 6 [Warning] Aborted connection 6 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:18 7 [Warning] Aborted connection 7 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:20 8 [Warning] Aborted connection 8 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:23 9 [Warning] Aborted connection 9 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:28 10 [Warning] Aborted connection 10 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)
mariadb_1 | 2022-09-13 19:37:36 11 [Warning] Aborted connection 11 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)
mariadb_1 | 2022-09-13 19:37:50 12 [Warning] Aborted connection 12 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)
mariadb_1 | 2022-09-13 19:38:17 13 [Warning] Aborted connection 13 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)

Blockquote

Hello J0nsan,

Thanks to you - we have discovered a bug within our configuration and will be deploying a fix for all users with this issue.

To update - please run the following:

sudo sed -i 's/mariadb:3306/$APP_MEMPOOL_DB_IP:3306/g' ~/umbrel/app-data/mempool/docker-compose.yml && sudo ~/umbrel/scripts/app restart mempool

Please let us know if this works for you. If it does not please run the following and provide us the results:

free -h

Thank you,
Max

Hi max,

sadly did not work for me.

Here is the output pf “free -h”:

              total        used        free      shared  buff/cache   available
Mem:          3.7Gi       1.8Gi       367Mi        20Mi       1.6Gi       1.8Gi
Swap:         4.0Gi        12Mi       4.0Gi

Hey @J0nsan ! It looks like an odd issue,
I need to ask you a little more details and hopefully, we’ll find a solution together :crossed_fingers:
Can you provide a screenshot of what your mempool app is showing you on the screen?
Could you also provide the electrs and bitcoin logs?

You can get the electrs logs using :
sudo ~/umbrel/scripts/app compose electrs logs --tail=100
You can get the bitcoin logs using :
tail -n 100 ~/umbrel/app-data/bitcoin/data/bitcoin/debug.log

As you have issues with Lightning, Mempool & Electrs the problem could come from Electrs, let’s check this first!

Hello,

the mempool app is always switching between “Connecting” and “Offline”

electrs logs - Pastebin.com

bitcoin logs - Pastebin.com

Thank you for your help!

Alright ! bitcoin logs seem normal, electrs seem to work normally although it fails at some point because it didn’t receive a response in time.
By re-checking your umbrel-debug.log I saw that you have multiple instances of bitcoin and electrs running can you check this topic and tell us here if it worked for you?

That doesn’t seem to work either. Same issue as before and electrs is still constantly “starting”

I’m sorry to hear that :confused:
We’ll need to go more in-depth into the electrs & mempool logs
Could you please provide these logs ?
sudo ~/umbrel/scripts/app compose electrs logs --tail=100 electrs
sudo ~/umbrel/scripts/app compose electrs logs --tail=100 app
sudo ~/umbrel/scripts/app compose mempool logs --tail=100 api
It should give us more details about what’s causing the electrs_app_1 to crash

electrs - Pastebin.com

The second comand only put out “Attaching to electrs_app_1”

api - Pastebin.com

bump

Hey @J0nsan! Sorry for the delay,
That’s weird; It looks like there’s something causing connection issues between apps. Have you modified your bitcoin.conf file?

Hello, I am having the exact same problem. It happened just after latest version upgrade. It seems from logs like it is not connecting to db backend; although all modules comes up with no problem when you restart node and/or module.

None of the proposed solutions in this thread worked for me. On the web interface there is a sign that reads “indexing in progress” forever, but seems unrelated to the issue. It has been like this for the past two weeks at least, since upgrade.

Hey @jhava! Sorry to hear that :confused:
Maybe bitcoin or electrs is still indexing data.
But let’s check the logs to know which one is indexing/causing the issue!
You can get the electrs logs using:
sudo ~/umbrel/scripts/app compose electrs logs --tail=100
You can get the bitcoin logs using:
tail -n 100 ~/umbrel/app-data/bitcoin/data/bitcoin/debug.log

Hello,

Thank you for your response. Here are the logs.

electrs logs: https://pastebin.com/Ru9RWyix

bitcoin logs: https://pastebin.com/3z4sES0s

It looks like bitcoin is fully synced and electrs too, but at some point, you have this error:

[2022-09-28T14:05:10.612Z WARN electrs::mempool] mempool sync failed: failed to get mempool txids

Is the Electrs app UI showing synchronized? Is it only the mempool app that is having issues?
In that case could you provide also the mempool app logs ?
sudo ~/umbrel/scripts/app compose mempool logs --tail=100

Electrs app UI shows fully synchronized. So far only mempool is giving the issue.

Here is the mempool log: https://pastebin.com/NYCwgJ8F

Off topic comment: The mempool website includes now insightful information on lightning network. It would be interesting to have it in umbrel’s own mempool app.

Hi,

I never modified anything beyond using the UI since I last set It up from scratch.

@J0nsan have you suffered power outage at any point?

Based on this error from electrs:

electrs_1     | [2022-09-20T15:01:11.007Z INFO  electrs::daemon] waiting for RPC warmup: Starting network threads…
electrs_1     | [2022-09-20T15:01:19.061Z INFO  electrs::index] indexing 2000 blocks: [313001..315000]
electrs_1     | [2022-09-20T15:01:19.580Z INFO  electrs::db] closing DB at /data/db/bitcoin
electrs_1     | Error: electrs failed
electrs_1     |
electrs_1     | Caused by:
electrs_1     |     0: sync failed
electrs_1     |     1: failed to get block 00000000000000001fdc9bcb5505b2200d1ef6d33607222f2c5003aa2a931dde
electrs_1     |     2: receiving on an empty and disconnected channel

It looks like maybe there is an issue with Bitcoin Core. electrs is requesting a block from Bitcoin Core but Bitcoin Core isn’t able to send it. This occasionally can happen if you suffer from instant power loss (power cut or unplugging the device directly from the wall) which leads to filesystem corruption in Bitcoin Core.

We can try clearing the blockchain data from Bitcoin Core and resyncing. You won’t lose any other data but your node will take a few days to resync. You might not be able to use your Lightning node during this time.

If you want to try this you can SSH into your Umbrel and run:

sudo ~/umbrel/scripts/app stop electrs && sudo ~/umbrel/scripts/app stop bitcoin && sudo rm -rf ~/umbrel/app-data/bitcoin/data/bitcoin/* ~/umbrel/app-data/electrs/data/electrs/* && sudo ~/umbrel/scripts/app start bitcoin && sudo ~/umbrel/scripts/app start electrs

Not that I know of.

Well I guess IBD for the 4th time now it is :smiley:
(Just to be clear, I am very thankfull for the support you are giving and for umbrel itself, but I seem to be very unlucky when it comes to problems with the node that require an IBD)

Of course, I can imagine that’s frustrating!

Why did you have to do an IBD the last three times? Could you link me to your previous issues? It’s not normal that you should have this many issues. There could potentially be an issue with your hardware.

Could you list me the full specs of all the hardware you’re using. Also can I just confirm you’re using the official Raspberry Pi power supply if you’re using a Raspberry Pi.