Mempool app not usable anymore

Hi there,

since a few days my mempool app only shows “connecting” anymore.
I already tried reinstalling the mempool and the Electrs app to no avail.

In the manage app menu Electrs is constantly showing “Starting”.

log: umbrel-1662563527281-debug.log (48.0 KB)

Thank you for your help!

Bump

Hello @J0nsan!

We’re sorry you’re facing this issue - lets collect some more information together so we can begin finding a resolution.

I reviewed the log file and found the following under the mempool log section:
mariadb_1 | 2022-09-07 14:37:00 1110 [Warning] Aborted connection 1110 to db: ‘unconnected’ user: ‘unauthenticated’ host: ‘[BLOCKING INTERNAL IP FOR PRIVACY]’ (This connection closed normally without authentication)

It looks like we are running into a back-end database authentication error. Just to confirm - are any other applications you are running facing this issue?

Thank you,
Max

Lightning Terminal gives the Error “EHOSTUNREACH” when trying to open it. Pi-hole seems to work fine. I dont have more apps installed currently.

Hey J0nsan,

We are actively investiagting this issue - while you wait could you please check out this issue that was posted and see if the resolutions & comments below assist you?

Link: Electrum Server (electrs) installed but not connecting to Bitcoin Core

Thanks,
Max

Hey J0nsan,

After some research and digging we found that this may be a bug with Maria DB (the back-end database service).

Check out this GitHub Issue: https://github.com/photoprism/photoprism/discussions/1966

We recommend running the following

sudo ~/umbrel/scripts/app restart mempool

If this still does not resolve the issue - please feel free to drop us as many logs as possible so we can investigate further by running:

sudo ~/umbrel/scripts/app compose mempool logs | grep -v 'app_proxy_1\|tor_server_1'

Thank you,
Max

Hey Max,

the first command didn’t seem to fix it.

Here is the output of the second command:

Blockquote
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 1 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
api_1 | Bus error (core dumped)
api_1 | wait-for-it.sh: waiting 720 seconds for mariadb:3306
api_1 | wait-for-it.sh: mariadb:3306 is available after 0 seconds
mariadb_1 | 2022-09-13 19:37:11+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.12+maria~focal started.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] mysqld (mysqld 10.5.12-MariaDB-1:10 .5.12+maria~focal) starting as process 1 …
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Uses event mutexes
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Number of pools: 1
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Using ARMv8 crc32 instructi ons
mariadb_1 | 2022-09-13 19:37:12 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Using Linux native AIO
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Initializing buffer pool, t otal size = 134217728, chunk size = 134217728
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: 128 rollback segments are a ctive.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Creating shared tablespace for temporary tables
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Setting file ‘./ibtmp1’ siz e to 12 MB. Physically writing the file full; Please wait …
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: 10.5.12 started; log sequen ce number 45118; transaction id 20
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Plugin ‘FEEDBACK’ is disabled.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Server socket created on IP: ‘::’.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] InnoDB: Buffer pool(s) load complet ed at 220913 19:37:12
mariadb_1 | 2022-09-13 19:37:12 0 [Warning] ‘proxies_priv’ entry ‘@% root@89 7a03be7cc2’ ignored in --skip-name-resolve mode.
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Reading of all Master_info entries succeeded
mariadb_1 | 2022-09-13 19:37:12 0 [Note] Added new Master_info ‘’ to hash ta ble
mariadb_1 | 2022-09-13 19:37:12 0 [Note] mysqld: ready for connections.
mariadb_1 | Version: ‘10.5.12-MariaDB-1:10.5.12+maria~focal’ socket: ‘/run/ mysqld/mysqld.sock’ port: 3306 mariadb.org binary distribution
mariadb_1 | 2022-09-13 19:37:12 3 [Warning] Aborted connection 3 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:13 4 [Warning] Aborted connection 4 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.26’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:14 5 [Warning] Aborted connection 5 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:16 6 [Warning] Aborted connection 6 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:18 7 [Warning] Aborted connection 7 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:20 8 [Warning] Aborted connection 8 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:23 9 [Warning] Aborted connection 9 to db: ‘unc onnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed no rmally without authentication)
mariadb_1 | 2022-09-13 19:37:28 10 [Warning] Aborted connection 10 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)
mariadb_1 | 2022-09-13 19:37:36 11 [Warning] Aborted connection 11 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)
mariadb_1 | 2022-09-13 19:37:50 12 [Warning] Aborted connection 12 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)
mariadb_1 | 2022-09-13 19:38:17 13 [Warning] Aborted connection 13 to db: ‘u nconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)

Blockquote

Hello J0nsan,

Thanks to you - we have discovered a bug within our configuration and will be deploying a fix for all users with this issue.

To update - please run the following:

sudo sed -i 's/mariadb:3306/$APP_MEMPOOL_DB_IP:3306/g' ~/umbrel/app-data/mempool/docker-compose.yml && sudo ~/umbrel/scripts/app restart mempool

Please let us know if this works for you. If it does not please run the following and provide us the results:

free -h

Thank you,
Max

Hi max,

sadly did not work for me.

Here is the output pf “free -h”:

              total        used        free      shared  buff/cache   available
Mem:          3.7Gi       1.8Gi       367Mi        20Mi       1.6Gi       1.8Gi
Swap:         4.0Gi        12Mi       4.0Gi

Hey @J0nsan ! It looks like an odd issue,
I need to ask you a little more details and hopefully, we’ll find a solution together :crossed_fingers:
Can you provide a screenshot of what your mempool app is showing you on the screen?
Could you also provide the electrs and bitcoin logs?

You can get the electrs logs using :
sudo ~/umbrel/scripts/app compose electrs logs --tail=100
You can get the bitcoin logs using :
tail -n 100 ~/umbrel/app-data/bitcoin/data/bitcoin/debug.log

As you have issues with Lightning, Mempool & Electrs the problem could come from Electrs, let’s check this first!

Hello,

the mempool app is always switching between “Connecting” and “Offline”

electrs logs - Pastebin.com

bitcoin logs - Pastebin.com

Thank you for your help!

Alright ! bitcoin logs seem normal, electrs seem to work normally although it fails at some point because it didn’t receive a response in time.
By re-checking your umbrel-debug.log I saw that you have multiple instances of bitcoin and electrs running can you check this topic and tell us here if it worked for you?

That doesn’t seem to work either. Same issue as before and electrs is still constantly “starting”

I’m sorry to hear that :confused:
We’ll need to go more in-depth into the electrs & mempool logs
Could you please provide these logs ?
sudo ~/umbrel/scripts/app compose electrs logs --tail=100 electrs
sudo ~/umbrel/scripts/app compose electrs logs --tail=100 app
sudo ~/umbrel/scripts/app compose mempool logs --tail=100 api
It should give us more details about what’s causing the electrs_app_1 to crash

electrs - Pastebin.com

The second comand only put out “Attaching to electrs_app_1”

api - Pastebin.com