My understanding is that you’re trying to use your Ledger device with the integrated Specter Umbrel app.
Either you need to use a HWI Bridge with Specter or you can use Specter Desktop directly on your computer, connected to your Umbrel. I’d recommend the latter, as it’s not easy to configure the former. IMO Specter on Umbrel is useful for watch only wallets or multisig using PSBTs.