You don't need to involve a private key there. Modern password authentication algorithms never reveal the bare secret (outside of initial registration ofc). For example, PAKE uses Diffie-Hellman coupled with the (salted) password hash to independently derive the same session key on both sides of the connection.

AFAIK the primary technical concerns are insecure storage by the server (bad hash or salt) or keylogging of the client device. But the real issue is the human factor - ie phishing. As long as the shared secret can't be phished it solves the vast majority of real world problems.

Point being, TOTP on a rooted phone handled by a FOSS password manager app whose secret store the end user retains full access to will successfully prevent the vast majority of real world attacks. You probably shouldn't use a FOSS password manager on a rooted device for your self hosted crypto wallet though.