If you're using a unique password per site (ie something generated by your password manager) I don't see how you face any higher risk with the site operator storing a password vs string some arbitrary token from a third party authenticator.