LDAP sends passwords in cleartext over an encrypted channel, I don't like the idea of someone else on the host I'm connecting to being able to read my password. Naturally you can use key based authentication with it.
I know that some people are required by law or regulation to log all these things, but personally I don't see much benefit of such audit logs. All bets are off when someone has local access to a machine. And there's this whole debate about using MitM-proxies for logging and security and the potential security implications they pose.
All that said, step-ca[0] and vault[1] looks really promising to provide some kind of provider independent way to authenticate in a secure manner. You also have teleport[2] but I haven't looked much into it.
[0] https://smallstep.com/docs/cli/ssh/ [1] https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-cert... [2] https://gravitational.com/teleport/