There is a security issue with using AD/OpenLDAP. That being passwords. An example is you can debug/trace the SSH process and see(log) passwords in clear text.
Now of course this is both a trust and a escalated privilege issue. But a lot harder(impossible) with keys.