Check if they follow the specs. Especially with SAML, I've found many, many implementations that are just broken. Such es logging a user out of the IdP after idling, when they should just revoke the session for their SP.

Another good one is when they INSIST on using an email address for the name-id. These things change, so let me PLEASE use an immutable I'd ... That's already close to not getting accepted because it invites problems.

Another one being Auto-Provision ing not being implemented, needing an additional user sync. This also contributes to not getting accepted.

If an SP does not implement certificate rollover, it's getting an Instant NO!

But to be fair, Microsoft's IdP side has some flaws as well, which is annoying.