Or you have your IDP (say ADFS) only permit issuance for authorization requests which are appropriate, again by reference to (say) AD group memberships.
If your SP doesn't even validate the audience for the assertions, then you're definitely going to be in a bad place - but the question there is "how do I limit the damage this braindead SAML implementation can do to my enterprise and how do I make sure I never buy software broken like this again?" rather than a screed suggesting authentication and authorization are the same thing.