This is where artifact binding can greatly increase the security….
Browser sends artifact to RP, RP fetches assertion from IdP via HTTPs, afterwards verifies the signature.
Signature verification is not implemented correctly? The attacker still needs to break HTTPS…. And then you would have a big problem anyway.