Sure, psylocybin won't kill you, but the threat of inescapable psychological distress is real. Maybe it's less likely if you aren't predisposed to psychosis, but you can't easily know for sure that you're not.
And yes, it's possible to misuse SSRIs, but taking an obviously oversize dose is still often a treatable situation. Combining them with depressants I think is a degree of misuse that is incomparable.