I hear you. I didn't try any of them until I was in my late forties. I don't drink and we basically tried pot a few times a year in our thirties. I read about set and setting and we decided to try it.
Taking mushrooms with my wife in our forties was probably one of the top ten most important things I've done in my life.
The traumas absolutely come up. That's kind of the point. They take you where you need to go even if you don't want to go there. I had a lot of rage and anger and deep embarrassments and now it feels like that's been scraped out of me, like a boil that needed to be lanced.
It's not for everyone to enjoy but I think it's for everyone, once or twice.