Your theory sounds more plausible to me. I'm male and find an all-male environment incredibly competitive, toxic and tiring to myself. It's very hard to get reasonable discussions and almost always degenerates into competition.

So I can imagine that you just bail if you aren't conditioned to put up with that kind of crap.

In the short term that may be true, but over a timespan of decades or generations, social attitudes about the roles of women more generally are more important. It is only very recently that the kind of constraints you mention are influential—previous generations would simply see women (or men) excluded from various domains.