We do segregate many sports on other dimensions, e.g. by weight in boxing.
The problem with trying to do something like that across the sexes is that the performance gap between elite men and elite women (i.e. those at the top of their statistical distributions) is so large in so many sports (even when controlling for height or weight).
For example, a large number of amateur men who are well under 6 feet tall can dunk a basketball (and Spud Webb was 5 foot 7!), but it's remarkable to find even a professional female player of the same height who could do this. Similarly being tackled by an 85kg male rugby player is a whole other universe of pain compared to being tackled by an 85kg female. Anybody who knows sports can find similar examples. It’s not about testosterone or some other individual item - it’s the whole complex biological organism acting in concert. When you are working with so many dimensions and those are also interacting in real time, it is very hard to divide things into "fair" categories.
Because of these issues, women fought very hard for (and won) the right to have their own competitions, free of male competitors.
For more examples, see https://boysvswomen.com/#/. If Olympic events were mixed sex, most events would have no female competitors at all. Teenage boys routinely beat female world records.