Trans men don't compete because women are essentially non competitive against men in top level athletics. Which is why trans women are controversial in women's sports. Every year there are hundreds of males highschoolers who outcompete females Olympic gold medalists. By allowing men to compete in women's sports you prioritize the notions of identity of what's usually a single individual over an entire class. It's plainly sexist.
As for intersex individuals, put them in their own competitive class.