How is this different from a white person wanting a "whites only space"? Because you are seemingly arguing for a right to segregation. I think we are better off reconsidering the root desire and how that should manifest itself in a concrete right.
For example, what right do you think the children have in youth sports?
Do they have a right to win?
Do they have a right to be on a team?
Do they have a right to compete?
Do they have the right to compete against someone with equal talent?
Should it be allowed to force a younger kid to compete against older kids?
What about a short kid against tall kids?
Should a Muslim fasting for Ramadan have to compete against Christians with no dietary restrictions?
Can a white child complain about having to compete against black children?
What if only one girl wants to play a specific sport, is it legal to force her to compete on the boys team? What if all the boys are better than her?
I just don't know how you answer these questions consistently and end up in a place in which trans athletes are your only fairness concern.