I mean, we do have weight categories in combat sports, right? I don't see why we couldn't come up with similarly neutral categories if we think it's good to segment people out by physical advantages. The parent comment is making a good point, though: it feels like some people care a lot about physical advantages that map onto gender stuff they care about, and not a lot about weird genetic anomalies that provide physical advantages that aren't gendered.