It comes down to where we draw the line. We limit healthy women from competing in paraplegic games for example, because of inherent advantages.
In certain sports, height might not be formally regulated, but weight classes are regulated. And in those sports it is arguably an advantage to be shorter, as you can be bulkier overall and dedicate more of the limited weight to pure muscle mass vs your skeleton. Although there are also considerations for things such as reach in some circumstances.
Overall though, the difference between a slightly taller athlete of a given sex is nowhere near the athletic prowess differences between a given athlete of the same height and of different sex. A 5' Lebron James would still dominate a 7' Caitlin Clark. Maybe there would be height classes just like there are weight classes and sex classes, if height were such an influencing factor.