Spec ops tend to be more averaged sized, with denser builds (as opposed to bulkier muscle mass)(1). Although, quick caveat, take a look at old photos of Vietnam War era special forces vs modern late GWOT; the difference in average muscle mass is rather stark.
Historically, military selection emphasized calisthenics and load carrying capacity (body armor/weapons/rucks) + endurance. Although modern assessments have somewhat shifted towards weightlifting components, calisthenics/rucking are still a major focus. There is generally a disadvantage with height, where longer limb length creates more leverage to overcome, which is a disadvantage in both calisthenics and moving external weight around.
Average size is ~5’10” @ 180lbs at selection
(1) https://sofprepcoach.com/special-forces-selection-body-compo...
