If you divided the competitors into “has freakishly long arms” and “doesn’t have freakishly long arms” groups to compete within, and Phelps met the metrics for freakishly long arms, are you saying you think he should be free to compete in either group?
If so, there was no point in dividing into groups.
That said, I am sure athletes and governing bodies could agree on a better solution than outright banning- for example all it takes is a group that pairs a freakishly long armed swimmer with not, and they compete as pairs. Or an open group- maybe someone without freakishly long arms will find a way to win.
Anyway, it’s sports, people will min/max everything you let them, and we know from history they may bend or break rules as well. At the end of the day someone has to make a rule and enforce it, over time it will evolve.