If a change in
other people's ability to do mathematics affects your level of enjoyment in doing mathematics, you don't really enjoy mathematics. You enjoy feeling smarter than other people, of belonging to an exclusive club.
Preserving people's access to this kind of enjoyment is not something that should carry any weight in my opinion.