Compare and contrast with people saying "neurodiverse" when they mean "neurodivergent". I think there is general agreement that a single person should be described as "neurodivergent" rather than "neurodiverse". Some people say the wrong word just because the two words are so similar and are happy to be corrected.

On the other hand, if we're talking about other kinds of diversity then I don't think there's a similar pair of terms. "The addition of this gender-divergent person increases the gender-diversity of the team"? I don't think so!