If there wasn't at least a little bit of complicity in it being interpreted as a general diss on masculinity, the term would never have caught on, or a different term would have been chosen in the first place. In the face of repeated misunderstandings, people using the term would pick a new one. They haven't done that. Why? It seems to me that they are quite happy to be "misunderstood" 95% of the time.
The gender swapped equivalent seems to be "internalised misogyny". Do you think I could get away with calling it "toxic femininity" instead? Nobody would let that pass. Do you think I could get away with calling any obviously bad behaviour associated with femininity, "toxic femininity", at all? I don't think I could.
It's widely considered by men to be an insulting term, and if people who use it don't want to insult people, they should pick another one. They don't though, which is telling.