Clothing is a much more visible identity marker: if you're dressed in clothing from particular African cultures, that's a strong signal that you belong to that culture—so if you do not, in fact, belong to it, that can cause a number of kinds of problems in the right (or wrong) circumstances.

Furthermore, for many cultures, the clothing that most recognizably signals membership is related to some form of cultural or religious ritual, and wearing it outside of that context can be very disrespectful.

Imagine if, for instance, you were in Nairobi (or Tokyo, or Mumbai) and saw someone wearing the traditional robes and cap of a Catholic cardinal—and you saw them partying, being vulgar, doing drugs, that sort of thing. Obviously, the connotations are not the same, because of the overall dominance of the Catholic Church and Western culture generally, but I hope that it gets across the general idea of why clothing can be more prone to appropriation, and why that's a problem.