It's also a useful catch-all to reject candidates who, while they have the technical skills, would cause significant personality clashes within the team. I don't disagree that the "culture fit" criteria can be abused in the way you're describing, and I'm sure it is, but I think it's a little less black and white than you're describing.
