Poor grammar and spelling lowers the signal-to-noise ratio. Upvotes and downvotes are quite specifically designed to raise the signal-to-noise ratio on the site. It doesn't have to reflect on the person as an individual.
Having said that, I don't downvote on grammar/spelling alone (as I mentioned elsewhere), and I do cut a little more slack when a person is not a native speaker (but not 100% slack - better grammar and spelling are always preferred, other things being equal).