The problem is that down-votes have account consequences. They aren't the mirror opposite of an up-vote. Ten upvotes and a post is just popular, ten downvotes and you might get banned.

If moderation was a flag that some people cared about but that left everything intact then people could ignore it if they wanted.