That is a misunderstanding. Community opinion isn't just expressed in upvotes. It's expressed in upvotes and flags. Countless stories make the front page, only to be flagged off it by other users. This happens many times a day.
I didn't downvote you, and indeed no one on HN (including me) can downvote a direct reply to them.