You can't downvote a post, so that's not a factor.
Also it's not as powerful as you think. In the past I have spent a lot of time looking at /new, and upvoting stories that I think should be surfaced. The vast majority of them still never hit near the front page.
It's a real shame, because some of the best and most relevant submissions don't seem to make it.