For example, for a low competition keyword, 1 link can make a big difference to a site's rankings. For a high-competition keyword it takes many many more links to change anything. It's the same algorithm affecting the rankings in both cases, just in the latter case there's a lot more data being used as input.
That's the way they seem to do things, so I'm guessing there's a fair chance they'd take a similar approach to the influence of down-votes on rankings, if they went down that road.
There'd have to have some sorts of thresholds, but there'll always be points above which the algorithm can be gamed, just like pretty much every other aspect of their ranking algorithm can be gamed.
Right now, in a low-volume-keyword niche, a sleazy operator can kill the rankings of competing sites pretty easily by buying them a few dirty links and letting Google's algorithm do the rest. If a site hasn't got lots of quality links pointing to it, and let's face it, the vast majority of sites don't, it's pretty easy for someone to kill its rankings.
A lot of people say that it's impossible to affect the rankings of someone else's site, but that's simply not true. You can't easily affect the rankings of a big established site with lots of good links, but a little small-business site? The reality is it's pretty easy. I don't think Google have any interest in quashing that particular myth, because the reality is actually kind of scary for small business operating on the web.
(Apologies if I'm going off on a tangent there. You are absolutely right in pointing out that I was assuming a flawed implementation.)