This would not remotely "make the whole downvote system useless". The primary use of the voting system here is to sort out discussion by subjective value. That would only be very indirectly affected (through the very distantly secondary use of karma to confer downvoting ability).
If anything, it would debatably enhance the primary use of the downvote system, by giving authors of unpopular comments an incentive to remove from the discussion what the community has judged to be noise.
Though I would personally hate that, as a fundamentalist anti-deletionist, and as a frequent dissenter from this community's judgements of value.
Jeeez.
Edit: My upvote brought him back to zero, for now.
Rational minds can disagree, and intelligence can't understand much-greater-intelligence (so lessers might downvote much-greaters for that reason, besides the first-mentioned reason), but you don't have to worry about armageddons of downvotes. Or the balance of votes.
Just upvote what you consider "signal", and downvote the worst kinds of noise.
Thus you, my parent comment, are barely voted on at all, while the one by jewbacca rises to the top all day, no matter the weather.
