That's what I increasingly find myself doing. The downside is that you get less and less exposed to new material; "old" HN would usually surface e.g. tptacek's comments, but there would also be some highly-voted and insightful comments by relative nobodies who were experts in the particular field being discussed. This still happens, but reading the highest-voted comments is becoming less and less worthwhile.

The problem is that down-votes have account consequences. They aren't the mirror opposite of an up-vote. Ten upvotes and a post is just popular, ten downvotes and you might get banned.

If moderation was a flag that some people cared about but that left everything intact then people could ignore it if they wanted.