My view is that there is a ton of noise of HN, and HN has this problem where threads devolve into tangents/pedantry/really nitpicky arguments where people start arguing over little logical details. And rather than just go down the rabbit hole of this morass of noise, I'd rather just downvote.
Sounds reasonable. A third option you don't mention is simply ignoring the noise to upvote the quality submissions. You apparently had to look at the noise anyway to downvote it. So, is there another reason you're taking that effort?