Yeah, I think being denied the opportunity to find out _why_ you are being downvoted, and maybe argue the point, makes you feel exposed and powerless.
Maybe one could even argue that at least a short reply should be a mandatory precodition for downvoting.