I do the same; a while back there was a discussion on HN (approximately): "what do you do when you see someone being downmodded unfairly?"

Up/down votes on comments should be, in my view, for whether you thought the comment was worth reading or not, irrespective of viewpoint. Likewise for upvotes on stories.

Downmod for bad grammar.

I'm kidding, but I've been dishing out karma lately, if I even so much as agree slightly with someone, or think that their viewpoint is slightly interesting/relevant.

edit: I suspect that the person that downmodded me has no sense of humour :)