I wouldn't worry about it. HN karma is effectively useless, and you can get downvoted for all sorts of silly reasons... ranging from "person was actually trying to upvote you but misclicked" to "you said something that goes against the prevailing collective wisdom and people downvoted you to express disagreement" to "you actually did something that's against the site terms", and everything in between.

Your karma is going to fluctuate over time... don't take it personally.