HN has comment karma that only the comment poster can see; presumably he used to have some number of downvotes.

I think he is referring to downvotes on his post.