For negative scores, you can kind of tell based on how grayed out it is. And while there's a lower bound at -4, I don't think HN does any scaling to translate vote counts into the scores it displays to you (but it does when calculating the contribution to total karma, where submission scores count half).