While I agree with you for the most part, the context and nature of the debate is important. In a casual conversation that happens to be an argument, there's (usually) some semblance of a logical argument that can be teased out from a large amount of non-logical communication. In that context, picking on any of that non-logical communication and calling it a fallacy is silly.

In the context of a real debate, however, when each party has the goal of arguing a side, and an audience is there to hear and evaluate each party's words in the context of the argument, I think a phrase like "you immoral moron" is indeed to be taken as an ad-hominum. In that situation every statement is assumed to be part of your argument.