In the US, it really doesn't matter who says it, the only thing that matters is who it's being said about.
If you are a "public figure" -- which is a much broader category in 1A law than you think -- then in order to prove defamation, you have to prove the thing was false _and_ that the person saying it knew it was false at the time. Not that they were mistaken, not that they were careless, not that they knew later, they deliberately lied and knew they lied as they said it.
If your next question is "how do you prove what someone was thinking", then yes. That is the reason it's nearly impossible.