The system I've proposed is: you can downvote a post if you've upvoted a reply to it, or if you have replied.
If somebody is breaking the rules in clear bad faith, that's what flagging is for.