CoCs didn't jump out of the void for no reason. They are a response to years of harassment and unprofessional behavior at conferences. Conferences have always had a list of rules to follow and breaking the rules could get you ejected from the conference. In theory all a CoC should do is codify some bad behaviors that people believe to be acceptable but are actually not, so in that regard they shouldn't be controversial.

You hit the nail on the head though: if the people enforcing the conference rules don't act in good faith or don't follow the rules then the rules don't matter.

In theory a conference could just have a rule that says "act professionally" and if properly enforced that would be sufficient. On the flip side such a conference could have a petty tyrant rules lawyer for an enforcer and decide anyone they don't like was not acting "professionally".

There is no outside body that could (or should) be adjudicators. Conferences live and die by their reputation. All we can do is judge them by their actions.