"I can see the boycott coordination angle (except, the offensive vendor can just change its identity)."If the vendor changes its identity but the public is still aware of the connection, the new identity goes on the sanction list. If the new identity can't be linked, then it has absolutely no reputation and has to start from scratch to convince people to buy from them. That is an effective deterrent.
"What if everyone around you believes that you should be detained, but you actually shouldn't (according to the theoretical 'rule')?"
There are no "rules". What I'm describing is a society organized via voluntary actions, but by following prescribed rules. If everyone around you thinks you should be detained, you get detained. That's how society works.
Sanctions don't prevent bad behavior, they punish bad behavior as a deterrent. Separately, a police force exists only as long as people fund it, and as long as people approve of their uses of force.
I'm not saying this is all bulletproof, but I think it's going to be fun to explore as we build the foundations for these sorts of systems.