The assistant captains (usually called vices) on each team are the arbiters. It’s in one’s best interest to keep the game moving and not get bogged in frivolities, there’s a bit of a “tie goes to the runner” heuristic when deciding on violations.

In my years of curling, I’ve never seen a disagreement on rules left unsettled between the vices, but my understanding is that one would refer to vices on the neighboring sheets for their opinion, acting as a stand-in impartial authority. In Olympic level play I do believe there are referees to avoid this, but I really can’t overstate how unusual that is for any other curlers.