There are other rules. Always yield to pedestrians. Two cars can sometimes go at the same time if they're not on a collision course. If pedestrians are blocking the person whose turn it is, you can go instead. Order is based on arrival time, unless there's a lot of cars then it's yield to the right if you're unsure.

These rules combine in such a way that it's genuinely unclear who should go in some situations - the above rules might contradict, or you might need information you don't have cause you got there after something broke "the order". But if you think it's your turn and start to go, people will let you, and ultimately no one gets stuck forever.