Who decides how a disagreement gets resolved? Even if you do contractual everything, what happens if a contract dispute arises? Sometimes contracts are done based on misunderstandings, or someone tries to press an advantage in a way the other party doesn't like.

Beyond all that, the "if you believe in letting everyone exercise rights as long as they don't infringe on someone else's rights" sounds a lot like the wishful thinking that goes along with advocating communism. If everyone believes in from each according to ability, and to each according to need, you don't need legal systems and law enforcement, either.