Yes, absolutely, and that's a great observation about medieval serfdom. People seem reluctant to engage with the problem in its full generality, which is essentially: What kinds of restrictions (if any) should we as a society place on contracts between parties who would otherwise agree?
Laissez-faire capitalism's answer is "None at all", which is appealingly simple and works well whenever the parties involved have similar levels of bargaining power. But in practice a huge number of negotiations occur between parties with vastly different levels of bargaining power -- and in that case, having few or no restrictions can lead to emergent behaviour (especially positive feedback of wealth and poverty over time) that many feel to be unfair.
It's difficult because the alternative -- adding restrictions -- seems almost impossible to do without "playing favourites", which leads to claims of unfairness that are hard to argue against because they are so plain: They're right there in writing. It's harder for many people to see how the absence of restrictions is also a kind of "playing favourites".
