The three major questions are, what do you mean by "violence" (which you have answered), and what do you mean by "legitimate", and what do you mean by "freedom"?
What if, say, your employer in cooperation with others were to blackball you so that the only employment you could get were as an unskilled laborer? That clearly wouldn't be violence. Would it restrict your freedom? Apparently not?
How about if a group of people arrange to ensure that you can only live in a certain area, purely by economic means? No violence, right? Legitimate? Are you less free? No?
Suppose you live in a society that makes collective decisions by voting. But, you are not allowed to participate in those votes, by virtue of material circumstance, say. Still no violence. Still no less free, right?
What about violence? Can I burn down your house if you don't do what I want? If I make sure no one is injured? Material circumstances are excluded, right?
Now, what makes violence legitimate versus illegitimate? If a group of people kill one of your neighbors for violating some extra-legal rule, that would clearly be a crime, right? But what if the people doing it cannot be identified? Or, if identified, arrested, and prosecuted, they are found to be not guilty. Repeatedly. Clearly, you would feel some pressure to follow said rule although that would not be a restriction on your freedom, right?
Is chattel slavery an imposition on the freedom of the slave, if physical violence is not used?
I suggest that your definition of "freedom" is very far off from the normal, colloquial definition ("the power or right to act, speak, or think as one wants without hindrance or restraint" according to the Goog')---there are plenty of restraints on your power to act and speak that do not involve violence. (Thinking? We're working on that.)
You mention insurance schemes, which is always a fun topic because I'm old and can remember when requiring liability insurance for drivers was controversial. Is it legitimate for anyone, especially the state, to force you to be financially responsible for your actions? Would that be a restriction on your freedom? Absolutely! Would it be a legitimate (oooh, there's that word) restriction?