It doesn't even stop at the ISP, unless you intend to span the globe with your own cables. Otherwise you're buying transit from private corporations and subject to their whims. It's pretty much build your own internet if you want true freedom of speech on the internet.

> Who determines the limits?

The limits are determined by society, usually through the vehicle of government.

> What if the determiner of the limits disagree with you?

Then I petition society to bend the collective opinion towards my own, or I change my opinion to be more in line with society.

> It's all fun and sexy to make limits until you are systematically excluded.

The whole point of societies, in the most general sense, is to improve the lives of it's members, in aggregate, over time. Or, as the Buddhists say, to reduce suffering. Systematic exclusion of antisocial and/or deleterious actors is an explicit design goal of functioning societies. To accomplish this goal societies must first acknowledge that the essential nature of their task is subjective, and requires moral judgment. This can produce bad outcomes, sure. But that possibility doesn't invalidate the approach. Laws can also be used to oppress, but we don't throw our hands up and say justice is impossible.