You are putting "individualism" where none was mentioned.
> "fine none of us then as long as I can insulate myself from the worse effects of it".
No. More like "there is no way that any central entity will be able to solve the conflicts of everyone without turning into authoritarianism and tyranny, so let's stop pretending that we can do that and create a plethora of different communities where people are closer to those with decision-making power."
I can (and want) to help my neighbor and those close to me as equals, but I have no interest in being a mere subject serving as an instrument to whoever is in power above.