You do. You can choose to not live in society. Go live in the jungle, or buy a raft and jump onto the sea; no one will follow you to tax you.
Of course, you won't do this, cause you like living with people. You like living in cities but cities cannot run without roads, piers, planning, etc. And you like being safe, knowing if people hurt you or take your stuff they'll be punished, if your house catches on fire it'll be turned off, and if you're out late at night the sidewalk will be lit so you can find your way back home.
You have the choice, it just doesn't seem like one because not living in society is just not worth it.