Not that many people actually use Twitter.

That you think you are in the minority is a weird cultural phenom - people with the loud voices behaviours signal to us what is 'normative' even if it is not!

Because you see a lot of people using Twitter, you think 'everyone's doing it'. In reality - most people are far more neutral/central than the voices in the web. And most people just don't care about such things.

Try ignoring Twitter for a few months. You won't feel left out, trust me. It's mostly noise.