Well, whose rights are we talking about.

There is "freedom to" and "Freedom from" lots of people not getting vaccinated affects people's freedom from getting infected.