By looking at everyone's choices, not just the one person you want to pick on.

Suppose Person A is an asymptomatic COVID-19 carrier and doesn't know it. They go to the grocery store without a mask. Person B also goes to the grocery store without a mask, and gets infected by Person A.

You could say Person A is responsible because they should have worn a mask. But you could also say Person B is responsible because they should have worn a mask. Yes, if you know there are a lot of asymptomatic carriers, you should take into account the possibility that you might be one. But you should also take into account the possibility that someone else might be one. So the appropriate rule in this case is that anyone who doesn't wear a mask in a public place where they can't social distance takes the consequences if they get sick. Someone who gets infected because they didn't wear a mask can't pawn off the responsibility on the person who infected them that also wasn't wearing a mask.

In other words, your scenario where "a few people who don't take care not to infect others can kill millions* simply cannot happen unless those millions of other people also don't take care not to get infected. So putting all the responsibility on the person who unknowingly infected someone else, instead of sharing it equally among all the people who didn't take common sense precautions, is wrong. Justifying invasive surveillance and tracking of everyone on those grounds is basically saying you want the government to make sure you don't have to take any prudent precautions or exercise any common sense at all in your daily life in order to avoid having anything bad happen to you. That's not reasonable.