Vaccinations are a public health concern. One of the few areas (in my personal philosophy) where the state can argue a legitimate interest in what would otherwise be a violation of personal privacy and autonomy.
The main problem is that most governments have historically shown themselves to use every possible angle they can come up with when they want to go after someone, no matter the legal stretch. Thus, collecting data that could be vital to a legitimate concern could also be later misused for other nefarious purposes. If governments would be the "bigger person" so to speak, and not stoop to dastardly levels, more people would have faith that the government would not infringe on their inherent rights without a damn good reason.