A case about access to contraception has nothing to do with free speech at all, nor is it even the same kind of "privacy." They're two different concepts, with the same word.
It's actually the seizure clause. The government can't seize your duck without a warrant, and that means if somebody's speech about you weighs as much as a duck, they've seized a duck's weight in information about you (measuring as if it's written on constitution-grade parchment), which violates your right to privacy as implied by the 4th amendment.