Ah, shadow profiles again.
Maybe the real problem here is that people talk about "your data". There's no "your data" in a shadow profile of you. There's data about you. I have a feeling that talking about the data as if it belonged to the person it refers to makes people reach weird conclusions.
I'm having problems with articulating clear examples, but I don't like the very concept of ownership of facts - whether they're algorithms or phone<->name associations.