The issue is not with individuals keeping track of relationships and their contact lists. It's with how that information is further used, shared and sold. I wouldn't be pleased if a friend whom I trusted with my contact information shared it with others without my consent, and I would be very displeased if it ended up on Facebook[1].
PII is very valuable to advertisers (or to adtech as I recently learned[2]) as it allows them to target individuals based on interest. Beyond the fact that I don't enjoy being forced into complicitness to being manipulated into purchasing a product, I strongly object to having a profile in some mega-corp's database that has my personal information I didn't agree to share with them, for them to disect, analyze and sell in perpetuity, and to wonder how future advancements in adtech might use this data in less benign ways than today.
At the very least, I would like a share of the profits they're making from me. Facebook and Google should be paying users to use their products, or everyone on the internet rather, but I don't think their shareholders would like that very much.
[1]: https://www.businessinsider.com/facebook-uploaded-1-5-millio...
[2]: https://news.ycombinator.com/item?id=27531714