I agree with you when it comes to random apps, but messengers are probably the only exception where I accept it.
If I gave my phone number to someone, I made an implicit social contract with that person or organization that they may contact me using that phone number. I don't see it as a big stretch if they use that number to send me a message or call me using WhatsApp, Signal or any other messenger where I registered using that same number. It's for the same purpose of communication, after all.
If I don't want to be contacted, I just don't share that number.
And it's not like people don't know that messengers or social networks can collect contact metadata about messages and calls. It's what traditional landline and mobile providers have been doing for ages.