So even though you may not have signed up for FB, they have [your name]{your number} in a table somewhere so that when and if you do sign-up it will be easy to link you with people you already have contacted.
It was called the telephone book.
I avoid FB more out of principle, not because of some deluded idea that I can avoid FB having my information. The solution "stop using Facebook and WhatsApp and other software" is harmful at worst and flippant at best.
Unless you never get your photo taken, your face is on FB and any other social media service your friends and family use. Unless you only have friends and family who never use FB, WhatsApp, LinkedIn, etc., your name, phone number, and email are on those services.
The idea that not using these services stops them from getting your information is fanciful, because your friends and family have no problem giving your information.
The problem is that sometimes this information is incorrect and based on gossips. When would you know that it resulted in real life consequences, can fb sell this info to e.g. insurance companies?