It’s worse in a lot of implementations because often SMS is often used as part of a recovery flow in cases where you lose the first factor.

I find it more secure in some contexts to never give a company my phone number at all if possible, so that it simply can’t be used as any kind of authentication no matter what.