A lot of the downsides are mitigated by using Google Voice as the SMS number, since attackers can't migrate your number away from Google.
But in general, I totally agree with you from a security perspective. I just think that it's a difficult thing to get people to use authenticator apps. Apple has resorted to baking the functionality into their OS.