This is not using it as a second factor. It is using it as the only factor. Having SMS as the only factor is not purely additive. As such it can (and obviously does) reduce security.
Account recovery is hard, SMS is quite usable there, but way to insecure to be the only basis for bootstrapping account recovery.