Except if you're using e.g. Google Authenticator and you lose that phone, you've now lost your TOTPs. The most unhoused-friendly solution there would be to use something like Authy instead (which is another password to remember, but at least it makes it easy to recover your TOTP keys on a new device without needing the old one); next best would be to use something like andOTP which supports backups (but then you'd need someplace to store those backups, which introduces the same problems as safely keeping a phone on your person).