It depends on the phone, but for many phones the security story remains very good even when lost, unless someone knows your passcode. So it’s still “something you know” protecting the password and the TOTP code, but it’s different things that you know and strict rate-limiting on the phone side that wouldn’t be possible on an internet-exposed authentication system makes it extremely difficult to guess the phone passcode.