Whoa. I hadn't realized this. So someone that knows my email address and has my phone has access to my entire life, because all password resets use my email address.
If it's come to this, to using "something you have", then we can all go back to using paper password notebooks. They offer the same security, surprisingly.