Also, the attacker would have to be really motivated to brute force into it, and if it happened that my phone got stolen and I was worried about your scenario, I'd be just rotating all the high-value passwords I have, which is something that I think can be done quickly.
Anyway, I hate to be Cassandra, I just lost count of how many stories I heard of people regretting taking "convenience over freedom" with critical software.