Threat modelling. In most models, if someone has uninterrupted physical access to a device, it's theirs.
Phones are more important in that you want to protect the assets from thieves, so we do add non-destructive physical access to our scope, but it's with a higher bug-bar. Someone being able to take your phone, compromise it, then give it back to you so you can input new assets means that a vulnerability has to be severe to be as important as a minor remote vulnerability.