I was taught that the first principle of security is physical security. Once your attacker has possession of your hardware, decrypting the data is only a matter of time.
They don't even need a brute force attack. If decryption works without a SIM card, then the key is on the device, protected by a code with a mere 10,000 possibilities. For the FBI, isn't that child's play?