General-purpose processors have to be secure while executing untrusted code, providing a large number of features, and providing good performance.
The secure enclave isn't subject to these constraints, allowing for more conservative design decisions.
You've found a privilege-escalation attack that can let sandboxed apps escape their sandbox? Still secure if the chip can't run apps in the first place. You've found a bug in the USB disk mode emulation code? Still secure if the chip doesn't have any USB code on it. You've found a bug in branch prediction? Still secure if your chip didn't use it. You've found a way to abuse the third party developers' debugging interface? Still secure if your chip provides no such interface...