The sandbox is enforced by App Store review. I don't mean the runtime enforcement, but rather the sandbox that apps opt into. It is the App Store that rejects apps when they have attempted to include sandbox entitlements that they have no business having.

Access to your photos, contacts, device identifiers, etc are all gated on sandbox entitlements, and if you remove the App Store review process then you remove the gate the prevents an app from saying "give me read/write access to the whole file system", "give me read/write access to the photos on the device", etc