Well, it is encrypted with a pin. Unless you make a long pin or switch to optional alphanumeric, the contact list is just protected by SGX and a short pin.
The SGX works like the iPhone auto-wipe. But it is in an online service with lots of juicy data in one place, so the target is bigger. A vulnerability could leak your contact list.
