Unfortunately, there's no way to differentiate between an app using the address book locally and an app uploading the address book to its servers. Both actions require the same permission.
It would be really cool if Apple provided some new "sandboxed" UI elements for displaying contact names and photos locally without allowing general programmatic access to that data, and thereby not requiring a permission. E.g. "Show sandboxed contact image for local contact with email address foo@example.com, or default image if none."
