In practice, apps that want access to my contacts usually keep asking for access every time I try to do anything, until I eventually either relent to make the prompts go away or click allow by accident.

And that's me as a computer-enthusiast. I would bet money most normal people just hit allow always. Because it's easier.