Assuming there are on the order of 10 billion valid phone-numbers (there could be on the order of 100 billion depending on how far down the rabbit hole of international numbers you go). A GTX 1080 can do SHA512 @ ~1 billion/sec[0]. This means it will take 10 seconds (longer if you have a >11 digit number) to recover each phone number. This is fast enough that you don't need to bother hashing the phone book, you just brute force every possible number.
I would expect the napkin math to come out to years of compute to unmask someone in the face of a DB leak, not seconds or minutes.
[0]: https://gist.github.com/epixoip/a83d38f412b4737e99bbef804a27...