I think the bigger problem is that most people already have a basic level of this type of functionality built in to whatever devices they're using, in the form of the contacts/address book. And obviously that's not nearly as powerful as a proper CRM - but it provides a way to store details about the individuals, notes, when you last communicated, etc.
So you're not just looking for people who want to keep regular notes/etc, but also ones who need something more powerful than the built in functionality already available to them. And they need to trust you with all that (potentially very sensitive) personal data, and hope that you'll be around for the long term (because it would be very annoying to lose that).
Could be an interesting project (especially integration with all the different platforms they may use); but hard to see how it would make much money.
