How would I know that an app has had lots of developer eyes on it or not? It's crazy difficult to uncover the latest known security posture of open source software.

Finding out eventually is the exact same risk I take when I use proprietary software. It requires my trust. And it's theoretically just as possible to discover vulnerabilities in closed-source software (Windows, for example).

The easier way would be to monitor network traffic. If random encrypted information is uploaded, then block it, whether or not you have an "open source" rom.

Better yet, just get a Nexus device without the carrier apps pre-installed. Stock Android does not have a "upload all data to the NSA" feature built in, it would be easily discovered and would be the biggest news story of the decade.

Unless you are the NSA, then I wouldn't advise using a custom version of Android, you'd likely be much less secure than with a stock (and up to date) version.