Exactly. Even more interesting, all of the source code can be OK and just some subtle configuration tweaks can be enough to compromise you. Or just some build flag that you don't even see in sources. Often you don't know the build flags of every binary as soon as you use binaries. You also don't know if the compiler is tweaked to do some preprocessing you don't know about (see Reflections on Trusting Trust by Ken Thompson):

http://cm.bell-labs.com/who/ken/trust.html

For security conscious the prefect state is the OS which changes very, very slowly, fixing only security bugs and having binaries used by as many people as possible and which change so seldom that more people can even check them by disassembling them. You don't want to only check sources, you want to disassemble the binaries and decide if they match the sources.

And only then you want to be sure that all configurations are what they should be. Not easy at all.