A program might be open source, still the binaries offered for download might be compromised. Who is able to notice that now?
You might compile the software yourself, but the majority of users wont.
You might have reverse engineered a closed source software, but I guess you won't do that for the OSS binaries.