At the same time, when your project takes off past a certain point, don't you gain some amount of responsibility if you know that you have become an attack vector?
If you don't want to take on that responsibility, then you could just quit hosting binary builds, and focus on the source. Then it becomes "someone else's problem" to deal with hosting binaries, and providing security.
Good point; though approaching the problem in an indignant and righteous fashion doesn't achieve anything; offering to help with a solution is more in line with the (ideal) OSS ethos.