I guess we can't win, can we? I worried more about random developers getting compromised since the surface area is much larger, but at the same time one entity compiling all packages makes them a more attractive target.
We've seen the released bundles being different to the source code before too AFAIR, so whether it's a single repository or F-Droid, both can easily screw users up if compromised.
I don't want to be paranoid but the world's not making it easy.