It would be a single point of failure for many apps in case the curators of F-Droid were dishonest or hacked. They could insert bad things into lots of packages without having to change the public source code. But it also becomes the only point where malware or backdoors could be inserted that way, instead of having to trust every single developer to build honestly off the source code, which we'd have to do if they just stuck prebuilt binaries up there. I don't know how rational I'm being, but it makes me trust F-Droid apps more that they build each one themselves.