It would be a single point of failure for many apps in case the curators of F-Droid were dishonest or hacked. They could insert bad things into lots of packages without having to change the public source code. But it also becomes the only point where malware or backdoors could be inserted that way, instead of having to trust every single developer to build honestly off the source code, which we'd have to do if they just stuck prebuilt binaries up there. I don't know how rational I'm being, but it makes me trust F-Droid apps more that they build each one themselves.

I personally like F-Droid's vetting process. It's true that updates always arrive a few days later, but you can be sure they don't contain any malicious code. Furthermore, they specify all of the antifeatures a program has, which makes it easier to avoid them. If you want faster updates, you can always download a program through Obtanium.