No, and yes

A partly because there is always small risk involved in everything in life. However, you do need to realize what the paranoia is supposed to help prevent. If you install an unverified os thar hijacks the machine, then you will need to live the loss of the machine. Particularly, your paranoia stems mostly because you are afraid of losing your privacy or machine to a possible malicious attack. The proper way of mitigating this is to build from a verified source instead, as in you know what the code is, but you can't because the systems are too complex for a single person to verify manually. Unfortunately, your paranoia is too shallow and unfounded, and you will need to do some self reflection to come to the acceptance of a world you can not know or control.

On the other hand, when you are doing proper procedure to do a basic verification step to prevent corruption or possible simplistic malicious attacks, it is good behavior. It great you feel the need to verify your distributions and that your paranoia is likely not paranoia at all! A simple checksum is nice. Also, how can you verify that a gpg key is also a good one and not a tampered gpg that matches the tampered ISO? There is a level of trust you have, and your paranoia is simply never going to be able to be paranoid enough to encompass everything. You are forced to place some trust in something. After all, there are vulnerabilities found in gpg software that does either the encrypting step or verification steps(either from gaining the private key or the verification step throwing false positives). There is only so much we as normal people can do.

So overall, it is both. You are responsible, and your paranoia is too shallow to be useful for you. Go the extra steps and compile from source. It is safer because the code is less likely to be tampered and you can know it is right because you made it yourself.