This might help with him rolling poorly. I run a swarms AC differently than other creatures. I wanted to give some the peoper feel of being everywhere around the party and settled on this as how to simulate it. When it's at full HP the swarm has a much lower AC, I usually subtract 5 from its stat block. This is because you can barely swing a sword without hitting them, theres just so many. Once the swarm has lost half of its HP I add 5 to the AC on the stat block, because as they thin out they get harder to hit.

Obviously this doesn't work in every situation. To keep combat from grinding to a halt this works best when the swarms are part of an encounter with other creatures. I used it as a supplement to a higher creature’s fight and it kept them relevant as a distraction during the fight. If it's just the swarms and the party struggles to finish them off don't forget that they can retreat under such loses. You don't want them to just keep swinging and missing if they're really struggling.