I think you're wrong. Downvotes are literally meant to be to provide a community powered mechanism to push irrelevant content into out of view, as per the community's purpose. They are not going to be used as part of an algorithm to push more relevant content to you specifically. Of course, that's not how a lot of people end up using them, so whether it's an effective mechanism is another question.

Meanwhile, I'm on an instance that doesn't federate downvotes, so they don't affect ranking here, maybe for the better?