Might be an idea to make your lemmy home on an instance that doesn't allow down votes (those exist). People on other servers might still downvote it, but you won't notice.

In fact, I think I'll go do that myself.