Or the laws of physics are just the same between all of the multiverses, and it's impossible to travel between them. Maybe the walls between universes are so thick that nobody will ever even detect that the other universes are there at all, making it basically the same as there being no other universes in the first place

There could be different laws of physics in this universe that makes it near impossible for somebody outside of it to visit. Or, from another perspective, would you wilingly visit yourself?

Well, no, for a few reasons:

We don't know that, if multiverses are real, that it is possible to travel between them. Infinite variations doesn't guarantee this, because that just effectively means that all possible universes exist. Were multiverse travel to turn out to be impossible, then nowhere in an infinite multiverse would it exist.

Smaller and larger infinities exist. If the set of universes where you exist is larger than the set of universes where you have multiverse travel, then most of the time, you won't be in a universe that gets you a visit, even though the number of both is infinite.

If the way the system works is that a new universe is created for each possibly each time something happens, then every time someone travels to a different universe, it will diverge into one where the visitor arrives, and one representing the option that the visitor does not arrive. No matter how much travelling is done, there will still be a set, indeed an infinite one, representing the scenario where no visitor arrives is what happens every single time any multiverse travel occurs. In that set of universes, you never have received any visits.