The older usages of it weren't as bad, supposedly (and I don't have the access to link a source that's authoritative without paying) and were generally akin to calling the wife the queen of the house, but also implying that she was more of a dowager queen, and one without power.

But it eventually just fell into the usual trope of men either dismissing their own feelings by joking about their wife, or expressing the idea that the wife is something you put up with rather than respect and love.

Now, that first part is important! Using terms that seem derogatory, but are really there to cover up genuine emotion that is untoward for a "real" man has been a thing for a very long time now, so you can't just assume that any given man using terms like "old lady" or "the old ball-and-chain" are being misogynistic. It's becoming less common for men to cloak their affection behind dismissive or derogatory terms, but it is still there.

It's like when you're petting your dog and you're babbling about them being a monster or beast. You love the dog, but you're using inverted meaning to express it. It's just that the freedom to babble to your dog about how wonderful they are became more acceptable sooner. Which is a bit of an indictment of the systemic misogyny we live in.

Anyway, if you compare that to the supposed origins of "old man" to refer to a father in specific (rather than the use to mean a husband/boyfriend which is one use of the phrase), it came from naval usage like so many other neat phrases.

Is was, and still is, a term used for a Captain or other commanding officer. When it got applied to dads, it was from a similar way of thinking, wherein the father is in command of the household, but it was also an honorific of sorts.

The reasons for it being used that way in the English and American navies is a whole essay by itself, but that essays are already out there online, so I'm not making this longer by going into it lol.

Anyway, the tl;dr that's horribly misleading is: a combination of ageism, patriarchal thinking, and a tinge of misogyny here and there.