Yes, I'm aware of that usage of the term "Man", the meme just got me wondering.

Perhaps there's a case to be made for elves and dwarves not even having a concept of gender - as you alluded to, dwarves don't differentiate physically, so they likely don't even acknowledge it at all. A dwarf is just a dwarf.

Elves, too, are highly androgynous, so it may be a similar thing with them. Sex is definitely a thing, though, since they reproduce just like humans do. But they certainly don't seem to have a social gender division.