To be completely fair they lived in a world where there actually were real differences between the races. In our world racism is silly because there aren't any actually relevant differences between the races, but the differences between elves, men, and dwarves are very real in middle earth. Not enough to pull everyone over a comb (to use a swedish proverb), but at least there's some substance to it.