There's nothing wrong with not wanting casual sex. There's nothing wrong with wanting it.
The problem comes from saying that what you want is the only right thing, and that everyone that doesn't want the same thing you do is wrong. Or from being deceptive about what you want in order to coerce the other person into sex.