For KDE specifically I think there's a dbus interface that can be called to switch it. You can find it with QDBusViewer or D-Feet.
I'd imagine XWayland would follow the same since it's essentially a Wayland client. But if you ran the xmodmap under xwayland, that may have inverted it in xwayland, and it's already inverted in KWin which would double invert it aka put it back to default.
Otherwise doing it at the evdev level will definitely work. It's a bit of a nuclear option but if it works...