I think part of the problem here is the dichotomy between fantasy and reality.

Saying that people fantasise about rape is nonsensical; rape is by definition non-consensual, and if one chooses of their own free will to fantasise (for pleasure) about being "raped" then what they are imagining isn't truly rape. They willingly engage in the fantasy. That's not rape; rape is about control and it is not something you can choose to participate in or not. The fantasy might include some power play elements but in the end the one fantasising is entirely in control of every tiny thing about the situation, which couldn't be further removed from the reality of rape.

It's very rare that erotic literature involving this kind of "rape" will actually depict the victim resisting, saying no, crying, or otherwise reacting negatively too much. Usually it's depicted simply as an initial reluctance that gradually fades. Consent is merely dubious, rather than outright not present, at first, and is rapidly given once things start moving ("becoming pliant", in the book's terms...) Not to say that all rape victims scream and cry in the real world, but they certainly don't... ugh, "become pliant", i.e. turned on and willing. Anyone who has experienced rape or sexual assault or has had some proper heart-to-hearts with someone who has will tell you: that's not how it works. Although in a fantasy your body might react pleasurably to the initial dubious-consent scenario, thus facilitating the fantasy and the consent being given soon after, the reality of having someone force themself onto you means you'd be drier than the Sahara/softer than cream cheese. Again, the dichotomy of fantasy and reality comes into play; what people are imagining isn't rape, it only looks like it.

Personally this is my greatest hatred of books like 50 Shades. It trivialises and normalises what rape, sexual assault, and emotional abuse victims have been through. Having unfortunately had an abundance of experience with all of those things myself and through a depressingly high portion of my (particularly female, honestly) friends, I couldn't loathe it any more.