The first questions I have for you is: Why are you trying to do exactly as he wants? What are your preferences and desires in this negotiation? You have the right to assert yourself equally is these negotiations. Did you tell him what you want?
A graduate school teacher of mine once said to the class "There is no such thing as meaningless sex!" I wasn't sure what she meant at the time but I've come to understand that for us humans, love, affection and genuine emotional intimacy are not separate from the act of sex. We fool ourselves into thinking we can act like the other animals and have emotionless sex.
You would benefit yourself immensely if you would take the time to ask yourself what you really want with a man. Do you just want a sexual release so you don't have to feel horny or have to masturbate or do you want a lover, friend and companion? If you want the second you will need to forgo casual sex with men.
This is guesswork on my part but chances are that he was doing one of two things: either he was just looking to get his rocks off and once that was done he wanted you gone, or, he does have some feelings for you but is unable to tell you that due to his own issues.
Either way, if you can learn to love yourself fully, independent of whether a guy is giving you attention or not, you will be able to avoid putting yourself in these kind of difficult situations.
ANSWER THIS QUESTION