Men don't simply walk away for now reason. That being said, that doesn't mean it is you, it could very well be him. After a short time together, it is impossible for you to know all about his issues and his past. He is an older man, he pursued you, and now he claims to have lost your number. Something isn't right.
Sadly, I think you need to examine how long after you slept together that his interest deteriorated. As a man, it is difficult to admit, but when your 50 you pursue 35 year old women, and at 40 you pursue 25, and at 30 you pursue 21 year olds. There is a certain element to feeding our ego's, feeling as though we are still capable, feeling as though we are attractive to younger, beautiful, women who have a surplus of younger and likely more attractive options.
The best thing to do would be to sit back, show him nothing more than cordial acknowledgment when you run into each other in the building, and to let it pass. If he shows interest again he is likely the type of guy who wants what he can't have, and if he doesn't show any interest than he is probably a man of poor character. Either way, a future between the two of you is likely not in your best interest.
ANSWER THIS QUESTION