There are certainly plenty pop culture moments reinforcing dong symbolism. From Kate Hudson dubbing Matthew McConaug-hottie's member "Princess Sofia" in How to Lose a Guy in Ten Days to John Mayer comparing his one-eyed snake to a white supremacist, the penis and its size are constantly pushed into our consciousnesses. Kate Gosselin became the embodiment of every man's worst penis nightmare when she likened Jon Gosselin's member to that of a 9-year-old boy, and tweeted pictures of his teeny-weeny wang. (Compared to her vitriol, my penile faux pas seems not so bad.)
The problem is no doubt made worse by the fact that the male anatomy isn't nearly as quantified as the female, and many don't accurately know how long their penis is or how it relates to other men's. According to the Kinsey Institute (and Wikipedia), the average erect schlong is between five and six inches. This opens the door both to mystique and doubt, something women don't have to deal with, since their sizes have been dissected, calculated and commercialized (there's no Victoria's Secret for jock straps).
In an article on penis size, columnist Jack Murnighan admits that he's "gone through the gamut of perceptions" regarding his endowment. "[But] as the apparatus itself never really changed, these opinions obviously have more to do with my sense of self and my relationship to my own sexuality than anything you could measure in inches." He concludes, then, that the penis for a man seems to be "a consolidation of his sexuality as a whole. No wonder we worry."
Another man I spoke to, whose ex-girlfriend had made a joke about his penis while they were still dating, said that "despite her insisting that it was 100 percent meant as a joke and not true, it still struck a chord. I think most guys are a little self-conscious about it, so even a tiny jab can open up a big area of self-doubt."