I believe it has to do with breastfeeding. Although feeding usually makes for bigger nips, it desensitizes them, and they are less apt to be a point of sexual arousal. Therefore women who have never breastfed - and all men - will have more sensitive nipples that trigger arousal when touched and played with. Of course, there will be exceptions.
I'd love to hear other's thoughts on this theory.