Research into sex differences in faces has focused primarily on shape rather than pigmentation. What work has been done on pigmentation has focused on 1-D signals, such as overall hue (Tarr et. al. 2001) and albedo (e.g. Frost 1988). The current study sought to investigate whether there are other more complicated sex differences in pigmentation, and to determine whether these differences are actually used to perform sex classification. Two images, produced by averaging 22 male and 22 female Caucasian faces, were distorted such that the two average faces had the same androgynous shape. With the images spatially registered, we can determine where the average faces differ from one another in terms of luminance and hue. The male average is darker than the female average everywhere but the eyes and lips, which are equally dark across the sexes. Thus, there is greater contrast about the eyes and lips of the female than the male. A subsequent investigation of 36 male and 32 female Caucasian faces found these differences to be statistically significant. The same pattern of results was found with a comparable set of East Asian faces. For an experiment, faces were manipulated to increase or decrease the luminance contrast between the eyes, lips, and the rest of the face. Subjects were more likely to report a face as being male when the contrast between the eyes, lips, and the rest of the face was decreased, and more likely to decide a face was female when the contrast was increased. In another experiment, for female faces a significant, positive correlation was found between a measure of contrast and the rated femininity, while for male faces a significant, negative correlation was found between contrast and rated masculinity. Together, these findings provide evidence that there are spatially organized sex differences in face pigmentation, and that people use these differences to determine the sex and degree of masculinity or femininity of faces.