Individuals tend to judge personality traits on the basis of physical characteristics, particularly facial traits, although this phenomenon has been mostly studied in relation to the halo effect of attractiveness. However, there are other facial traits which may also have an impact on personality attributions; here, we focused on masculinity. We carried out principal component analysis (PCA) of 15 anthropometric measurements from 71 male faces, resulting in three components: Face Height (C1), Inner Face Breadth (C2), and Cheekbones-Jaw Prominence (C3). The targets' photographs were rated by 210 women and 177 men on scales for masculinity, attractiveness, and nine psychological characteristics (selected Cattell's factors): Warmth, Reasoning, Emotional Stability, Dominance, Liveliness, Rule-Consciousness, Social Boldness, Abstractedness, and Privateness. We found that masculinity correlated positively with ratings of Dominance and Social Boldness and that masculinity rated by men correlated positively with ratings of Emotional Stability and Privateness. We found no relationship between masculinity rated by women and the PCA components, while masculinity rated by men correlated negatively with C2 (possibly related to babyface features) and positively with C3 (which included features developed under the control of testosterone, such as jaw prominence). Our results imply sex differences in masculinity ratings. In particular, men used Cheekbones-Jaw Prominence and Inner Face Breadth as cues for masculinity judgments; on the other hand, women apparently perceive masculinity in a more holistic way.