Children with fragile X syndrome (fraX) are at risk for manifesting abnormalities in social function that overlap with features of autism and social anxiety disorder. In this study, we analyzed brain activation in response to face and gaze stimuli to better understand neural functioning associated with social perception in fraX. Eleven female subjects with fraX, aged 10 to 22 years, were compared with age-matched female control subjects. Photographs of forward-facing and angled faces, each having direct and averted gaze (4 types of stimuli), were presented in an event-related design during functional magnetic resonance imaging. Subjects were instructed to determine the direction of gaze for each photograph. Activation in brain regions known to respond to face and gaze stimuli, the fusiform gyrus (FG) and superior temporal sulcus (STS), were compared between groups to isolate neural abnormalities in the perception of directed social stimuli. The fraX subjects had decreased accuracy in determining the direction of gaze compared with controls. Region of interest analysis of the FG revealed a significant interaction between diagnostic group and face orientation. Specifically, control subjects had greater FG activation to forward than to angled faces, whereas fraX subjects had no difference in FG activation to forward and angled faces. Controls showed greater left STS activation to all stimuli compared with fraX subjects. Our results suggest that gaze aversion in fraX subjects is related to decreased specialization of the FG in the perception of face orientation. Decreased STS activation in fraX suggests aberrant processing of gaze. These data suggest that gaze aversion in fraX may be related to dysfunction of neural systems underlying both face and gaze processing.