Fragile X syndrome (FXS) is the leading known genetic cause of autism spectrum disorder (ASD) with 60–74% of males with FXS meeting diagnostic criteria for ASD. Infants with FXS have demonstrated atypical neural responses during face processing that are unique from both typically developing, low-risk infants and infants at high familial risk for ASD (i.e., infants siblings of children with ASD). In the current study, event-related potential (ERP) responses during face processing measured at 12 months of age were examined in relation to ASD symptoms measured at ~48 months of age in participants with FXS, as well as siblings of children with ASD and low-risk control participants. Results revealed that greater amplitude N290 responses in infancy were associated with more severe ASD symptoms in childhood in FXS and in siblings of children with ASD. This pattern of results was not observed for low-risk control participants. Reduced Nc amplitude was associated with more severe ASD symptoms in participants with FXS but was not observed in the other groups. This is the first study to examine ASD symptoms in childhood in relation to infant ERP responses in FXS. Results indicate that infant ERP responses may be predictive of later symptoms of ASD in FXS and the presence of both common and unique pathways to ASD in etiologically-distinct high-risk groups is supported (i.e., syndromic risk vs. familial risk).