Fragile X Syndrome (FXS) is the most common heritable form of mental retardation and monogenic cause of autism spectrum disorder (ASD). FXS is due to a mutation in the X-linked FMR1 gene and is characterized by motor, cognitive and social alterations, mostly overlapping with ASD behavioral phenotypes. The severity of these symptoms and their timing may be exacerbated and/or advanced by environmental adversity interacting with the genetic mutation. We therefore tested the effects of the prenatal exposure to unpredictable chronic stress on the behavioral phenotype of juveniles of both sexes in the Fmr1 knock-out (KO) mouse model of FXS. Mice underwent behavioral tests at 7–8 weeks of age, that is, when most of the relevant behavioral alterations are absent or mild in Fmr1-KOs. Stress induced the early appearance of deficits in spontaneous alternation in KO male mice, without exacerbating the behavioral phenotype of mutant females. In males stress also altered social interaction and communication, but mostly in WT mice, while in females it induced effects on locomotion and communication in mice of both genotypes. Our data therefore highlight the sex-dependent relevance of early environmental stressors to interact with genetic factors to influence the appearance of selected FXS- and ASD-like phenotypes.