BackgroundPrevious studies have demonstrated aggregation of autistic traits in undiagnosed family members of children with autism spectrum disorder (ASD), which has significant implications for ASD risk in their offspring. This study capitalizes upon a large, quantitatively characterized clinical-epidemiologic family sample to establish the extent to which family transmission pattern and sex modulate ASD trait aggregation.MethodsData were analyzed from 5515 siblings (2657 non-ASD and 2858 ASD) included in the Interactive Autism Network. Autism symptom levels were measured using the Social Responsiveness Scale (SRS) and by computing Diagnostic and Statistical Manual of Mental Disorders—Fifth Edition (DSM-5) symptom scores based on items from the SRS and Social Communication Questionnaire. Generalized estimating equation models evaluated the influence of family incidence types (single versus multiple incidence families; male-only ASD-affected families versus families with female ASD-affected children), diagnostic group (non-ASD children with and without a history of language delay with autistic speech and ASD-affected children), and sibling sex on ASD symptom levels.ResultsNon-ASD children manifested elevated ASD symptom burden when they were members of multiple incidence families—this effect was accentuated for male children in female ASD-containing families—or when they had a history of language delay with autistic qualities of speech. In this sample, ASD-affected children from multiple incidence families had lower symptom levels than their counterparts in single incidence families. Recurrence risk for ASD was higher for children from female ASD-containing families than for children from male-only families.ConclusionsSex and patterns of family transmission modulate the risk of autism symptom burden in undiagnosed siblings of ASD-affected children. Identification of these symptoms/traits and their molecular genetic causes may have significant implications for genetic counseling and for understanding inherited liabilities that confer risk for ASD in successive generations. Autism symptom elevations were more dramatic in non-ASD children from multiple incidence families and those with a history of language delay and autistic qualities of speech, identifying sub-groups at substantially greater transmission risk. Higher symptom burden and greater recurrence in children from female ASD-containing families indicate that familial aggregation patterns are further qualified by sex-specific thresholds, supportive of the notion that females require a higher burden of deleterious liability to cross into categorical ASD diagnosis.Electronic supplementary materialThe online version of this article (doi:10.1186/s13229-015-0050-z) contains supplementary material, which is available to authorized users.