Introduction: Chronic stress and related changes in serum cortisol have adverse effects on brain structure and cognition in animal models. However, evidence from population-based studies is scant. We assessed the association of early morning serum cortisol with cognition and brain structural integrity in middle-aged adults without dementia. Hypotheses: High or low levels of serum cortisol are associated with lower cognitive performance and brain volumes. Methods: We evaluated dementia-free Framingham Study (Generation 3) participants (mean age 48.5 years; 46.8% men), who underwent cognitive testing of memory, abstract reasoning, visual perception, attention, and executive function (n= 2231), and brain MRI (n=2018) to assess total white matter, lobar gray matter, and white matter hyperintensity volumes and fractional anisotropy (FA) measures. We used linear or logistic (cortisol categorized in tertiles, middle tertile as the reference) regression to assess the relations of cortisol with cognition, MRI volumes and voxel-based microstructural white matter integrity and gray matter density, adjusting for age, sex, APOE and vascular risk factors. Results: Higher cortisol (highest tertile vs. middle tertile) was associated with worse memory and visual perception, as well as lower total cerebral brain, occipital and frontal lobar gray matter volumes (Table ) . Higher cortisol was associated with multiple areas of microstructural changes on voxel-based analyses (gray matter density and FA). The association of cortisol with total cerebral brain volume varied by sex ( Pinteraction =0.048 , highest cortisol tertile inversely associated with cerebral brain volume in women [ P= 0.001] but not in men [ P =0.717]). There was no effect modification by the apoE4 genotype of the relations of cortisol and cognition or imaging traits. Conclusions: Higher serum cortisol was associated with lower brain volumes and impaired memory in asymptomatic young adults in their forties; women may be particularly susceptible to this influence.