Insulin-like growth factor 1 (IGF-1) signaling has been implicated in Alzheimer's disease pathogenesis, and further evidence suggests inflammation can be a moderator of this association. However, most research to date has been conducted on older adults. To investigate the association of serum IGF-1 and IGF binding protein 3 (IGFBP-3) concentrations with MRI markers of Alzheimer's disease in predominantly middle-aged adults, and further assess moderation by chronic inflammation. We included participants from the Framingham Heart Study (n = 1,852, mean age 46±8, 46% men) and the Study of Health in Pomerania (n = 674, mean age 50±13, 42% men) with available serum IGF-1, IFGBP-3, as well as brain MRI. IGF-1 and IFGBP-3 were related to MRI outcomes (i.e., total brain, cortical gray matter, white matter, white matter hyperintensities (WMH), and hippocampal volumes) using multivariable regression models adjusting for potential confounders. Subgroup analyses by C-reactive protein (CRP) concentrations were also performed. Cohort-specific summary statistics were meta-analyzed using random-effects models and corrected for multiple comparisons. Meta-analysis results revealed that higher IGF-1 concentrations were associated with lower WMH (estimate [β] [95% CI], -0.05 [-0.09, -0.02], p = 0.006) and larger hippocampal volumes (0.07 [0.02, 0.12], p = 0.01), independent of vascular risk factors. These associations occurred predominantly in individuals with CRP concentrations < 75th percentile. We did not observe associations between IGFBP-3 and MRI outcomes. Our findings suggest that IGF-1-related signaling may be implicated in brain health as early as midlife.