Deficits in memory are seen as a canonical sign of aging and a prodrome to dementia in older adults. However, our understanding of age-related cognition and brain morphology occurring throughout a broader spectrum of adulthood remains limited. We quantified the relationship between cognitive function and brain morphology (sulcal width, SW) using three cross-sectional observational datasets (PISA, AIBL, ADNI) from mid-life to older adulthood, assessing the influence of age, sex, amyloid (Aβ) and genetic risk for dementia. The data comprised cognitive, genetic and neuroimaging measures of a total of 1570 non-clinical mid-life and older adults (mean age 72, range 49–90 years, 1330 males) and 1365 age- and sex-matched adults with mild cognitive impairment (MCI) or Alzheimer’s disease (AD). Among non-clinical adults, we found robust modes of co-variation between regional SW and multidomain cognitive function that differed between the mid-life and older age range. These cortical and cognitive profiles derived from healthy cohorts predicted out-of-sample AD and MCI. Furthermore, Aβ-deposition and educational attainment levels were associated with cognition but not SW. These findings underscoring the complex interplay between factors influencing cognition and brain structure from mid-life onwards, providing valuable insights for future research into neurodegeneration and the development of future screening algorithms.