How and when education improves cognitive capacity is an issue of profound societal importance. Education and later-life education-related factors, such as occupational complexity and engagement in cognitive-intellectual activities, are frequently considered indices of cognitive reserve, but whether their effects are truly causal remains unclear. In this study, after accounting for general cognitive ability (GCA) at an average age of 20 y, additional education, occupational complexity, or engagement in cognitive-intellectual activities accounted for little variance in late midlife cognitive functioning in men age 56-66 (n = 1009). Age 20 GCA accounted for 40% of variance in the same measure in late midlife and approximately 10% of variance in each of seven cognitive domains. The other factors each accounted for <1% of the variance in cognitive outcomes. The impact of these other factors likely reflects reverse causation-namely, downstream effects of early adult GCA. Supporting that idea, age 20 GCA, but not education, was associated with late midlife cortical surface area (n = 367). In our view, the most parsimonious explanation of our results, a meta-analysis of the impact of education, and epidemiologic studies of the Flynn effect is that intellectual capacity gains due to education plateau in late adolescence/early adulthood. Longitudinal studies with multiple cognitive assessments before completion of education would be needed to confirm this speculation. If cognitive gains reach an asymptote by early adulthood, then strengthening cognitive reserve and reducing later-life cognitive decline and dementia risk may really begin with improving educational quality and access in childhood and adolescence.