Cross-sectional studies have suggested that corpus callosum (CC) atrophy is related to impairment in global cognitive function, mental speed, and executive functions in the elderly. Longitudinal studies confirming these findings have been lacking. We investigated whether CC tissue loss is associated with change in cognitive performance over time in subjects with age-related white matter lesions (WML). Two-hundred-fifty-three subjects, aged 65–84 years, were evaluated by using repeated MRI and neuropsychological evaluation at baseline and after 3 years. The effect of overall and regional CC tissue loss on cognitive decline was analyzed with hierarchical linear regression models. After controlling for age, sex, education, and baseline cognitive performance, the rates of tissue loss in the total CC area, and in rostrum/genu and midbody subregions were significantly associated with decline in a compound measure of cognitive speed and motor control, but not in those of executive functions, memory, or global cognitive function. Total CC area and midbody remained significant predictors of speed also after adjusting for baseline WML volume, WML progression, and global brain atrophy. However, the relationship between anterior CC and speed performance was mediated by WML volume. In conclusion, the overall and regional rate of CC tissue loss parallels longitudinal slowing of psychomotor performance. The adverse effect of CC tissue loss on psychomotor function may be driven by altered interhemispheric information transfer between homologous cortical areas.