This study evaluated longitudinal brain iron accumulation in older adults, its association with cognition, and the role of specific nutrients in mitigating iron accumulation. MRI-based, quantitative susceptibility mapping estimates of brain iron concentration were acquired from seventy-two healthy older adults (47 women, ages 60–86) at a baseline timepoint (TP1) and a follow-up timepoint (TP2) 2.5–3.0 years later. Dietary intake was evaluated at baseline using a validated questionnaire. Cognitive performance was assessed at TP2 using the uniform data set (Version 3) neuropsychological tests of episodic memory (MEM) and executive function (EF). Voxel-wise, linear mixed-effects models, adjusted for longitudinal gray matter volume alterations, age, and several non-dietary lifestyle factors revealed brain iron accumulation in multiple subcortical and cortical brain regions, which was negatively associated with both MEM and EF performance at T2. However, consumption of specific dietary nutrients at TP1 was associated with reduced brain iron accumulation. Our study provides a map of brain regions showing iron accumulation in older adults over a short 2.5-year follow-up and indicates that certain dietary nutrients may slow brain iron accumulation.