Omega-3 intake has been positively associated with healthy brain aging, yet it remains unclear whether high omega-3 intake beginning early in life may optimize its protective effects against brain aging. We examined whether omega-3 intake is associated with brain microstructure over 2 decades later among dementia-free older adults. The 128 participants (62% women; age at magnetic resonance imaging: 76.6 ± 7.9) from the Rancho Bernardo Study of Healthy Aging completed at least 1 dietary assessment between 1984 and 1996 and underwent restriction spectrum imaging (RSI) 22.8 ± 3.1 years later. We evaluated associations between prior omega-3 intake and RSI metrics of gray and white matter (WM) microstructure. Higher prior omega-3 intake was associated with greater restricted diffusion in the superior cortico-striatal fasciculus. A correlation between higher prior omega-3 intake and greater cingulum restricted diffusion was stronger among participants >80 years old. Higher omega-3 intake correlated with greater restricted diffusion in the inferior longitudinal and inferior fronto-occipital fasciculus more strongly for apolipoprotein E (APOE) ε4 carriers than noncarriers. Associations were not modified by adjustment for dietary pattern, health, or lifestyle. High omega-3 intake in midlife may help to maintain WM integrity into older age, particularly in the latest decades of life and among APOE ε4 carriers.