Asymptomatic low-grade carotid artery stenosis (LGCS) is a common finding in patients with manifest arterial disease, however its relationship with brain MRI changes and cognitive decline is unclear. We included 902 patients (58 ± 10 years; 81% male) enrolled in the Second Manifestations of Arterial Disease – Magnetic Resonance (SMART-MR) study without a history of cerebrovascular disease. LGCS was defined as 1–49% stenosis on baseline carotid ultrasound, whereas no LGCS (reference category) was defined as absence of carotid plaque. Brain and white matter hyperintensity (WMH) volumes and cognitive function were measured at baseline and after 4 (n = 480) and 12 years (n = 222) of follow-up. Using linear mixed-effects models, we investigated associations of LGCS with progression of brain atrophy, WMH, and cognitive decline. LGCS was associated with greater progression of global brain atrophy (estimate −0.03; 95%CI, −0.06 to −0.01; p = 0.002), and a greater decline in executive functioning (estimate −0.02; 95%CI, −0.031 to −0.01; p < 0.001) and memory (estimate −0.012; 95%CI, −0.02 to −0.001; p = 0.032), independent of demographics, cardiovascular risk factors, and incident brain infarcts on MRI. No association was observed between LGCS and progression of WMH. Our results indicate that LGCS may represent an early marker of greater future brain atrophy and cognitive decline.