Diabetes frequently results in cognitive impairment, but it is less clear if brain health is adversely affected during the prediabetic stage. Our aim is to identify possible changes in brain volume as measured by magnetic resonance imaging (MRI) in a large elderly population stratified according to level of "dysglycemia." This is a cross-sectional study of 2144 participants (median age 69years, 60.9% female) who underwent 3-T brain MRI. Participants were divided into 4 dysglycemia groups based on HbA1c levels (%): normal glucose metabolism (NGM) (< 5.7%), prediabetes (5.7 to < 6.5%), undiagnosed diabetes (6.5% or higher), and known diabetes (defined by self-report). Of the 2144 participants, 982 had NGM, 845 prediabetes, 61 undiagnosed diabetes, and 256 known diabetes. After adjustment for age, sex, education, body weight, cognitive status, smoking, drinking, and disease history, total gray matter volume was significantly lower among participants with prediabetes (0.41% lower, standardized β = - 0.0021 [95% CI - 0.0039, - 0.00039], p = 0.016), undiagnosed diabetes (1.4% lower, standardized β = - 0.0069 [95% CI - 0.012, - 0.002], p = 0.005), and known diabetes (1.1% lower, standardized β = - 0.0055 [95% CI - 0.0081, - 0.0029], p < 0.001) compared to the NGM group. After adjustment, total white matter volume and hippocampal volume did not differ significantly between the NGM group and either the prediabetes group or the diabetes group. Sustained hyperglycemia may have deleterious effects on gray matter integrity even prior to the onset of clinical diabetes. • Sustained hyperglycemia has deleterious effects on gray matter integrity even prior to the onset of clinical diabetes.