ABSTRACT Objective This study aims to examine the effect of the Mediterranean diet (MeDi) on cognitive decline among the Chinese elderly with a 3-year follow-up. Methods This study is divided into two waves: wave-1 January 2019 to June 2019 (n = 2313); wave-2 January 2022 to March 2022 (n = 1648). MeDi scores were calculated from the Mediterranean Diet Adherence Screener (MEDAS), with the scoring of low compliance (0–6 points) and high compliance (7–14 points). The Mini-Mental State Examination (MMSE) was used to assess cognitive function. An MMSE score dropping ≥ 2 points from baseline was defined as cognitive decline. The relationships between MeDi score and cognitive decline were analyzed by linear regression models or Binary logistic regression. Results During the 3-year follow-up, 23.8% of patients exhibited cognitive decline. The study revealed a significant difference in MMSE score changes between low and high MeDi adherence groups (p < 0.001). MeDi score was negatively correlated with cognitive deterioration (β = −0.020, p = 0.026). MeDi score was only negatively associated with cognitive decline in the female subgroup aged ≥65 years (β = −0.034, p = 0.033). The food beans (OR = 0.65, 95%CI:0.51, 0.84), fish (OR = 0.72, 95%CI:0.54, 0.97), and cooked vegetables (OR = 0.68, 95%CI:0.53, 0.84) were protective factors for cognitive decline. Conclusions This study suggests that greater adherence to the MeDi is linked to a reduced risk of cognitive decline in elderly people. However, this is found only in women who are 65 years old or older. It also found long-term adherence to beans, fish, and vegetables are more effective in improving cognitive function.