Cognitive decline and depressive symptoms often co-occur among older adults, and they share several mechanisms. Despite the fact that cognitive dysfunction has been linked to increased depressive symptoms, the directionality of this association remains unclear. To examine whether there is a bidirectional association between depressive symptoms and cognitive function in English adults aged 50 years or older throughout a 16-year follow-up period. This cohort study included a nationally representative sample of community-dwelling English adults aged 50 years or older. The current analysis included 8268 eligible participants with relevant data. These participants were examined every other year from 2002 and 2003 until 2018 and 2019, resulting in a follow-up period of up to 16 years. Data were analyzed from July to November 2023. The bivariate dual change score models were used to estimate the multivariable associations between depressive symptoms and cognitive function, which were interchangeably used as exposures and outcomes. Cognitive measures include memory and verbal fluency tests, while the Center for Epidemiologic Studies Depression Scale evaluated depressive symptoms. The study population of 8268 participants had a mean (SD) age of 64 (10) years at the study baseline, and 4517 participants (55%) were female. Higher depressive symptoms were cross-sectionally associated with poorer memory (β intercept, -0.018; standard error [SE], 0.004; P < .001) and verbal fluency (β intercept, -0.009; SE, 0.004; P = .02) at study baseline. A steeper linear change in depressive symptoms was associated with an accelerated memory change (β intercept, -0.253; SE, 0.079; P = .001), and a linear change in memory was associated with an acceleration in depressive symptoms over time (β intercept, 0.016; SE, 0.006; P = .005). This bidirectional change was not observed with verbal fluency. In this study, greater depressive symptoms were associated with poorer memory at the study baseline and steeper memory change over time. A gradual linear change in depressive symptoms contributed to accelerated memory loss and vice versa, suggesting that psychological mood and memory performance are intrinsically associated.