There has been much research into the mental health impact of the Coronavirus Disease 2019 (COVID-19) pandemic and how it is related to time-invariant individual characteristics. However, there is still a lack of research showing long-term trajectories of mental health across different stages of the pandemic. And little is known regarding the longitudinal association of time-varying factors with mental health outcomes. This study aimed to provide a longitudinal profile of how mental health in adults changed across different stages of the COVID-19 pandemic and to examine their longitudinal associations with time-varying contextual (e.g., COVID-19 policy response and pandemic intensity) and individual level factors. This study used data from a large panel study of over 57,000 adults living in England, who were followed up regularly for 2 years between March 2020 and April 2022. Mental health outcomes were depressive and anxiety symptoms. Depressive symptoms were assessed by the Patient Health Questionnaire (PHQ-9) and anxiety symptoms by the Generalized Anxiety Disorder assessment (GAD-7). Entropy balancing weights were applied to restore sample representativeness. After weighting, approximately 50% of participants were female, 14% from ethnic minority backgrounds, with a mean age of 48 years. Descriptive analyses showed that mental health changes were largely in line with changes in COVID-19 policy response and pandemic intensity. Further, data were analysed using fixed-effects (FE) models, which controlled for all time-invariant confounders (observed or not). FE models were fitted separately across 3 stages of the COVID-19 pandemic, including the first national lockdown (21/03/2020-23/08/2020), second and third national lockdowns (21/09/2020-11/04/2021), and "freedom" period (12/04/2021-14/11/2021). We found that more stringent policy response (measured by stringency index) was associated with increased depressive symptoms, in particular, during lockdown periods (β = 0.23, 95% confidence interval (CI) = [0.18 to 0.28], p < 0.001; β = 0.30, 95% CI = [0.21 to 0.39], p < 0.001; β = 0.04, 95% CI = [-0.03 to 0.12], p = 0.262). Higher COVID-19 deaths were also associated with increased depressive symptoms, but this association weakened over time (β = 0.29, 95% CI = [0.25 to 0.32], p < 0.001; β = 0.09, 95% CI = [0.05 to 0.13], p < 0.001; β = -0.06, 95% CI = [-0.30 to 0.19], p = 0.655). Similar results were also found for anxiety symptoms, for example, stringency index (β = 0.17, 95% CI = [0.12 to 0.21], p < 0.001; β = 0.13, 95% CI = [0.06 to 0.21], p = 0.001; β = 0.10, 95% CI = [0.03 to 0.17], p = 0.005), COVID-19 deaths (β = 0.07, 95% CI = [0.04 to 0.10], p < 0.001; β = 0.04, 95% CI = [0.00 to 0.07], p = 0.03; β = 0.16, 95% CI = [-0.08 to 0.39], p = 0.192). Finally, there was also evidence for the longitudinal association of mental health with individual level factors, including confidence in government/healthcare/essentials, COVID-19 knowledge, COVID-19 stress, COVID-19 infection, and social support. However, it is worth noting that the magnitudes of these longitudinal associations were generally small. The main limitation of the study was its non-probability sample design. Our results provided empirical evidence on how changes in contextual and individual level factors were related to changes in depressive and anxiety symptoms. While some factors (e.g., confidence in healthcare, social support) clearly acted as consistent predictors of depressive and/or anxiety symptoms, other factors (e.g., stringency index, COVID-19 knowledge) were dependent on the specific situations occurring within society. This could provide important implications for policy making and for a better understanding of mental health of the general public during a national or global health crisis.