During the COVID-19 pandemic, there has been a growing prevalence of mental disorders, particularly in China, where anti-epidemic measures have been more stringent compared to other countries. This has resulted in a heightened psychological burden on individuals. However, the actual psychological impacts of COVID-19 on Chinese residents have not been clearly established. In this study, we aimed to investigate the effects of various factors, including personal and household characteristics, social networks, and epidemic-related variables (such as rural residents' knowledge of COVID-19 virus transmission and prevention), on rural residents' depression. Using a two-year balanced panel dataset encompassing 152 villages and 676 samples in both 2019 and 2020, we employed the Ordinary Least Squares (OLS) model to estimate the influence of these factors and compare their coefficients to examine the changes in rural residents' depression between the pre-epidemic and during-epidemic periods. Our study highlights significant factors contributing to rural residents' depression during the COVID-19 pandemic, including isolation (2.063, p < 0.01) and concern about COVID-19 infection (0.128, p < 0.05). Conversely, factors such as health status (−0.714, p < 0.01) and the number of friends (−0.017, p < 0.01) were found to significantly alleviate depression among rural residents. Moreover, we identify key moderators that mitigate the negative impact of infection concerns on mental health. Specifically, the health status of rural residents (−0.166, p < 0.01), the size of their family network (−0.036, p < 0.05), and their knowledge of COVID-19 transmission and prevention (−0.184, p < 0.05) significantly moderated the relationship between infection concerns and depression. These findings contribute to a better understanding of the psychological implications of the pandemic and provide valuable insights for the development of targeted interventions to address mental health challenges in rural populations.