Strict social isolation and physical distancing measures implemented during the COVID-19 pandemic had significant implications for the psychological well-being of middle-aged and older adults. This study aimed to investigate factors associated with depressive symptoms during the pandemic period among individuals who reported no significant depressive symptoms before the pandemic. Individuals from the Korean Longitudinal Study of Aging without a previous report of significant depressive symptoms across Waves 6 (2016) and 7 (2018) were investigated for the development of depressive symptoms in Wave 8 (2020). The multivariable logistic regression results revealed that both men and women who participated in social gatherings and physical exercise less than once a week were associated with an increase in the likelihood of depressive symptoms (odds ratio [OR] 2.88; 95% confidence interval [CI] 1.80–4.61 and OR 2.61; 95% CI 1.64–4.15, respectively for men and OR 2.58; 95% CI 1.80–3.70 and OR 1.51; 95% CI 1.02–2.23, respectively for women). In addition, unmarried men (OR 2.38; 95% CI 1.37–4.14) and women with one chronic disease (OR 1.98; 95% CI 1.14–3.43) or two or more chronic diseases (OR 2.28; 95% CI 1.31–3.99) reported a significant increase in the likelihood of depressive symptoms. Regular social gatherings and physical exercise were identified as key factors in mitigating depressive symptoms among middle-aged and older adults. The findings can inform the development of public health strategies that promote regular social interactions and physical activity to enhance the psychological resilience and overall well-being of middle-aged and older adults in the endemic era.