Depression is a relevant mental illness affecting hundreds of millions of people worldwide. As urbanization accelerates, agglomeration of populations has altered individual social network distances and life crowding, which in turn affects depressive prevalence. However, the association between depression and population agglomeration (PA) remains controversial. This study aims to explore whether and how PA could influence individual depression. Based on the China Health and Retirement Longitudinal Study (CHARLS) 2018, the empirical results showed that there was a U-shaped association between PA and individual CES-D scores. As PA increases, the risk of depression first decreases and then increases. CES-D was lowest at moderate aggregation. Dialect diversity (DD) was positively related to the incidence of individual depression. The higher the DD, the higher the risk of depression. Meanwhile, DD also played a moderating role in the association between PA and individual depression. Our observations suggest that the optimistic level of agglomeration for individual mental health is within 1500 to 2000 persons per square kilometer.