Low-grade inflammation (LGI) mainly acted as the mediator of the association of obesity and inflammatory diet with numerous chronic diseases, including neuropsychiatric diseases. However, the evidence about the effect of LGI on brain structure is limited but important, especially in the context of accelerating aging. This study was then designed to close the gap, and we leveraged a total of 37,699 participants from the UK Biobank and utilized inflammation score (INFLA-score) to measure LGI. We built the longitudinal relationships of INFLA-score with brain imaging phenotypes using multiple linear regression models. We further analyzed the interactive effects of specific covariates. The results showed high level inflammation reduced the volumes of the subcortex and cortex, especially the globus pallidus (β [95% confidence interval] = -0.062 [-0.083, -0.041]), thalamus (-0.053 [-0.073, -0.033]), insula (-0.052 [-0.072, -0.032]), superior temporal gyrus (-0.049 [-0.069, -0.028]), lateral orbitofrontal cortex (-0.047 [-0.068, -0.027]), and others. Most significant effects were observed among urban residents. Furthermore, males and individuals with physical frailty were susceptive to the associations. The study provided potential insights into pathological changes during disease progression and might aid in the development of preventive and control targets in an age-friendly city to promote great health and well-being for sustainable development goals.