Understanding the impact of depression on brain aging benefits the prognosis of this disease and of the risk that other age-related brain disorders will develop in the same population. The aim of the present study was to explore the genetic effect of depression on longitudinal changes in brain structure throughout the lifespan using a Mendelian randomization approach. Summary data from a genome-wide association study of 195,321 to 377,277 participants in the FinnGen consortium were used to predict depression, anxiety disorders, mood disorders, and antidepressant use genetically. Data from 15,640 participants in the ENIGMA consortium were included to predict changes in 15 brain structures throughout the lifespan. The causal relationship between these depressive traits and the brain structure parameters was assessed by two-sample Mendelian randomization (including inverse-variance weighted). Sensitivity analyses were conducted for quality control. Depression slowed the decrease of cortical gray matter volume significantly throughout the lifespan (p = 0.001). Depression, anxiety, and mood disorders nominally decreased the rates of change of volume in the cerebellum gray matter, lateral ventricles, and cortical gray matter throughout the lifespan (p = 0.048, p = 0.021, p = 0.038, respectively). Antidepressants did not affect these rates of change significantly (p > 0.05). Sensitivity analyses confirmed the reliability of this study. Depression and its main symptoms have a slight effect on longitudinal changes in a few brain structures throughout the lifespan at the genetic level. These findings do not support the notion that depression affects macro-aging in the brain crucially.