Early-life environmental factors have been suggested in the pathophysiology of dementia. Season of birth has previously been used as a proxy for these external exposures. We investigated the link between season of birth and the risk of dementia and further explored underlying pathways by studying structural brain changes on MRI. From the Dutch, population-based Rotterdam Study, 12,964 participants born between 1887 and 1960 were followed between 1990 and 2018 for dementia. Cox regression was conducted to assess the association between season of birth and dementia. In addition, we distinguished between mild and cold winters. The association of season of birth with structural brain markers on MRI was examined in 5237 participants. The risk of dementia in participants born in winter and fall was higher than of those born in summer (hazard ratio (HR) 1.15 [95% confidence interval (CI) 1.01–1.31] for winter and HR 1.17 [95% CI 1.01–1.33] for fall), especially for Alzheimer’s disease (HR 1.23 [1.06–1.43] for winter and HR 1.15 [95% CI 0.99–1.35] for fall). The risk was particularly increased for participants born in a cold winter. Except for slightly lower hippocampus in fall born participants (β − 0.03; 95% CI − 0.06 to 0.00), we did not find associations with brain imaging markers. In conclusion, winter and fall births were associated with a higher incidence of dementia, especially of AD. We did not find evidence for structural brain changes as an underlying mechanism.