Traumatic brain injury (TBI), a risk factor for later-life dementia, leads to salient brain atrophy, particularly in the white matter. It is not clear how white matter atrophy progresses or why some brain regions are damaged while others are spared. We hypothesized that spatial variations of cell-specific gene expression contributed to the selective white matter loss vulnerability following mild TBI (mTBI). Gene expression data were sourced from the publicly available Allen Human Brain Atlas, which comprises microarray data spanning nearly the entire brain, derived from six neurologically normal adult donors. A total of 100 patients with acute stage (within 7 days post-injury) mTBI were enrolled. Of these, 60 patients were followed up at 3 months post-injury and 37 were followed up at 6-12 months post-injury. In addition, 59 healthy controls (HCs), matched for age, gender, and education, were included for comparative analysis. White matter volume changes were analyzed at both the acute stage, 3 months, and 6-12 months follow-up in mTBI patients compared with HCs. Patients with mTBI exhibited significant white matter atrophy in the frontal, parietal, and temporal cortices at 3 months post-injury, which even persisted at 6-12 months follow-up. In addition, mTBI patients with cognitive deficits showed more severe brain atrophy compared with those without cognitive deficits. Crucially, the gene expression marking endothelial cells and S1 pyramidal neurons were associated with increased brain atrophy, whereas the gene expression marking microglia and CA1 pyramidal neurons were associated with decreased brain atrophy in mTBI patients at 3 months post-injury. Microglia and endothelial cells can explain 23.6% of regional variations in the white matter atrophy. These findings suggested that modulating cellular activation, especially by promoting microglial activation at 3 months post-injury, might be a promising approach to prevent white matter atrophy, enhance cognitive outcomes, and reduce the risk of later-life dementia.