Radiation therapy (RT) is an integral treatment component in patients with glioma but associated with neurotoxicity. Proton RT (PRT), as compared with photon RT (XRT), reduces excess radiation to nontarget tissue. We used a retrospective method to evaluate brain imaging metrics of neurotoxicity after treatment with PRT and XRT for glioma. We analyzed brain volume change in thirty-four patients with WHO grade 2-3 gliomas treated with either PRT (n = 17) or XRT (n = 17). Both groups were carefully matched by demographic/clinical criteria and assessed longitudinally for two years post-radiotherapy. Brain volume change was measured as ventricular volume expansion in the tumor free hemisphere (contralateral to RT target) as a proxy indicator of brain volume loss. We further assessed the impact of volumetric changes on cognition in PRT patients, who completed neuropsychological testing as part of an outcome study. We found significant ventricular volume increases in the contralesional hemisphere in both groups at two years post-RT (F(1, 31) = 18.45, p < 0.000, partial η2 = 0.373), with greater volume change observed in XRT (26.55%) vs. PRT (12.03%) (M = 12.03%, SD = 16.26; F(1,31) = 4.26, p = 0.048, partial η2 = 0.121). Although, there was no group-level change on any cognitive test in PRT treated patients, individual changes on cognitive screening, working memory, processing speed and visual memory tasks correlated with contralesional brain volume loss. This study suggests progressive brain volume loss following cranial irradiation, with greater severity after XRT vs. PRT. Radiation-induced brain volume loss appears to be associated with measurable cognitive changes on an individual level. Prospective studies are warranted to validate these findings and their impacts on long-term cognitive function and quality of life. An improved understanding of the structural and functional consequences of cranial radiation is essential to develop neuroprotective strategies.