Fatigue is one of the most common and disabling nonmotor symptoms seen in Parkinson's disease (PD) and is also commonly seen in healthy older adults. Our understanding of the etiology of fatigue in older adults with or without PD is limited and it remains unclear whether fatigue in PD is specifically related to PD pathology. The objective of this study was thus to determine whether fatigue in PD was associated with structural changes in gray or white matter and explore whether these changes were similar in older adults without PD. Magnetic resonance imaging (T1 weighted) and diffusion tensor imaging were performed in 60 patients with PD (17 females; age = 67.58 ± 5.51; disease duration = 5.67 ± 5.83 years) and 41 age- and sex- matched healthy controls. FSL image processing was used to measure gray matter volume, fractional anisotropy, and leukoariosis differences. Voxel-based morphometry confirmed gray matter loss across the dorsal striatum and insula in the PD patient cohort. PD patients with fatigue had reduced gray matter volume in dorsal striatum relative to PD patients without fatigue (P < 0.05 False Discovery Rate corrected). No significant fatigue-related structural atrophy was found in controls. There were no areas of significant fractional anisotropy differences between high and low fatigue subjects in either the PD or non-PD groups. Control participants with high fatigue, but not PD, showed significantly greater total leukoariosis volumes (p = 0.03). Fatigue in PD is associated with unique structural changes in the caudate and putamen suggesting fatigue in PD is primarily related to PD pathology, particularly in the dorsal striatum, and not simply a consequence of aging.