Cognitive impairment is known to occur in patients with prolactinoma, but the underlying mechanism is unclear. To evaluate cognitive function in patients with prolactinoma and to investigate the basis of possible cognitive impairment in brain white matter changes using diffusion tensor imaging (DTI). 37 consecutive patients with prolactinoma and 37 healthy controls of similar age, sex, and education were enrolled in the study. Hormone levels were determined in all participants, comprehensive neuropsychological testing was performed, and DTI was used to reconstruct and evaluate white matter tracts. In patients with prolactinoma, short- and long-term visual and verbal memory, attention, concentration, and executive and language functions were impaired compared to the healthy group. When comparing the DTI results, lower fractional anisotropy (FA) values were found in the patients' right uncinate fasciculus (R-UF), indicating neuronal damage. After applying the Bonferroni correction, the two groups had no significant difference in 42 tracts (p > 0.0012 for all). A positive correlation was found between poor FA scores on the R-UF and low scores on long-term memory, category and letter fluency tests. In addition, patients with hypoprolactinemia had the worst short-term memory scores, while normoprolactinemia had the best scores. Also, the poorer R-UF FA values were found in the patients with hypoprolactinemia and the highest in those with normoprolactinemia. This study is the first to investigate reasons for cognitive dysfunction in patients with prolactinoma by DTI. No significant structural changes were found in brain tracts of patients with prolactinoma. Still, there may be a link between potential damage in the R-UF and cognitive dysfunction, and further research is needed. In addition, the results showed that the development of hypoprolactinemia is associated with cognitive dysfunction and emphasized that overtreatment should be avoided.