This study was to investigate clinical applicability of diffusion spectrum imaging (DSI) for quantitative detection of visual pathway abnormalities to predict the degree of visual field defects (VFD) in patients with pituitary adenomas. Sixty-five patients with pituitary adenomas and 33 healthy controls underwent conventional MRI and DSI scanning that allowed high-angular-resolution fiber tracking. Optic chiasmal compression and VFD were confirmed in all patients via radiological and neuro-ophthalmological examinations. Quantitative assessments of chiasmal lift, VFD, and DSI parameters from the optic nerve, optic tract, and optic radiation were performed. Group comparisons and correlation analyses were conducted in patients and controls. Using the 5-fold cross-validation method, the support vector machine classifiers were constructed to predict the degree of visual defects. The mean values of quantitative anisotropy and generalized fractional anisotropy in optic nerve and optic tract showed significant differences between patients and controls (p < 0.05). These parameters were also significantly correlated with the chiasmal lift distance and degree of visual defects (p < 0.05). All patients were divided into mild (n = 42) and severe (n = 23) VFD groups, using the mean deviation value of -8 dB as the threshold. The classifiers achieved an accuracy of 0.83, sensitivity of 0.78, and specificity of 0.86 to discriminate patients with mild and severe visual defects. Using high-angular-resolution fiber tracking, DSI may provide quantitative information to detect visual pathway abnormalities and be a potential diagnostic tool for determining the degree of visual field defects in pituitary adenomas. • Abnormal QA and GFA values of optic nerve and optic tract in adenoma patients • Close relationship between DSI parameters and VFD degree in adenoma patients • The classifiers achieved an accuracy of 0.83, sensitivity of 0.78, and specificity of 0.86 to discriminate patients with mild and severe VFD.