Auditory neuropathy (AN), a complex hearing disorder, presents challenges in diagnosis and management due to limitations of current diagnostic assessment. This study aims to determine whether diffusion-weighted magnetic resonance imaging (MRI) can be used to identify the site and severity of lesions in individuals with AN. This case-control study included 10 individuals with AN of different etiologies, 7 individuals with neurofibromatosis type 1 (NF1), 5 individuals with cochlear hearing loss, and 37 control participants. Participants were recruited through the University of Melbourne's Neuroaudiology Clinic and the Murdoch Children's Research Institute specialist outpatient clinics. Diffusion-weighted MRI data were collected for all participants and the auditory pathways were evaluated using the fixel-based analysis metric of apparent fiber density. Data on each participant's auditory function were also collected including hearing thresholds, otoacoustic emissions, auditory evoked potentials, and speech-in-noise perceptual ability. Analysis of diffusion-weighted MRI showed abnormal white matter fiber density in distinct locations within the auditory system depending on etiology. Compared with controls, individuals with AN due to perinatal oxygen deprivation showed no white matter abnormalities ( p > 0.05), those with a neurodegenerative conditions known/predicted to cause VIII cranial nerve axonopathy showed significantly lower white matter fiber density in the vestibulocochlear nerve ( p < 0.001), while participants with NF1 showed lower white matter fiber density in the auditory brainstem tracts ( p = 0.003). In addition, auditory behavioral measures of speech perception in noise and gap detection were correlated with fiber density results of the VIII nerve. Diffusion-weighted MRI reveals different patterns of anatomical abnormality within the auditory system depending on etiology. This technique has the potential to guide management recommendations for individuals with peripheral and central auditory pathway abnormality.