Single-sided deafness (SSD) induces cortical neural plastic changes according to duration of deafness. However, it is still unclear how the auditory cortical changes accompany the subcortical neural changes. The present study aimed to find the neural plastic changes in the cortical and subcortical auditory system following adult-onset single-sided deafness (SSD) using Mn-enhanced magnetic resonance imaging (MEMRI). B57BL/6 mice (postnatal 8-week-old) were divided into three groups: the SSD-4-week group (postnatal 12-week-old, n = 11), the SSD-8-week group (postnatal 16-week-old, n = 11), and a normal-hearing control group (postnatal 8-week-old, n = 9). The left cochlea was ablated in the SSD groups. White Gaussian noise was delivered for 24 h before MEMRI acquisition. T1-weighted MRI data were analyzed from the cochlear nucleus (CN), superior olivary complex (SOC), lateral lemniscus (LL), inferior colliculus (IC), medial geniculate body (MG), and auditory cortex (AC). The differences in relative Mn2+-enhanced signal intensities (Mn2+SI) and laterality were analyzed between the groups. Four weeks after the SSD procedure, the ipsilateral side of the SSD showed significantly lower Mn2+SI in the CN than the control group. On the other hand, the contralateral side of the SSD demonstrated significantly lower Mn2+SI in the SOC, LL, and IC. These decreased Mn2+SI values were partially recovered at 8 weeks after the SSD procedure. The interaural Mn2+SI differences representing the interaural dominance were highest in CN and then became less prominently higher in the auditory neural system. The SSD-8-week group still showed interaural differences in the CN, LL, and IC. In contrast, the MG and AC did not show any significant intergroup or interaural differences in Mn2+SI. In conclusion, subcortical auditory neural activities were decreased after SSD, and the interaural differences were diluted in the higher auditory nervous system. These findings were attenuated with time. Subcortical auditory neural changes after SSD may contribute to the change in tinnitus severity and the outcomes of cochlear implantation in SSD patients.