Age-related hearing loss (ARHL) is a common, increasing problem for older adults, affecting about 1 billion people by 2050. We aim to correlate the different reductions of hearing from cochlear hair cells (HCs), spiral ganglion neurons (SGNs), cochlear nuclei (CN), and superior olivary complex (SOC) with the analysis of various reasons for each one on the sensory deficit profiles. Outer HCs show a progressive loss in a basal-to-apical gradient, and inner HCs show a loss in a apex-to-base progression that results in ARHL at high frequencies after 70 years of age. In early neonates, SGNs innervation of cochlear HCs is maintained. Loss of SGNs results in a considerable decrease (~50% or more) of cochlear nuclei in neonates, though the loss is milder in older mice and humans. The dorsal cochlear nuclei (fusiform neurons) project directly to the inferior colliculi while most anterior cochlear nuclei reach the SOC. Reducing the number of neurons in the medial nucleus of the trapezoid body (MNTB) affects the interactions with the lateral superior olive to fine-tune ipsi- and contralateral projections that may remain normal in mice, possibly humans. The inferior colliculi receive direct cochlear fibers and second-order fibers from the superior olivary complex. Loss of the second-order fibers leads to hearing loss in mice and humans. Although ARHL may arise from many complex causes, HC degeneration remains the more significant problem of hearing restoration that would replace the cochlear implant. The review presents recent findings of older humans and mice with hearing loss.