Aging listeners, even in the absence of overt hearing loss measured as changes in hearing thresholds, often experience impairments processing temporally complex sounds such as speech in noise. Recent evidence has shown that normal aging is accompanied by a progressive loss of synapses between inner hair cells and auditory nerve fibers. The role of this cochlear synaptopathy in degraded temporal processing with age is not yet understood. Here, we used population envelope following responses, along with other hair cell- and neural-based measures from an age-graded series of male and female CBA/CaJ mice to study changes in encoding stimulus envelopes. By comparing responses obtained before and after the application of the neurotoxin ouabain to the inner ear, we demonstrate that we can study changes in temporal processing on either side of the cochlear synapse. Results show that deficits in neural coding with age emerge at the earliest neural stages of auditory processing and are correlated with the degree of cochlear synaptopathy. These changes are seen before losses in neural thresholds and particularly affect the suprathreshold processing of sound. Responses obtained from more central sources show smaller differences with age, suggesting compensatory gain. These results show that progressive cochlear synaptopathy is accompanied by deficits in temporal coding at the earliest neural generators and contribute to the suprathreshold sound processing deficits observed with age.SIGNIFICANCE STATEMENT Aging listeners often experience difficulty hearing and understanding speech in noisy conditions. The results described here suggest that age-related loss of cochlear synapses may be a significant contributor to those performance declines. We observed aberrant neural coding of sounds in the early auditory pathway, which was accompanied by and correlated with an age-progressive loss of synapses between the inner hair cells and the auditory nerve. Deficits first appeared before changes in hearing thresholds and were largest at higher sound levels relevant to real world communication. The noninvasive tests described here may be adapted to detect cochlear synaptopathy in the clinical setting.