Pure-tone audiograms often poorly predict elderly humans' ability to communicate in everyday complex acoustic scenes. Binaural processing is crucial for discriminating sound sources in such complex acoustic scenes. The compromised perception of communication signals presented above hearing threshold has been linked to both peripheral and central age-related changes in the auditory system. Investigating young and old Mongolian gerbils of both sexes, an established model for human hearing, we demonstrate age-related supra-threshold deficits in binaural hearing using behavioral, electrophysiological, anatomical, and imaging methods. Binaural processing ability was measured as the binaural masking level difference (BMLD), an established measure in human psychophysics. We tested gerbils behaviorally with "virtual headphones," recorded single-unit responses in the auditory midbrain and evaluated gross midbrain and cortical responses using positron emission tomography (PET) imaging. Furthermore, we obtained additional measures of auditory function based on auditory brainstem responses, auditory-nerve synapse counts, and evidence for central inhibitory processing revealed by PET. BMLD deteriorates already in middle-aged animals having normal audiometric thresholds and is even worse in old animals with hearing loss. The magnitude of auditory brainstem response measures related to auditory-nerve function and binaural processing in the auditory brainstem also deteriorate. Furthermore, central GABAergic inhibition is affected by age. Because the number of synapses in the apical turn of the inner ear was not reduced in middle-aged animals, we conclude that peripheral synaptopathy contributes little to binaural processing deficits. Exploratory analyses suggest increased hearing thresholds, altered binaural processing in the brainstem and changed central GABAergic inhibition as potential contributors.