Although we perceive a spatially contiguous, integrated world, sensory systems sample information discretely via small spatial and narrow frequency receptive fields (RFs) in vision and audition, respectively. Undersampling of stimuli by narrow RFs produces ambiguous representations (e.g., the classic “aperture problem” that visual motion cannot be uniquely determined by one RF). Here we posit a similar problem for encoding interaural time differences (ITDs). Current models of ITD processing propose large ensembles of ITD-sensitive neurons spanning wide ranges of preferred ITD and characteristic frequency (CF). To account for anomalous lateralization of narrowband but robust lateralization of broadband stimuli, such models require two paradoxical assumptions that have recently come into question: (1) the existence of neurons preferring larger than physiologically possible ITDs; and (2) population biases overemphasizing small ITDs. Here, we consider an alternative account based on a highly constrained ITDxCF aperture through which the stimuli are perceived. ITDs bounded by ±π interaural phase difference and CFs by dominance weighting, where ∼600-750 Hz contributes most to lateralization. A more detailed consideration of this “binaural aperture” sheds light on how ambiguous lateralization data emerge from limited ensembles of neurons constrained to physiologically relevant ranges of ITD and frequency. [R01-DC011555 (DJT), R01-DC014948 (MJG), and R01-DC016643 (GCS).]