Listeners exhibit varying levels of tolerance for background noise during speech communication. It has been proposed that low tolerance of background noise may be the consequence of abnormally amplified gain in the central auditory system (CAS). Here, using a dataset of young adults with normal hearing thresholds, we asked whether central gain mechanisms might also explain cases of hyper tolerance of background noise, as well as cases of reduced, but not abnormal, tolerance. We used the auditory brainstem response to derive a measure of CAS gain (Wave V/Wave I ratio) to compare listeners' background noise tolerance while listening to speech, grouping them into three categories: hyper, high, and medium tolerance. We found that hyper tolerant listeners had reduced CAS gain than those with high tolerance. This effect was driven by Wave V not Wave I. In addition, the medium tolerant listeners trended towards having reduced Wave I and reduced Wave V amplitudes and generally higher levels of exposure to loud sound, suggestive of the early stages of noise-compromised peripheral function without an apparent compensatory increase in central gain. Our results provide physiologic evidence that (1) reduced CAS gain may account for hyper tolerance of background noise but that (2) increased CAS gain is not a pre-requisite for medium tolerance of background noise.