The effect of unattended task-irrelevant auditory stimuli in the context of an auditory task is not well understood. Using human functional magnetic resonance imaging (fMRI) we compared blood oxygenation level dependent (BOLD) signal changes resulting from monotic task-irrelevant stimulation, monotic task-relevant stimulation and dichotic stimulation with an attended task-relevant stream to one ear and an unattended task-irrelevant stream to the other ear simultaneously. We found strong bilateral BOLD signal changes in the auditory cortex (AC) resulting from monotic stimulation in a passive listening condition. Consistent with previous work, these responses were largest on the side contralateral to stimulation. AC responses to the unattended (task-irrelevant) sounds were preferentially contralateral and strongest for the most difficult condition. Stronger bilateral AC responses occurred during monotic passive-listening than to an unattended stream presented in a dichotic condition, with attention focused on one ear. Additionally, the visual cortex showed negative responses compared to the baseline in all stimulus conditions including passive listening. Our results suggest that during dichotic listening, with attention focused on one ear, (1) the contralateral and the ipsilateral auditory pathways are suppressively interacting; and (2) cross-modal inhibition occurs during purely acoustic stimulation. These findings support the existence of response suppressions within and between modalities in the presence of competing interfering stimuli.