When listening to speech in a background masker, normal‐hearing listeners take advantage of envelope fluctuations or dips in the masker (or the moments that the SNR is instantaneously favorable). Cochlear implant listeners, however, do not exhibit such ability. In a previous study [Nelson et al., J. Acoust. Soc. Am. 113, 961–968 (2003)], speech recognition performance with modulated maskers was similar or slightly worse than with a steady noise. Clinical observation indicates that implant listeners usually have more difficulty understanding speech in modulated backgrounds. In the present study, the recognition of IEEE sentences by Nucleus recipients was measured in a variety of backgrounds. In tightly controlled conditions via direct stimulation, performance is often substantially poorer with modulated backgrounds (the differences in score are as large as 20–30%), strongly indicating that they are subject to modulation interference [B. J. Kwon and C. W. Turner, J. Acoust. Soc. Am. 110, 1130–1140 (2001)]. A mixture of speech and modulated backgrounds might be perceived as one distorted signal, rather than two signals, compromising the identity of the speech, as perceptual segregation of signals is very challenging. A future implant system should incorporate a strategy to provide multiple input sounds without sacrificing the sound identity.