Listeners can use lexical information to drive adaptation to talker-specific speech characteristics with a small amount of exposure. This suggests that adaptation may be strongest when listeners are initially exposed to speech in a condition that facilitates word recognition. This was tested by examining the intelligibility of second-language (L2) speech-in-noise when stimuli were initially presented in Quiet versus low signal-to-noise ratio (SNR) conditions. Recordings of 10 L2 English talkers were tested. For each talker, 120 sentences were mixed with speech-shaped noise ranging from −4 dB to 8 dB in steps of 2 dB and in Quiet (15 sentences at each SNR). These stimuli were presented from lowest to highest SNR (forward; −4 dB to Quiet) in one group of L1 English listeners (n = 10/talker) and from highest to lowest SNR (reverse; Quiet to −4 dB) in the other (n = 10/talker). The results showed that early exposure to a clean signal led to better speech perception. At lower SNRs, word recognition accuracy in the reverse order was significantly higher than the forward order, resulting in a steeper slope of psychometric function of intelligibility in the forward versus reverse condition. The quality of initial speech input may influence talker adaptation, particularly in adverse listening conditions.