Using task-evoked pupillary response (TEPR) and dual-task procedures, prior research has shown that adults recruit additional cognitive resources when processing fully intelligible L2-accented speech compared to speech in their own L1 accent (McLaughlin and Van Engen, 2020; Brown et al., 2020). In this study, we investigated the relationship between three prosodic measures (relative word duration variance, pitch stability, and pitch range) and changes in listening effort for L1 and L2 speakers over the course of an experiment. Using growth curve analysis, we found significant interactions between each of the three measures and trial number, indicating that prosodic factors influence speech adaptation. However, these interactions were in opposite directions for the L1 and L2 speakers. Thus, adult monolingual English listeners rely on prosodic information to adapt to a particular speaker’s voice, but use that information differently based on speaker identity.