The present study examines whether listeners flexibly adapt to unfamiliar speech patterns such as those encountered in foreign-accented English vowels, where the relative informativeness of primary (spectral quality) and secondary (duration) cues tends to be reversed (e.g., spectrally similar but exaggerated duration differences between bet and bat). This study further tests whether listeners’ adaptive strategies are related to individual differences in phoneme categorization gradiency and cognitive abilities. Native English listeners (N = 36) listened to a continuum of vowels from /ɛ/ to /æ/ (as in head and had) varying in spectral and duration values to complete a perceptual adaptation task and a visual analog scaling (VAS) task. Participants also completed cognitive tasks examining executive function capacities. Results showed that listeners mostly used spectral quality to signal vowel category at baseline, but flexibly adapted by up-weighting reliance on duration when spectral quality became no longer diagnostic. In the VAS task, some listeners made more categorical responses while others made more gradient responses in vowel categorization, but these differences were not linked to their adaptive patterns. Results of cognitive tasks revealed that individual differences in inhibitory control correlated, to some degree, with the amount of adaptation. Together, these findings suggest that listeners flexibly adapt to unfamiliar speech categories using distributional information in the input and individual differences in cognitive abilities may influence their adaptability.