Evidence suggests that word recognition depends on numerous talker-, listener-, and stimulus-related characteristics. The current study examined the effects of talker variability and lexical difficulty on spoken-word recognition among four groups of listeners: native listeners with normal hearing or hearing impairment (moderate sensorineural hearing loss) and non-native listeners with normal hearing or hearing impairment. The ability of listeners to accommodate trial-to-trial variations in talkers' voice was assessed by comparing recognition scores for a single-talker condition to those obtained in a multiple-talker condition. Lexical difficulty was assessed by comparing word-recognition performance between lexically "easy" and "hard" words as determined by frequency of occurrence in language and the structural characteristics of similarity neighborhoods formalized in the Neighborhood Activation Model. An up-down adaptive procedure was used to determine the sound pressure level for 50% performance. Non-native listeners in both normal-hearing and hearing-impaired groups required greater intensity for equal intelligibility than the native normal-hearing and hearing-impaired listeners. Results, however, showed significant effects of talker variability and lexical difficulty for the four groups. Structural equation modeling demonstrated that an audibility factor accounts for 2-3 times more variance in performance than does a linguistic-familiarity factor. However, the linguistic-familiarity factor is also essential to the model fit. The results demonstrated effects of talker variability and lexical difficulty on word recognition for both native and nonnative listeners with normal or impaired hearing. The results indicate that linguistic and indexical factors should be considered in the development of speech-recognition tests.