This study randomly recruited 245 college students aged between 18 and 21 years old who had never studied Spanish and selected 60 subjects (30 with high refresh ability and 30 with low refresh ability) based on their scores on a refresh ability test. The mixed experimental design of 2 (refresh ability: high, low) ×3 (background sound: meaningful, meaningless, silent) was adopted. The Eyelink1000 eye tracker was used to record the eye movement data of the subjects when reading the sentences of scientific papers. At the same time, meaningful (news broadcasting) or meaningless (Spanish) background sounds are played. Based on this, the moderating effect of individual refresh ability with the impact of irrelevant speech was investigated. According to the results, the interaction between refresh ability and the meaningful background sound condition was significant, and it had an impact on the totalreading time, gaze count, regression count, total fixation time, and regression-path duration. However, the interaction with the meaningless background sound condition was not significant. The results showed that: (1) refresh ability significantly regulates the irrelevant speech effect in reading, compared with individuals with high refresh ability. The meaningful background sounds have a greater negative impact on the reading of individuals with low refresh ability; (2) the regulating effect of refresh ability occurs in the late stage of vocabulary processing and semantic integration. The results supported the attentional capture theory and found that the irrelevant speech effect in Chinese reading is modulated by individual refresh ability, which mainly acts on semantic components in late reading processing.