Two experiments served to examine how people arrive at stimulus-specific prospective judgments about the distracting effects of speech on cognitive performance. The direct-access account implies that people have direct metacognitive access to the cognitive effects of sounds that determine distraction. The processing-fluency account implies that people rely on the processing-fluency heuristic to predict the distracting effects of sounds on cognitive performance. To test these accounts against each other, we manipulated the processing fluency of speech by playing speech forward or backward and by playing speech in the participants’ native or a foreign language. Forward speech and native speech disrupted serial recall to the same degree as backward speech and foreign speech, respectively. However, the more fluently experienced forward speech and native speech were incorrectly predicted to be less distracting than backward speech and foreign speech. This provides evidence of a metacognitive illusion in stimulus-specific prospective judgments of distraction by speech, supporting the processing-fluency account over the direct-access account. The difference between more and less fluently experienced speech was largely absent in the participants’ global retrospective judgments of distraction, suggesting that people gain access to comparatively valid cues when experiencing the distracting effects of speech on their serial-recall performance firsthand.