Physically salient stimuli are difficult to ignore, frequently eliciting fixations even when they are known to be task-irrelevant. A recent study demonstrated that distractor fixation-contingent auditory feedback was highly effective in reducing the frequency of fixations on such stimuli. The present study explores more specifically what it is about feedback that makes it effective in curbing oculomotor behavior. In one experiment, we removed the immediacy of the feedback by informing participants after each trial via textual feedback if they had fixated the distractor. A comparable reduction in the frequency of oculomotor capture was observed. In a second experiment, we only provided summary feedback concerning the frequency of oculomotor capture after each block of trials. Not only were the benefits of feedback again robustly comparable, but a benefit was observed even in the first block before any feedback had actually been presented. Simply knowing that the frequency of distractor fixations was being monitored was sufficient to substantially reduce the frequency of oculomotor capture. Interestingly, trial-level feedback predominantly reduced the frequency of capture by slowing oculomotor responses, reflecting a speed-accuracy tradeoff, whereas block-wise feedback resulted in a reduction in the frequency of capture with saccadic reaction time equated, reflecting a bona fide improvement in task performance. Our findings have implications for our understanding of the role of motivation, strategy, and selection history in oculomotor control.