Inhibition of return (IOR) refers to the finding that, when the time lag between a cue and a target is prolonged, the reaction to the target, when it eventually appears, is actually slower than with no cue. This phenomenon is thought to make visual search more efficient, and it is subserved by the left inferior parietal cortex and the supramarginal gyrus bilaterally. Interestingly, the very same brain structures are also involved in letter processing. Accordingly, we asked whether the two mental processes interfere with each other when simultaneously probed. The first experiment used a typical IOR procedure, but the cue/target placeholders were either simple geometric shapes or English letters. The results show that, although IOR is approximately the same across visual fields when shape placeholders are used, it is significantly lessened in the right visual field when letters are used as cue and target placeholders. To examine if this finding was due to potential spatial frequency differences between the placeholders, a second experiment using shapes and Japanese letters was conducted, and no differences in IOR were found. The supramarginal gyrus appears to be the most likely locus for the letter-IOR interference effect because it is active bilaterally in IOR, but only in the left hemisphere during letter processing. These findings provide support for the notion that IOR is not simply due to subcortical processes but also involves processing from cortical structures.