During visual search, the spatial configuration of the stimuli can be learned when the same displays are presented repeatedly, thereby guiding attention more efficiently to the target location (contextual cueing effect). This study investigated how the presence of a task-irrelevant object influences the contextual cueing effect. Experiment 1 used a standard T/L search task with “old” display configurations presented repeatedly among “new” displays. A green-filled square appeared at unoccupied locations within the search display. The results showed that the typical contextual cueing effect was strongly reduced when a square was added to the display. In Experiment 2, the contextual cueing effect was reinstated by simply including trials where the square could appear at an occupied location (i.e., underneath the search stimuli). Experiment 3 replicated the previous experiment, showing that the restored contextual cueing effect did not depend on whether the square was actually overlapping with a stimulus or not. The final two experiments introduced a display change in the last epoch. The results showed that the square does not only hinder the acquisition of contextual information but also its manifestation. These findings are discussed in terms of an account where effective contextual learning depends on whether the square is perceived as part of the search display or as part of the display background.