There is behavioral evidence that different visual categorization tasks on various types of stimuli (e.g., faces) are sensitive to distinct visual characteristics of the same image, for example, spatial frequencies. However, it has been more difficult to address the question of how early in the processing stream this sensitivity to the information relevant to the categorization task emerges. The current study uses scalp event-related potentials recorded in humans to examine how and when information diagnostic to a particular task is processed during that task versus during a task for which it is not diagnostic. Subjects were shown diagnostic and anti-diagnostic face images for both expression and gender decisions (created using Gosselin and Schyns’ Bubbles technique), and asked to perform both tasks on all stimuli. Behaviorally, there was a larger advantage of diagnostic over anti-diagnostic facial images when images designed to be diagnostic for a particular task were shown when performing that task, as compared to performing the other task. Most importantly, this interaction was seen in the amplitude of the occipito-temporal N170, a visual component reflecting a perceptual stage of processing associated with the categorization of faces. When participants performed the gender categorization task, the N170 amplitude was larger when they were presented with gender diagnostic images than with expression-diagnostic images, relative to their respective non-diagnostic stimuli. However, categorizing faces according to their facial expression was not significantly associated with a larger N170 when subjects categorized expression diagnostic cues relative to gender-diagnostic cues. These results show that the influence of higher-level task-oriented processing may take place at the level of visual categorization stages for faces, at least for processes relying on shared diagnostic features with facial identity judgments, such as gender cues.