Perception of color as a task-relevant stimulus can affect cognition and behavior in the flanker task; however, it remains unclear whether it has the same impact when it is a task-irrelevant stimulus dimension. To this end, we applied four-letter flanker tasks with or without colored (red/blue) to 23 healthy young adults, while recording the event-related potentials (ERPs) and behavioral performance. The flanker task included four kinds of color types: non-color letter (NC), all color letter (AC), flanker color letter (FC), and target color letter (TC), each flanker task included congruent and incongruent conditions. The behavioral data demonstrated the classic conflict effect across all color types of flanker tasks in both reaction times (RTs) and accuracy, the significant interaction and main effect of color type factors were only observed in accuracy. The ERP results showed significant interaction between conflict factor (congruent, incongruent) and color type (NC, AC, FC, and TC), and the color type factor enhanced the fronto-central P2 (180–200 ms), descended the fronto-centro-parietal N2b (260–320 ms), and increased the fronto-central P3b (360–520 ms). The fronto-central P2 and the fronto-central P3b were larger for TC than NC, AC, and FC in the congruent condition, while the fronto-central P3b was smaller for NC than AC, FC, and TC in the incongruent condition. Furthermore, the fronto-centro-parietal N2b was decreased successively in NC, AC, FC, and TC in both congruent and incongruent conditions. Overall, our findings suggested that the task-irrelevant stimuli dimension of color can capture some attentional resources and is affected by the location of color (target/flanker) and the type of task trial (congruent/incongruent) in the flanker task.