There has been much debate on whether color categories affect how we perceive color. Recent theories have put emphasis on the role of top-down influence on color perception that the original continuous color space in the visual cortex may be transformed into categorical encoding due to top-down modulation. To test the influence of color categories on color perception, we adopted an RSVP paradigm, where color stimuli were presented at a fast speed of 100 ms per stimulus and were forward and backward masked by the preceding and following stimuli. Moreover, no explicit color naming or categorization was required. In theory, backward masking with such a short interval in a passive viewing task should constrain top-down influence from higher-level brain areas. To measure any potentially subtle differences in brain response elicited by different color categories, we embedded a sensitive frequency-tagging-based EEG paradigm within the RSVP stimuli stream where the oddball color stimuli were encoded with a different frequency from the base color stimuli. We showed that EEG responses to cross-category oddball colors at the frequency where the oddball stimuli were presented was significantly larger than the responses to within-category oddball colors. Our study suggested that the visual cortex can automatically and implicitly encode color categories when color stimuli are presented rapidly.