Visually evoked potentials (VEPs) are widely used for diagnoses of different neurological diseases. Interestingly, there is limited research about the impact of the stimulus color onto the evoked response. Therefore, in our study we investigated the possibility of automatically classifying the stimulus color. The visual stimuli were selected to be red/black and green/black checkerboard patterns with equal light density. Both of these stimuli were presented in a random manner to nine subjects, while the electroencephalogram was recorded at the occipital lobe. After pre-processing and aligning the evoked potentials, an artificial neural network with one hidden layer was used to investigate the general possibility to automatically classify the stimulus color in three different settings. First, color classification with individually trained models, color classification with a common model, and color classification for each individual volunteer with a model trained on the data of the remaining subjects. With an average accuracy (ACC) of 0.83, the best results were achieved for the individually trained model. Also, the second (mean ACC = 0.76) and third experiments (mean ACC = 0.71) indicated a reasonable predictive accuracy across all subjects. Consequently, machine learning tools are able to appropriately classify stimuli colors based on VEPs. Although further studies are needed to improve the classification performance of our approach, this opens new fields of applications for VEPs.