Augmented-reality-based steady-state visual evoked potential (AR-SSVEP) has improved the flexibility and portability of the brain–computer interface (BCI) system. However, in AR-SSVEP, the visual stimuli are displayed in the real world. It remains to be seen whether the color of visual stimuli has the same effect on AR-SSVEP as on computer screen-based steady-state visual evoked potential (PC-SSVEP). To explore the impact of stimulus color on the classification accuracy of augmented-reality-based brain–computer interface (AR-BCI), we designed stimulus interfaces with four colors (white, red, green, and blue). A PC and optical see-through head-mounted display (OST-HMD) device (HoloLens) displayed a similar layout. Experimental results showed that the effect of visual stimulus color on classification in AR-SSVEP and PC-SSVEP was inconsistent. In AR-SSVEP, green visual stimuli performed better when the stimulation duration was less than 1.5 s, while red and white visual stimuli were preferred when the stimulation duration was more than 1.5 s. In PC-SSVEP, red visual stimuli obtained the highest information transmission rate (ITR). Blue was the worst in both types of SSVEP-BCI. The results of this study can guide the color selection of visual stimuli in AR-SSVEP.