There is a surprisingly strong effect on color appearance when low levels of luminance contrast are added to visual targets in which only S-cones are modulated. This phenomenon can be studied with checkerboard patterns composed of alternating S-cone-modulated checks and gray checks. + S checks look purple when surrounded by slightly brighter gray checks but look highly desaturated (lavender, almost white) when surrounded by darker gray checks. −S checks change in hue with luminance contrast; they look yellow when surrounded by darker gray checks but are greener when surrounded by lighter checks. Psychophysical paired comparisons confirm these perceptions. Furthermore, visual evoked potentials (VEPs) recorded from human posterior cortex indicate that signals evoked by low luminance contrast interact nonlinearly with S-cone-evoked signals in early cortical color processing. Our new psychophysics and electrophysiology results prove that human perception of color appearance is not based on neural computations within a separate, isolated color system. Rather, signals evoked by color contrast and luminance contrast interact to produce the colors we see.