Physical colors, i.e. reflected or emitted lights entering the eyes from a visual environment, are converted into perceived colors sensed by humans by neurophysiological mechanisms. These processes involve both three types of photoreceptors, the LMS cones, and spectrally opponent and non-opponent interactions resulting from the activity rates of ganglion and lateral geniculate nucleus cells. Thus, color perception is a phenomenon inherently linked to an experimental environment (the visual scene) and an observing apparatus (the human visual system). This is clearly reminiscent of the conceptual foundation of both relativity and quantum mechanics, where the link is between a physical system and the measuring instruments. The relationship between color perception and relativity was explicitly examined for the first time by the physicist H. Yilmaz in 1962 from an experimental point of view. The main purpose of this contribution is to present a rigorous mathematical model that, by taking into account both trichromacy and color opponency, permits to explain on a purely theoretical basis the relativistic color perception phenomena argued by Yilmaz. Instead of relying directly on relativistic considerations, we base our theory on a quantum interpretation of color perception together with just one assumption, called trichromacy axiom, that summarizes well-established properties of trichromatic color vision within the framework of Jordan algebras. We show how this approach allows us to reconcile trichromacy with Hering’s opponency and also to derive the relativistic properties of perceived colors without any additional mathematical or experimental assumption. In doing so, we also introduce several novel and mathematically rigorous definitions of chromatic attributes and discuss their counterparts in classical colorimetry. Finally, we underline the important role played by the Hilbert metric in our framework and its compatibility with known experimental data.