This study investigates the impact of media on color sense: our ability to see different colors and use them to interpret the world. Specifically, we examine the role of color in the cultural construction of the Orient—an ‘imagined geography’ used to justify colonial domination—in two turn-of-the-twentieth-century types of color(ed) photographs: photochromes, where a printer added color, and autochromes, where colors were captured during exposure. While most research on visual Orientalism has focused on content, we use machine learning methods to study the most important formal element of visual Orientalism: color. After using K-means clustering to extract sixteen dominant colors from each photograph in our dataset, we train three different random forest classification algorithms to make a distinction between (A) the two color media (B) photochromes of the Orient and the Occident; and (C) autochromes of the Orient and the Occident. Subsequently, we apply Shapley Additive Explanations, an explainable AI method, to interpret the output of the classifiers. This allows us to examine how specific features (colors) impacted the classifiers’ predictions. While the algorithm can easily separate photochromes from autochromes (0.95) and Oriental from Occidental photochromes (0.93), it struggles with the same task in the autochrome collection (0.68). These findings support three interconnected conclusions: (1) color sense became mediated in the late nineteenth century, (2) in photochromes, the presence and absence of specific colors was a vital aspect of visual Orientalism, (3) the autochrome, where color was derived from light, provided a more objective picture of countries in the near and middle East than the photochrome.