Multiple Sclerosis (MS) results in color vision impairment regardless of optic neuritis (ON). The exact location of injury remains undefined. The objective of this study is to identify the region leading to dyschromatopsia in MS patients' NON-eyes. We evaluated Spearman correlations between color vision and measures of different regions in the afferent visual pathway in 106 MS patients. Regions with significant correlations were included in logistic regression models to assess their independent role in dyschromatopsia. We evaluated color vision with Hardy-Rand-Rittler plates and retinal damage using Optical Coherence Tomography. We ran SIENAX to measure Normalized Brain Parenchymal Volume (NBPV), FIRST for thalamus volume and Freesurfer for visual cortex areas. We found moderate, significant correlations between color vision and macular retinal nerve fiber layer (rho = 0.289, p = 0.003), ganglion cell complex (GCC = GCIP) (rho = 0.353, p < 0.001), thalamus (rho = 0.361, p < 0.001), and lesion volume within the optic radiations (rho = -0.230, p = 0.030). Only GCC thickness remained significant (p = 0.023) in the logistic regression model. In the final model including lesion load and NBPV as markers of diffuse neuroaxonal damage, GCC remained associated with dyschromatopsia [OR = 0.88 95 % CI (0.80-0.97) p = 0.016]. This association remained significant when we also added sex, age, and disease duration as covariates in the regression model. Dyschromatopsia in NON-eyes is due to damage of retinal ganglion cells (RGC) in MS. Color vision can serve as a marker of RGC damage in MS.