This work addresses the long-standing debate surrounding the origin of color variation in fluorite (CaF2) through a novel quantitative approach. By examining eight carefully selected fluorite samples having different hue of colors from the Amba Dongar mine in Gujarat, India, a rigorous quantitative analysis was conducted. This approach combined chemical compositional data and optical spectroscopic features to elucidate the relationship between elemental composition, concentration, and color variation in fluorite. Precise elemental concentration data for trace transition metals, alkali metals, and rare earth elements (REEs) were obtained through inductively coupled plasma mass spectroscopy (ICP-MS) analysis of powdered fluorite samples. Optical spectroscopic techniques, including UV–visible absorption, emission (photoluminescence and fluorescence), and Raman spectroscopy, were employed to capture characteristic spectral signatures for specific color of the study sample. The work unveils a strong correlation between specific elemental concentrations and observed spectral features, particularly influenced by alkaline metals, transition group elements, and REEs. Fluorite’s optical absorption behavior lacks a clear pattern in UV and infrared wavelength ranges but correlates well with transition metal, alkaline element, and REE concentrations in visible wavelength regions, influencing coloration. Luminescent centers in the study fluorite samples correspond to specific REE concentrations, indicating a strong linkage between emission wavelengths with the presence of specific REE. UV–visible and fluorescence in fluorite result from trivalent REE or Eu2+ ions, with emission intensity affected by REE concentration and specific REE or combinations thereof. Raman spectroscopy identifies characteristic modes related to F-substitution and REE impurities, providing insights into fluorite’s structural composition. This quantitative correlation between elemental composition and spectroscopic characteristics represents a novel contribution for understanding color variation mechanisms in fluorite. The comprehensive analysis of this present work underscores the intricate interplay of mineral composition, and element concentration particularly alkaline metals, transition group elements, and REEs, for variation in spectral signatures with variation in fluorite’s visual attributes.