Smartphone-based colorimetry has been widely applied in clinical analysis, although significant challenges remain in its practical implementation, including the need to consider biases introduced by the ambient imaging environment, which limit its potential within a clinical decision pathway. In addition, most commercial devices demonstrate variability introduced by manufacturer-to-manufacturer differences. Here, we undertake a systematic characterization of the potential imaging interferences that lead to this limited performance in conventional smartphones and, in doing so, provide a comprehensive new understanding of smartphone color imaging. Through derivation of a strongly correlated parameter for sample quantification, we enable real-time imaging, which for the first time, takes the first steps to turning the mobile phone camera into an analytical instrument - irrespective of model, software, and the operating systems used. We demonstrate clinical applicability through the imaging of patients' skin, enabling rapid and convenient diagnosis of cyanosis and measurement of local oxygen concentration to a level that unlocks clinical decision-making for monitoring cardiovascular disease and anemia. Importantly, we show that our solution also accounts for the differences in individuals' skin tones as measured across the Fitzpatrick scale, overcoming potential clinically significant errors in current optical oximetry.