Improving the discriminability of standard subjective quality assessment methods: a case study

Abstract

Subjective assessment for image or video qualities is considered as the most reliable way to obtain the ground truth for the development of objective quality metrics, especially when leaded by Mean Opinion Score (MOS approaches). However, obtained MOS with standard protocols are noisy due to subject's personal characteristics, such as viewing experience, gender or profession, leading to uncertain ground truth driven by the number of panelists/subjects. The usual way to reduce uncertainty relies on raising this number. In this paper, we demonstrate how a recently introduced Maximum Likelihood Estimation (MLE) based quality recovery model can improve the discriminability of standard subjective quality assessment. Compared to straightforward MOS computation, we present a case study where one can save between 26% to 39% in terms of numbers of subjects at the same discriminability.