Interpreting MOS scores, when can users see a difference? Understanding user experience differences for photo quality

Stamos Katsigiannis,Glenn Van Wallendael,Philip Corriveau,Michele A Saad,Naeem Ramzan,James Scovell,Lucjan Janowski

doi:10.1007/s41233-018-0019-8

Abstract

The use of no-reference image quality evaluation tools that produce MOS scores, like the VIQET tool which was released by the Video Quality Expert Group, raises the question of whether the produced MOS differences between images correspond to noticeable differences in quality by the consumers. In this work, we attempted to approximate the minimum MOS difference that is required in order for people to be able to distinguish between a higher and a lower quality image under realistic conditions that are commonly encountered in the current consumer space. 91 people participated in a subjective just-noticeable-differences study across three countries that used non-simulated image stimuli, produced and evaluated through crowd sourcing for the validation of the VIQET no-reference image quality tool. The image dataset consisted of 15 different scenes belonging to three different scene types, with a total of 210 different image pairs being used. After evaluating the quality of the collected data, a logistic regression analysis approach was employed in order to estimate the minimum MOS difference required between two images in order for a given percentage of people to be able to detect the higher quality image.

Highlights

For several decades, the media industry and the image quality research community have worked on developing and deploying no-reference image quality evaluation tools [10]
We present a subjective study that seeks to determine the smallest noticeable Mean Opinion Score (MOS) difference on a set of images that are representative of consumer usage scenarios, i.e. using image examples that are commonly encountered in the current consumer space
The MOS deltas of the image pairs were taken into consideration for creating the logistic regression model described above

Summary

Introduction

The media industry and the image quality research community have worked on developing and deploying no-reference image quality evaluation tools [10]. This challenging problem requires developing prediction models that algorithmically map photos to scores representative of human judgments of perceived image quality. Pixel-domain methods [13, 19, 24, 26, 29] are directly calculated on the spatial domain and are more convenient and less computationally complex As a result, they have been used for a wide variety of applications such as visual quality assessment and enhancement [26]

Objectives

Methods

Conclusion