Efficient, robust and divisible paired comparison for subjective quality assessment

Abstract

The scale of a database is important for machine learning based image and video quality assessment. Nevertheless, it is greatly limited by the subjective test method. Among the various methods, Paired Comparison(PC) is acknowledged as the most reliable one. However, the test duration grows with square of the number of samples. To solve the dilemma, we propose an improved paired comparison method in this paper. Three types of priori are incorporated to cut down the test duration, including the long-term priori as experience results condensed in existing quality metric, the short-term priori as the subjective scores calculated by the predecessor in ongoing session, and the dynamic priori as the previous decision made by the current assessor. Based on these priori knowledge, only indispensable part of decision is needed to be made by the assessor. Equivalent performance could be achieved in one-tenth of the time used in full paired comparison method. While it is robust to mis-click and divisible to expand the database to a large scale.