Gated CNN for visual quality assessment based on color perception

Abstract

Visual quality assessment aims to build a computational model which can evaluate the quality of image with respect to human perception. As one of the important aspects to represent images, color provides rich information of images and deeply influences the visual perception. Thus, color harmony, which is defined as “two or more colors are sensed together as a single, pleasing, collective impression” (Holtzschue, 2011), is one of the important features to determine the visual/aesthetic quality of images. Despite the recent progress of color harmony models in respect to aesthetic quality assessment, most conventional approaches represent the color harmony only considering the distribution of co-occurrence colors but ignoring the spatial relationships between those neighbored colors. To overcome this limitation, we propose to take advantage of the intrinsic structural properties from conditional random field (CRF) to model the color harmony of images. In the CRF framework, we present a novel method that uses gated convolutional neural networks (CNNs) to calculate the probabilities of being high aesthetic quality for small patches and compute the harmony compatibilities between them, which can be considered as the associated and interactive potentials of CRF. Semantic tag of each image is also employed in our work to improve the proposed harmony model’s discriminant capability, which shows promising improvements for aesthetic quality assessment compared with existing color harmony models.