Privileged multi-task learning for attribute-aware aesthetic assessment

Abstract

Aesthetic attributes are crucial for aesthetics because they explicitly present some photo quality cues that a human expert might use to evaluate a photo’s aesthetic quality. However, the aesthetic attributes have not been largely and sufficiently exploited for photo aesthetic assessment. In this paper, we propose a novel approach to photo aesthetic assessment with the help of aesthetic attributes. The aesthetic attributes are used as privileged information (PI), which is often available during training phase but unavailable in prediction phase due to the high collection expense. The proposed framework consists of a deep multi-task network as generator and a fully connected network as discriminator. Deep multi-task network learns the aesthetic attributes and score simultaneously to capture their dependencies and extract better feature representations. Specifically, we use ranking constraint in the label space, similarity constraint and prior probabilities loss in the privileged information space to make the output of multi-task network converge to that of ground truth. Adversarial loss is used to identify and distinguish the predicted privileged information of a deep multi-task network from the ground truth PI distribution. Experimental results on two benchmark databases demonstrate the superiority of the proposed method to state-of-the-art.