Image emotion multi-label classification based on multi-graph learning

Abstract

Images contain rich information and can induce various emotions in the audience. Image emotion classification aims to identify the emotion categories that images can evoke. It is widely used in mental health assessment, human–computer interaction, etc. There are two main problems in existing image emotion classification methods: (1) Most of them only focus on a single emotion label; (2) The global structural relationship among semantic objects in the image is ignored. Therefore, this paper proposes an Image Emotion classification method based on Multi-Graph Multi-Label learning (IE-MGML). In contrast to the existing approaches, the image is transformed into a graph-based representation by extracting the emotional features of semantic objects and calculating the similarity between the features. The local (semantic objects) features and global structure (relationship among semantic objects) features of the image are fused by the relationship between nodes. Furthermore, the graph representation of an image from the perspective of multiple emotional features is pooled and modeled as a graph bag containing multiple graphs (i.e., multi-graph). In multi-graph learning, the graph kernel directly evaluates a graph-label dependency score to avoid the loss of structural information caused by graph-instance degradation. The bag(image)-label dependency score is obtained by aggregating the graph-label dependency score from different perspectives through the aggregation function. The problem of error accumulation in the learning process is handled by proposing a threshold-based ranking loss objective function. Moreover, the non-convex optimization problem is addressed using a subgradient descent algorithm to deal with the required high-dimensional space computation. Experimental results on three general image emotion datasets show that the proposed method outperforms the state-of-the-art methods.