Emotion-aware hierarchical interaction network for multimodal image aesthetics assessment

Abstract

Image aesthetics assessment (IAA) has attracted increasing attention recently but is still challenging due to its high abstraction and complexity. Intuitively, image emotion and aesthetics are both human subjective feelings evoked by visual content. Previous researches have demonstrated that image aesthetics has intrinsic relationships with emotion. In fact, human emotional experience has potential impact on the perception of image aesthetics. Therefore, emotion information can be exploited to enhance aesthetic representation learning. Inspired by this, this paper presents an Emotion-Aware Hierarchical Interaction NETwork (EAHI-NET) for multimodal image aesthetics assessment, which explores both intra-modal and inter-modal interactions between aesthetics and emotions hierarchically. Specifically, we first propose the intra-modal emotion-aesthetics interaction module to obtain emotion-enhanced visual and textual aesthetic representations respectively. Then we propose the inter-modal feature enhancement to obtain the cross-modal aesthetic and emotion features. Finally, we design the inter-modal emotion-aesthetics interaction module to further investigate the cross-modal interplay between aesthetics and emotion, based on which hierarchical feature representations are achieved for multimodal IAA. The extensive experiments show that the proposed method can outperform the state-of-the-arts on multimodal AVA and Photo.net datasets.