Hierarchical Multi-Label Attribute Classification With Graph Convolutional Networks on Anime Illustration

Abstract

In this study, we present a hierarchical multi-modal multi-label attribute classification model for anime illustrations using graph convolutional networks (GCNs). The focus of this study is multi-label attribute classification, as creators of anime illustrations frequently and deliberately emphasize subtle features of characters and objects. To analyze the connections between attributes, we develop a multi-modal GCN-based model that can use semantic features of anime illustrations. To create features representing the semantic information of anime illustrations, we construct a novel captioning framework by combining real-world images with their animated style transformations. In addition, because the attributes of anime illustrations are hierarchical, we introduce a loss function that considers the hierarchy of attributes to improve classification accuracy. The proposed method has two main contributions: 1) By introducing a GCN with semantic features into the multi-label attribute classification task of anime illustrations, we capture more comprehensive relationships between attributes. 2) By following certain rules to build a hierarchical structure of attributes that appear frequently in anime illustrations, we further capture subordinate relationships between attributes. In addition, we demonstrate the effectiveness of the proposed method by experiments.