Automatic Image Annotation using Tag Relations and Graph Convolutional Networks

Abstract

Automatic image annotation is a mechanism to assign a list of appropriate tags that describe the visual content of a given image. Most methods only focus on the content of the images and ignore the relationship between the tags in vocabulary. In this work, we propose a new deep learning-based automatic image annotation architecture, which considers label dependencies in a graph convolution neural network structure and extracts tag descriptors to re-weight the output class scores based on their relationships. The proposed architecture has three main parts: feature extraction, graph convolutional network, and annotation. In graph convolutional network, we apply one layer convolution on vocabulary graph to get some tag descriptors that are applied on the image features. In the annotation part, we use two previous algorithms to rectify the tags. Compared to related work, the experimental results using Corel5k, Esp Game, and IAPRTC-12 datasets indicate the best performance of the annotating model with $F1\_score$ in IAPRTC-12 and the second-best result with $F1\_score$ and $N^{+}$ measure in Corel5K and finally the second-best result with $F1\_score$ in ESP Game.