Learning with Incomplete Labels for Multi-label Image Annotation Using CNN and Restricted Boltzmann Machines

Abstract

Multi-label image annotation based on convolutional neural networks (CNN) has seen significant improvements in recent years. One problem, however, is that it is difficult to prepare complete labels for the training images and usually training data has missing or incomplete labels. Restricted Boltzmann Machines (RBM) can explore the co-occurrence distribution of the labels and estimate the missing labels efficiently. Hence we intend to propose a novel learning model for multi-label image annotation with missing labels based on CNNs, which aims to regenerate the missing labels for an image by learning the generative model of labels using an RBM. Firstly, label sets are reconstructed by the pre-trained RBM model which is trained on data with some missing labels. Then the reconstructed label sets are used as a teacher signal to train the CNN. The effectiveness of the proposed approach is confirmed by comparing the performance with baseline CNNs using various performance evaluation metrics on two different data sets. Experimental results prove that our RBM-CNN formulation exceeds the performance of the baseline CNN.