Personalized Image Annotation Using Deep Architecture

Abstract

In image-centric social networks, such as Instagram and Pinterest, users tend to share photos with several tags. These tags describe the content of the image or provide additional contextual information, and therefore may not be necessarily tied to image content and usually carry personal preference. Annotating images in social networks in a personalized manner is in demand. However, the existing image annotation models, which rely only on image content information, cannot capture the user’s tagging preference. In this paper, we propose a deep architecture for personalized image annotation by leveraging the wealth of information in user’s tagging history. The proposed architecture consists of three components: two components for learning features of the image content and user’s history tags and the other one for combining the two learned features to predict the tags. We also explore two ways to model user’s history tags: 1) simply average the embeddings of user’s history tags and 2) model user’s history tags with a sequence model by long short-term memory recurrent neural network. We evaluate our proposed deep architecture on a large-scale and realistic data set, consisting of ~22.8 million public images uploaded by ~4.69 million users. Experimental results show that our proposed deep architecture is effective on a personalized image annotation task.