Image annotation in social networks using graph and multimodal deep learning features

Abstract

Thanks to the evolution of technology, we find a very large number of internet users who use social networks to react and share things with each other. These networks are exploited in the study of several domains. In fact, an Internet user can easily share his images with a simple click on his/her Smartphone. However, we get a large amount of published images on the Internet. Such an amount requires specific access techniques to be used by the search engines to provide searchers with the desired results. An effective way to access the target images is their keywords (or tags). Nevertheless, tags, which people manually attach to images, are of low quality and negatively affect the search engines. Consequently, automatic annotation of images by tags has become an active topic of research in recent years. In this paper, we introduce an automatic annotation method in social networks named MDL-STag (Multimodal features Deep Learning approach for Social image Tagging). This method provides high-quality features using the visual content of the image as well as the textual content of the annotation tag history to personalize the annotation, and that’s with the help of some deep learning models. Then, it merges these features and makes multimodal features for the images of the annotator’s contacts who share the same interests to provide more useful tags through the propagation of images tags that are similar to the target image. In fact, we find that tests give good results on real social networks as the well known Instagram and Flickr.