A Deep Two-Stream Network for Bidirectional Cross-Media Information Retrieval

Abstract

The recent development in deep learning techniques has showed its wide applications in traditional vision tasks like image classification and object detection. However, as a fundamental problem in artificial intelligence that connects computer vision and natural language processing, bidirectional retrieval of images and sentences is not as popular as the traditional problems, and the results are far from satisfying. In this paper, we consider learning a cross-media representation model with a deep two-stream network. Previous models generally use image label information to train the dataset or strictly correspond the local features in images and texts. Unlike those models, we learn globalized local features, which can reflect the salient objects as well as the details in the images and sentences. After mapping the cross-media data into a common feature space, we use max-margin as the criterion function to update the network. The experiment on the dataset of Flickr8k shows that our approach achieves superior performance compared with the state-of-the-art methods.