Large-scale semantic web image retrieval using bimodal deep learning techniques

Abstract

Semantic web image retrieval is useful to end-users for semantic image searches over the Internet. This paper aims to develop image retrieval techniques for large-scale web image databases. An advanced retrieval system, termed Multi-concept Retrieval using Bimodal Deep Learning (MRBDL), is proposed and implemented using Convolutional Neural Networks (CNNs) which can effectively capture semantic correlations between a visual image and its free contextual tags. Different from existing approaches using multiple and independent concepts in a query, MRBDL considers multiple concepts as a holistic scene for retrieval model learning. In particular, we first use a bimodal CNN to train a holistic scene classifier in two modalities, and then semantic correlations of the sub-concepts included in the images are leveraged to boost holistic scene recognition. The predicted semantic scores obtained from holistic scene classifier are combined with complementary information on web images to improve the retrieval performance. Experiments have been carried out over two publicly available web image databases. The results show that our proposed approach performs favorably compared with several other state-of-the-art methods.