Semantic Image Retrieval Research Articles

An ontology‐based semantic description model of ubiquitous map images

AbstractMap images with various themes and cartographic representations have become ubiquitous on the Internet. Such ubiquitously and openly accessible data, named ubiquitous map images in this study, are a potential resource for many geographic information applications such as cartographic design. However, there is a semantic gap between the simple physical form and the complex connotation of ubiquitous map images, which hinders their further applications. To mitigate such barrier, this article develops an ontology‐based semantic description model for ubiquitous map images. First, we discuss the design concerns and principles of the semantic description model of ubiquitous map images. Second, three semantic layers of the semantic description model are proposed, that is, image semantic description layer, cognitive tool layer, and information source layer, and detailed semantic description items are defined for each layer. Furthermore, a formalized semantic description model for ubiquitous map images is developed using ontology construction tools, which lays the foundation for automated and fine‐grained reasoning with the information embedded in map images. We construct a small test dataset consisting of weather maps, and use three types of constraints, namely “time‐topic,” “region‐topic,” and “map auxiliary elements” for the semantic retrieval experiments. The experiments show that the proposed semantic ontology model can enable complex semantic retrieval of ubiquitous map images. Finally, the scalability of the model is discussed from three perspectives: the depth of description, the combination with intelligent methods, and the integration with other open knowledge bases. The proposed model provides a semantic label system for applying data‐driven approaches to decode ubiquitous map images, which also paves the path to the development of cartographic theory in the era of information and communications technologies.

Semantic Level Image Retrieval in Domain Adaptive Semantic Segmentation

The traditional domain adaptation task is not sufficient in mining inter domain image context. We propose a solution from the perspective of image semantic level retrieval. Image semantic level retrieval is based on the high-level semantics of images, retrieving the similarity or correspondence between images in terms of content and scene between domains. Our work focuses on the meaning, structure, and contextual information expressed by the target image, and retrieves the most relevant image from the source domain to improve the accuracy of domain adaptation. Furthermore, in response to the negative transfer brought about by image retrieval, we will introduce a category level weight adjustment module to further improve the accuracy of semantic level retrieval and weaken the negative impact caused by low precision categories. Finally, we demonstrated the effectiveness of semantic level image retrieval through experiments.

Towards Codebook-Free Deep Probabilistic Quantization for Image Retrieval.

As a classical feature compression technique, quantization is usually coupled with inverted indices for scalable image retrieval. Most quantization methods explicitly divide feature space into Voronoi cells, and quantize feature vectors in each cell into the centroids learned from data distribution. However, Voronoi decomposition is difficult to achieve discriminative space partition for semantic image retrieval. In this paper, we explore semantic-aware feature space partition by deep neural network instead of Voronoi cells. To this end, we propose a new deep probabilistic quantization method, abbreviated as DeepIndex, which constructs inverted indices without explicit centroid learning. In our method, the deep neural network takes an image as input and outputs its probability of being put into each inverted index list. During training, we progressively quantize each image into the inverted lists with the top- T maximal probabilities, and calculate the reward of each trial based on retrieval accuracy. We optimize the deep neural network to maximize the probability of the inverted list with maximal reward. In this way, the retrieval performance is directly optimized, leading to a more semantically discriminative space partition than other quantization methods. The experiments on public image datasets demonstrate the effectiveness of our DeepIndex method on semantic image retrieval.

Sketch-based semantic retrieval of medical images

The volume of medical images stored in hospitals is rapidly increasing; however, the utilization of these accumulated medical images remains limited. Existing content-based medical image retrieval (CBMIR) systems typically require example images, leading to practical limitations, such as the lack of customizable, fine-grained image retrieval, the inability to search without example images, and difficulty in retrieving rare cases. In this paper, we introduce a sketch-based medical image retrieval (SBMIR) system that enables users to find images of interest without the need for example images. The key concept is feature decomposition of medical images, which allows the entire feature of a medical image to be decomposed into and reconstructed from normal and abnormal features. Building on this concept, our SBMIR system provides an easy-to-use two-step graphical user interface: users first select a template image to specify a normal feature and then draw a semantic sketch of the disease on the template image to represent an abnormal feature. The system integrates both types of input to construct a query vector and retrieves reference images. For evaluation, ten healthcare professionals participated in a user test using two datasets. Consequently, our SBMIR system enabled users to overcome previous challenges, including image retrieval based on fine-grained image characteristics, image retrieval without example images, and image retrieval for rare cases. Our SBMIR system provides on-demand, customizable medical image retrieval, thereby expanding the utility of medical image databases.

Deep Incremental Hashing for Semantic Image Retrieval With Concept Drift

Hashing methods are widely used for content-based image retrieval due to their attractive time and space efficiencies. Several dynamic hashing methods have been proposed for image retrieval tasks in non-stationary environments. However, concept drift problems in non-stationary environment are seldomly considered which lead to significant deterioration of performance. Therefore, we propose Deep Incremental Hashing (DIH). For the learning part, similarity-preserving object codes of each newly arriving data chunk are computed using the product of its label matrix and a random Gaussian matrix generated offline. A point-wise loss function is then devised to guide the learning of a deep hash neural network. To retain the learned knowledge of former chunks, a weighting-based method is utilized to combine different hash tables trained at different time steps to form a multi-table hashing system. Experimental results on 13 simulated concept drift environments show that DIH adapts to non-stationary data environments well and yields better retrieval performance than existing dynamic hashing methods.

Deep semantic image compression via cooperative network pruning

Incorporating semantic analysis into image compression can significantly reduce the repetitive computation of fundamental semantic analysis in downstream applications such as semantic image retrieval. In this paper, we tackle the semantic image compression task, which embeds semantics in the compressed bitstream. An intuitive solution to this task is joint multi-task training, which generally results in the trade-off of one task to accommodate the other. We thus provide an alternative pilot solution: given a pair of pre-trained teacher networks that specialize in image compression and semantic inference respectively, we first fuse both models to acquire an ensemble model and then leverage cooperative network pruning and retraining to condense the knowledge. Various experiments on five benchmark datasets validate that the proposed method achieves on par and in many cases better performance than the teachers yet comes in a more compact size, and outperforms its multi-task learning and knowledge distillation counterparts.

(Retracted) Mimicking human vision systems: deep-learning-based feature fusion for semantic image retrieval

Cross-form feature combination is an important multifeature fusion technique where the purpose is to implicitly discover the relationship between samples from different modalities, i.e., to retrieve another image encoded by similar semantics through one example image. In the past decade, cross-modal image retrieval has becoming a hotspot investigated by many academicians. Moreover, it is now a significant tool for the future performance enhancement of image retrieval. A long-short term memory (LSTM)-based feature fusion model is proposed. First, aiming at the competitiveness of nonmixed deep architecture for image retrieval, the mechanism of LSTM is introduced in detail. Among them, ground-truth-based methods are used to improve cross-modality. We notice that LSTM can mimic human visual understanding of image semantics well. To improve the accuracy of oblique-form image retrieval, systems based on binary representation are proposed to improve cross-modal similarity and effectiveness of message recovery. Second, we use a quality model to measure the commonly used image low-/high-level visual features, where the disqualified features are abandoned accordingly. This in turn achieves an optimal set of highly descriptive features for image retrieval. Furthermore, we use LSTM and the refined visual features to build a biological model for image retrieval, wherein the multimodel features can be optimally incorporated at the temporal level. Extensive experimental validations on multiple well-known image sets have shown the superiority of our method.

Length adaptive hashing for semi-supervised semantic image retrieval

Length adaptive hashing for semi-supervised semantic image retrieval

Improving the Efficiency of Semantic Image Retrieval Using a Combined Graph and SOM Model

Improving the Efficiency of Semantic Image Retrieval Using a Combined Graph and SOM Model

A Cross-Attention Mechanism Based on Regional-Level Semantic Features of Images for Cross-Modal Text-Image Retrieval in Remote Sensing

With the rapid development of remote sensing (RS) observation technology over recent years, the high-level semantic association-based cross-modal retrieval of RS images has drawn some attention. However, few existing studies on cross-modal retrieval of RS images have addressed the issue of mutual interference between semantic features of images caused by “multi-scene semantics”. Therefore, we proposed a novel cross-attention (CA) model, called CABIR, based on regional-level semantic features of RS images for cross-modal text-image retrieval. This technique utilizes the CA mechanism to implement cross-modal information interaction and guides the network with textual semantics to allocate weights and filter redundant features for image regions, reducing the effect of irrelevant scene semantics on retrieval. Furthermore, we proposed BERT plus Bi-GRU, a new approach to generating statement-level textual features, and designed an effective temperature control function to steer the CA network toward smooth running. Our experiment suggested that CABIR not only outperforms other state-of-the-art cross-modal image retrieval methods but also demonstrates high generalization ability and stability, with an average recall rate of up to 18.12%, 48.30%, and 55.53% over the datasets RSICD, UCM, and Sydney, respectively. The model proposed in this paper will be able to provide a possible solution to the problem of mutual interference of RS images with “multi-scene semantics” due to complex terrain objects.

A METHOD OF SEMANTIC-BASED IMAGE RETRIEVAL USING GRAPH CUT

Semantic extraction for images is a topical problem and is applied in many different semantic search systems. In this paper, a method of semantic image retrieval is proposed based on the set of similar images to the input image; then, the semantics of the images are queried on the ontology through the visual words vector. The objects of each image are extracted and classified by the Mask R-CNN and stored on the cluster graph to extract semantics for the image. The similar images of query image are extracted on the cluster graph; then, the k-NN algorithm is applied to find the visual words vector as the basis for querying the semantic of the query image on the ontology by the SPARQL query. On the basis of the proposed method, an experiment was built and evaluated on two large-volume image datasets MIRFLICKR-25K and MS COCO. Experimental results are compared with recently published works on the same datasets to demonstrate the effectiveness of the proposed method. According to the experimental results, the method of semantic image retrieval in this paper has improved the accuracy to 0.897 for MIRFLICKR-25K, 0.833 for MS COCO.

A Semantic Image Retrieval Method Based on Interest Selection

There is a semantic gap between people’s understanding of images and the underlying visual features of images, which makes it difficult for image retrieval results to meet the needs of individual interests. To overcome the semantic gap in image retrieval, this paper proposes a semantic image retrieval method based on interest selection. This method analyses the interest points of individual selections and gives the weight of the interest selection in different regions of an image. By extracting the underlying visual features of different regions, this paper constructs two feature vector methods after users’ interest point weighting. The two methods are called interest weighted summation and interest weighting. Finally, this paper compares the accuracy of different image classification methods using a support vector machine classification algorithm. The experimental results show that the target classification accuracy of the classification algorithm based on interest weighted summation is higher than that of the traditional and interest weighted methods. The classification algorithm based on interest weighted summation has the highest overall effect on target object classification in the four experimental scenarios. Therefore, the interest point selection method can effectively improve the overall satisfaction of image recommendation and can be used as a novel solution to overcome the semantic gap.

A Model of Semantic-Based Image Retrieval Using C-Tree and Neighbor Graph

The problems of image mining and semantic image retrieval play an important role in many areas of life. In this paper, a semantic-based image retrieval system is proposed that relies on the combination of C-Tree, which was built in our previous work, and a neighbor graph (called Graph-CTree) to improve accuracy. The k-Nearest Neighbor (k-NN) algorithm is used to classify a set of similar images that are retrieved on Graph-CTree to create a set of visual words. An ontology framework for images is created semi-automatically. SPARQL query is automatically generated from visual words and retrieve on ontology for semantics image. The experiment was performed on image datasets, such as COREL, WANG, ImageCLEF, and Stanford Dogs, with precision values of 0.888473, 0.766473, 0.839814, and 0.826416, respectively. These results are compared with related works on the same image dataset, showing the effectiveness of the methods proposed here.

A Study on Image Search System using Semantics Based on Smartphone

Image semantic retrieval has been a crux to bridge "semantic gap" between the simple visual features and the abundant semantics delivered by a image. Effective image retrieval using semantics is one of the major challenges in image retrieval. We suggest a semantic retrieval and clustering method of image using image annotation user interface. And also design and implement a image semantic search management system that facilitates image management and semantic retrieval, which fully relies on the MPEG-7 standard as information base, and using a native XML database, which is Berkeley DB XML.

Scene Graph Embeddings Using Relative Similarity Supervision

Scene graphs are a powerful structured representation of the underlying content of images, and embeddings derived from them have been shown to be useful in multiple downstream tasks. In this work, we employ a graph convolutional network to exploit structure in scene graphs and produce image embeddings useful for semantic image retrieval. Different from classification-centric supervision traditionally available for learning image representations, we address the task of learning from relative similarity labels in a ranking context. Rooted within the contrastive learning paradigm, we propose a novel loss function that operates on pairs of similar and dissimilar images and imposes relative ordering between them in embedding space. We demonstrate that this Ranking loss, coupled with an intuitive triple sampling strategy, leads to robust representations that outperform well-known contrastive losses on the retrieval task. In addition, we provide qualitative evidence of how retrieved results that utilize structured scene information capture the global context of the scene, different from visual similarity search.

Bangla language textual image description by hybrid neural network model

Automatic image captioning task in different language is a challenging task which has not been well investigated yet due to the lack of dataset and effective models. It also requires good understanding of scene and contextual embedding for robust semantic interpretation of images for natural language image descriptor. To generate image descriptor in Bangla, we created a new Bangla dataset of images paired with target language label, named as Bangla Natural Language Image to Text (BNLIT) dataset. To deal with the image understanding, we propose a hybrid encoder-decoder model based on encoder-decoder architecture and the model is evaluated on our newly created dataset. This proposed approach achieves significance performance improvement on task of semantic retrieval of images. Our hybrid model uses the Convolutional Neural<br />Network as an encoder whereas the Bidirectional Long Short Term Memory is used for the sentence representation that decreases the computational complexities without trading off the exactness of the descriptor. The model yielded benchmark accuracy in recovering Bangla natural language and we also conducted a thorough numerical analysis of the model performance on the BNLIT dataset.

Data modeling and evaluation of deep semantic annotation for cultural heritage images

PurposeTo meet the emerging demand for fine-grained annotation and semantic enrichment of cultural heritage images, this paper proposes a new approach that can transcend the boundary of information organization theory and Panofsky's iconography theory.Design/methodology/approachAfter a systematic review of semantic data models for organizing cultural heritage images and a comparative analysis of the concept and characteristics of deep semantic annotation (DSA) and indexing, an integrated DSA framework for cultural heritage images as well as its principles and process was designed. Two experiments were conducted on two mural images from the Mogao Caves to evaluate the DSA framework's validity based on four criteria: depth, breadth, granularity and relation.FindingsResults showed the proposed DSA framework included not only image metadata but also represented the storyline contained in the images by integrating domain terminology, ontology, thesaurus, taxonomy and natural language description into a multilevel structure.Originality/valueDSA can reveal the aboutness, ofness and isness information contained within images, which can thus meet the demand for semantic enrichment and retrieval of cultural heritage images at a fine-grained level. This method can also help contribute to building a novel infrastructure for the increasing scholarship of digital humanities.

Additive deep feature optimization for semantic image retrieval

Rapid increase in the distribution of multimedia content in recent times presents a challenging problem for content-based image retrieval systems. Image contents, such as position and shape of objects alongside contextual features such as background can be used to retrieve visually similar images. Variations in contrast, color, intensity and texture of contextually similar images make it an interesting research problem. A deep convolutional neural network-based model called MaxNet for content-based image retrieval is presented in this paper. The proposed system bypasses the reliance on handcrafted features and extracts deep features directly from the images, which are then used to retrieve contextually similar images from the database. The proposed MaxNet model is built by stacking the updated inception module in a hierarchical fashion. Features extracted from various pipelines in the inception module are aggregated after each inception maximizing the feature values. This novel aggregation step generates a model that is able to adapt to variety of datasets. Various types of aggregations are discussed in this study. Model overcomes the over-fitting problem by using a dropout layer after each inception block and just before the output layer. The system outputs softmax probabilities, which are stored in the feature database and are used to compute the similarity index to retrieve images similar to the query image. The MaxNet model is evaluated using four popular image retrieval datasets namely, Corel-1k, Corel-5k, Corel-10k and Caltech-101, where it outperforms state-of-the-art methods in key performance indicators.

Deep Hashing Learning for Visual and Semantic Retrieval of Remote Sensing Images

Driven by the urgent demand for managing remote sensing big data, large-scale remote sensing image retrieval (RSIR) attracts increasing attention in the remote sensing field. In general, existing retrieval methods can be regarded as visual-based retrieval approaches that search and return a set of similar images to a given query image from a database. Although these retrieval methods have delivered good results, there is still a question that needs to be addressed: can we obtain the accurate semantic labels of the returned similar images to further help analyzing and processing imagery? To this end, in this article, we redefine the image retrieval problem as visual and semantic retrieval of images. Especially, we propose a novel deep hashing convolutional neural network (DHCNN) to retrieve similar images and classify their semantic labels simultaneously in a unified framework. In more detail, a convolutional neural network (CNN) is used to extract high-dimensional deep features. Then, a hash layer is perfectly inserted into the network to transfer the deep features into compact hash codes. In addition, a fully connected layer with a softmax function is performed on the hash layer to generate the probability distribution of each class. Finally, a loss function is elaborately designed to consider the label loss of each image and similarity loss of pairs of images simultaneously. Experimental results on three remote sensing data sets demonstrate that the proposed method can achieve state-of-art retrieval and classification performance.

A Knowledge-Driven Multimedia Retrieval System Based on Semantics and Deep Features

In recent years the information user needs have been changed due to the heterogeneity of web contents which increasingly involve in multimedia contents. Although modern search engines provide visual queries, it is not easy to find systems that allow searching from a particular domain of interest and that perform such search by combining text and visual queries. Different approaches have been proposed during years and in the semantic research field many authors proposed techniques based on ontologies. On the other hand, in the context of image retrieval systems techniques based on deep learning have obtained excellent results. In this paper we presented novel approaches for image semantic retrieval and a possible combination for multimedia document analysis. Several results have been presented to show the performance of our approach compared with literature baselines.