High-level Semantic Concepts Research Articles

Learning Relationship-Enhanced Semantic Graph for Fine-Grained Image-Text Matching.

Image-text matching of natural scenes has been a popular research topic in both computer vision and natural language processing communities. Recently, fine-grained image-text matching has shown its significant advance in inferring the high-level semantic correspondence by aggregating pairwise region-word similarity, but it remains challenging mainly due to insufficient representation of high-order semantic concepts and their explicit connections in one modality as its matched in another modality. To tackle this issue, we propose a relationship-enhanced semantic graph (ReSG) model, which can improve the image-text representations by learning their locally discriminative semantic concepts and then organizing their relationships in a contextual order. To be specific, two tailored graph encoders, visual relationship-enhanced graph (VReG) and textual relationship-enhanced graph (TReG), are respectively exploited to encode the high-level semantic concepts of corresponding instances and their semantic relationships. Meanwhile, the representations of each graph node are optimized by aggregating semantically contextual information to enhance the node-level semantic correspondence. Further, the hard-negative triplet ranking loss, center hinge loss, and positive-negative margin loss are jointly leveraged to learn the fine-grained correspondence between the ReSG representations of image and text, whereby the discriminative cross-modal embeddings can be explicitly obtained to benefit various image-text matching tasks in a more interpretable way. Extensive experiments verify the advantages of the proposed fine-grained graph matching approach, by achieving the state-of-the-art image-text matching results on public benchmark datasets.

Parallel encoder–decoder framework for image captioning

Recent progress in deep learning has led to successful utilization of encoder–decoder frameworks inspired by machine translation in image captioning models. The stacking of layers in encoders and decoders has made it possible to use several modules in encoders and decoders. However, just one type of module in encoder or decoder has been used in stacked models. In this research, we propose a parallel encoder–decoder framework that aims to take advantage of multiple of types modules in encoders and decoders, simultaneously. This framework contains augmented parallel blocks, which include stacking modules or non-stacked ones. Then, the results of the blocks are integrated to extract higher-level semantic concepts. This general idea is not limited to image captioning and can be customized for many applications that utilize encoder–decoder frameworks. We evaluated our proposed method on the MS-COCO dataset and achieved state-of-the-art results. We got 149.92 for CIDEr-D metric outperforming state-of-the-art image captioning models.

Global Concept-Based Interpretability for Graph Neural Networks via Neuron Analysis

Graph neural networks (GNNs) are highly effective on a variety of graph-related tasks; however, they lack interpretability and transparency. Current explainability approaches are typically local and treat GNNs as black-boxes. They do not look inside the model, inhibiting human trust in the model and explanations. Motivated by the ability of neurons to detect high-level semantic concepts in vision models, we perform a novel analysis on the behaviour of individual GNN neurons to answer questions about GNN interpretability. We propose a novel approach for producing global explanations for GNNs using neuron-level concepts to enable practitioners to have a high-level view of the model. Specifically, (i) to the best of our knowledge, this is the first work which shows that GNN neurons act as concept detectors and have strong alignment with concepts formulated as logical compositions of node degree and neighbourhood properties; (ii) we quantitatively assess the importance of detected concepts, and identify a trade-off between training duration and neuron-level interpretability; (iii) we demonstrate that our global explainability approach has advantages over the current state-of-the-art -- we can disentangle the explanation into individual interpretable concepts backed by logical descriptions, which reduces potential for bias and improves user-friendliness.

Unsupervised Hashing with Semantic Concept Mining

Recently, to improve the unsupervised image retrieval performance, plenty of unsupervised hashing methods have been proposed by designing a semantic similarity matrix, which is based on the similarities between image features extracted by a pre-trained CNN model. However, most of these methods tend to ignore high-level abstract semantic concepts contained in images. Intuitively, concepts play an important role in calculating the similarity among images. In real-world scenarios, each image is associated with some concepts, and the similarity between two images will be larger if they share more identical concepts. Inspired by the above intuition, in this work, we propose a novel Unsupervised Hashing with Semantic Concept Mining, called UHSCM, which leverages a VLP model to construct a high-quality similarity matrix. Specifically, a set of randomly chosen concepts is first collected. Then, by employing a vision-language pretraining (VLP) model with the prompt engineering which has shown strong power in visual representation learning, the set of concepts is denoised according to the training images. Next, the proposed method UHSCM applies the VLP model with prompting again to mine the concept distribution of each image and construct a high-quality semantic similarity matrix based on the mined concept distributions. Finally, with the semantic similarity matrix as guiding information, a novel hashing loss with a modified contrastive loss based regularization item is proposed to optimize the hashing network. Extensive experiments on three benchmark datasets show that the proposed method outperforms the state-of-the-art baselines in the image retrieval task.

BIM product recommendation for intelligent design using style learning

Building information modeling (BIM) has been widely adopted in architectural interior design due to its characteristics such as visualization, coordination and simulation. Currently, an increasing number of BIM products are created to facilitate BIM interior design, which triggers new demand for BIM product recommendations to improve design efficiency by directly recommending satisfactory BIM products for designers. Current efforts mainly focus on the text-based BIM product retrieval and ignore the high-level semantic concept of style features. Although style consistency is one of the critical design principles, BIM product recommendation satisfying style consistency remains to be unexplored. This study proposes a novel design recommendation scheme for BIM products using style learning, termed DrStyle. Firstly, a product representative image selection model is designed to identify a representative image retaining the best style of a BIM product. Then, the DrStyle employs a style learning algorithm to extract the style feature vector using the representative image as input. Finally, a BIM product recommendation scheme is proposed to recommend BIM products by computing the similarities of style feature vectors. The performance of the proposed DrStyle was evaluated, and the experiment results showed that the DrStyle achieved an average precision of 68.8% purely using the style. This study complements intelligent design from a novel perspective of style consistency and will inspire more comprehensive intelligent design schemes.

Integration of the latent variable knowledge into deep image captioning with Bayesian modeling

AbstractAutomatic image captioning systems assign one or more sentences to images to describe their visual content. Most of these systems use attention‐based deep convolutional neural networks and recurrent neural networks (CNN‐RNN‐Att). However, they must optimally use latent variables and side information within the image concepts. This study aims to integrate a latent variable into image captioning using CNN‐RNN‐Att. A Bayesian modeling framework is used for this work. As an instance of a latent variable, High‐Level Semantic Concepts (HLSCs) of tags are used to implement the proposed model. The Bayesian model output interpretation is to localize the entire image description process and breaks it down into sub‐problems. Thus, a baseline descriptor subnet is trained independently for each sub‐problem, and it is the only expert in captioning for a given HLSC. The final output is the caption derived from the subnet; its HLSC is closest to the image content. The results indicate that CNN‐RNN‐Att applied to data localized using HLSCs improves the captioning accuracy of the proposed method, which can be compared to the latest state‐of‐the‐art and most accurate captioning systems.

Shrinking the Semantic Gap: Spatial Pooling of Local Moment Invariants for Copy-Move Forgery Detection

Copy-move forgery is a manipulation of copying and pasting specific patches from and to an image, with potentially illegal or unethical uses. Recent advances in the forensic methods for copy-move forgery have shown increasing success in detection accuracy and robustness. However, for images with high self-similarity or strong signal corruption, the existing algorithms often exhibit inefficient processes and unreliable results. This is mainly due to the inherent semantic gap between low-level visual representation and high-level semantic concept. In this paper, we present a very first study of trying to mitigate the semantic gap problem in copy-move forgery detection, with spatial pooling of local moment invariants for midlevel image representation. Our detection method expands the traditional works on two aspects: 1) we introduce the bag-of-visual-words model into this field for the first time, may meaning a new perspective of forensic study; 2) we propose a word-to-phrase feature description and matching pipeline, covering the spatial structure and visual saliency information of digital images. Extensive experimental results show the superior performance of our framework over state-of-the-art algorithms in overcoming the related problems caused by the semantic gap.

Bayesian Cnn Modeling Based on Latent and High-Level Semantic Concepts for Automatic Image Description

Bayesian Cnn Modeling Based on Latent and High-Level Semantic Concepts for Automatic Image Description

OntoImVid-ontology based semantic image/video processing

The rapid increase in the available amount of video data is creating a growing demand for efficient methods for understanding and managing it at the semantic level. Domain ontology is used to define high level semantic concepts and their relations in the context of the examined domain. A low-level features and video content analysis algorithms are integrated into the ontology to enrich video semantic analysis. OWL is used for the ontology description. Rules in Description Logic are defined to describe how features and algorithms for video analysis should be applied according to different perception content and low-level features. OntoImVid structure is created from both image and video semantic analysis results. Proposed OntoImVid combines the image and Video retrieval in same semantic retrieval technique in efficient manner.

Attribute disentanglement and registration for occluded person re-identification

Occluded person re-identification is a challenging task which suffers from various obstacles. However, existing occluded Re-ID methods tend to exploit body detectors for pedestrian alignment, which are over-reliant on detection and lack of a flexible matching mechanism. To address this issue, we propose an Attribute Disentanglement and Registration (ADR) network to excavate non-occluded regions via attribute feature disentanglement, which can be matched effectively with a robust and soft attribute registration. The proposed ADR takes full advantages of pedestrian attributes’ high-level semantic concepts to alleviate the occlusion problem. First, the Attribute Disentanglement (AD) module obtains the representations of different attributes by localizing their spatial positions. Then the Attribute Registration (AR) module searches and matches these localized regions between different pedestrian images to conduct a registration, which allows the attribute features to be adaptively and efficiently matched. Extensive experiments on occluded, partial, and holistic Re-ID benchmarks demonstrate the effectiveness of the proposed ADR framework as well as its superiority over the existing state-of-the-art methods.

Integrating semantic features in fruit recognition based on perceptual color and semantic template

Many fruit recognition works have applied statistical approaches to make an exact correlation between low-level visual feature information and high-level semantic concepts given by predefined text caption or keywords. Two common fruit recognition models include bag-of-features (BoF) and convolutional neural network (ConvNet), which achieve high-performance results. In most cases, the overfitting problem is unavoidable. This problem makes it difficult to generalize new instances with only a slightly different appearance, although belonging to the same category. This article proposes a new fruit recognition model by associating an object's low-level features in an image with a high-level concept. We define a perceptual color for each fruit species to construct a relationship between fruit color and semantic color name. Furthermore, we develop our model by integrating the perceptual color and semantic template concept to solve the overfitting problem. The semantic template concept as a mapping between the high-level concept and the low-level visual feature is adopted in this model. The experiment was conducted on three different fruit image datasets, with one dataset as train data and the two others as test data. The experimental results demonstrate that the proposed model, called perceptual color on semantic template (PCoST), is significantly better than the BoF and ConvNet models in reducing the overfitting problem.

Vision-to-Language Tasks Based on Attributes and Attention Mechanism.

Vision-to-language tasks aim to integrate computer vision and natural language processing together, which has attracted the attention of many researchers. For typical approaches, they encode image into feature representations and decode it into natural language sentences. While they neglect high-level semantic concepts and subtle relationships between image regions and natural language elements. To make full use of these information, this paper attempt to exploit the text-guided attention and semantic-guided attention (SA) to find the more correlated spatial information and reduce the semantic gap between vision and language. Our method includes two-level attention networks. One is the text-guided attention network which is used to select the text-related regions. The other is SA network which is used to highlight the concept-related regions and the region-related concepts. At last, all these information are incorporated to generate captions or answers. Practically, image captioning and visual question answering experiments have been carried out, and the experimental results have shown the excellent performance of the proposed approach.

A comprehensive survey on the reduction of the semantic gap in content-based image retrieval

In the last few decades, content-based image retrieval is considered as one of the most vivid research topics in the field of information retrieval. The limitation of current content-based image retrieval systems is that low-level features are highly ineffective to represent the semantic contents of the image. Most of the research work in content-based image retrieval is focused on bridging the semantic gap between the low-level features and high-level semantic concepts of image. This paper presents a thorough study of different techniques for the reduction of semantic gap. The existing techniques are broadly categorised as: 1) image annotation techniques to define the high-level concepts in image; 2) relevance feedback techniques to integrate user's perception; 3) machine learning and deep learning techniques to associate low-level features with high-level concepts. In addition, the general architecture of semantic-based image retrieval system has been discussed in this survey. This paper also highlights the current and future applications of content-based image retrieval. The paper concludes with promising future research directions.

Image annotation based on multi-view robust spectral clustering

Nowadays, image annotation has been a hot topic in the semantic retrieval field due to the abundant growth of digital images. The purpose of these methods is to realize the content of images and assign appropriate keywords to them. Extensive efforts have been conducted in this field, which effectiveness is limited between low-level image features and high-level semantic concepts. In this paper, we propose a Multi-View Robust Spectral Clustering (MVRSC) method, which tries to model the relationship between semantic and multi-features of training images based on the Maximum Correntropy Criterion. A Half-Quadratic optimization framework is used to solve the objective function. According to the constructed model, a few tags are suggested based on a novel decision-level fusion distance. The stability condition and bound calculation of MVRSC are analyzed, as well. Experimental results on real-world Flickr and 500PX datasets, and Corel5K confirm the superiority of the proposed method over other competing models.

Person Retrieval in Surveillance Videos Via Deep Attribute Mining and Reasoning

Person retrieval largely relies on the appearance features of pedestrians. This task is rather more difficult in surveillance videos due to the limitations of extracting robust appearance features brought by the cross-view and cross-camera data with lower image resolution, motion blur, occlusion and other kinds of image degradation. To build up a more reliable person retrieval system, recent works introduced appearance attribute models to describe and distinguish different persons with high-level semantic concepts. Despite the progress of previous works, the value of utilizing appearance attributes is still under-explored. On one hand, existing methods lack for concise and precise attribute representations that are specific for each attribute category and, in the meantime, are able to filter noisy information in irrelevant spatial locations and useless patterns. On the other hand, correlation and reasoning between different attributes are neglected, which could generate more useful information and add more robustness to the retrieval system. In this paper, we propose an Attribute Mining and Reasoning (AMR) framework which is capable to handle the issues in question. The AMR makes better use of appearance attributes with two main components. First, the AMR disentangles the representations of different attributes by localizing their spatial positions and identifying their effective patterns in a weakly supervised manner. To achieve more reliable localization, we propose the Attribute Localization Ensemble (ALE) module that is consisted of multiple localization heads and a voting mechanism. Second, we introduce the Attribute Reasoning (AR) module to correlate different attributes together with the global appearance features and discover their latent relations to generate more comprehensive descriptions of pedestrians. Extensive experiments on DukeMTMC-ReID and Market-1501 datasets demonstrate the effectiveness of the proposed AMR framework as well as its superiority over the existing state-of-the-art methods. The AMR model also shows great generalization ability on the unseen CUHK03 dataset when it is only trained on Market-1501 dataset.

Discrete Semantic Alignment Hashing for Cross-Media Retrieval.

Cross-media hashing, which maps data from different modalities to a low-dimensional sharing Hamming space, has attracted considerable attention due to the rapid increase of multimodal data, for example, images and texts. Recent cross-media hashing works mainly aim at learning compact hash codes to preserve the class label-based or feature-based similarities among samples. However, these methods ignore the unbalanced semantic gaps between different modalities and high-level semantic concepts, which generally results in less effective hash functions and unsatisfying retrieval performance. Specifically, the key words of texts contain semantic meanings, while the low-level features of images lack of semantic meanings. That means the semantic gap in image modality is larger than that in text modality. In this paper, we propose a simple yet effective hashing method for cross-media retrieval to address this problem, dubbed discrete semantic alignment hashing (DSAH). First, DSAH formulates to exploit collaborative filtering to mine the relations between class labels and hash codes, which can reduce memory consumption and computational cost compared to pairwise similarity. Then, the attribute of image modality is employed to align the semantic information with text modality. Finally, to further improve the quality of hash codes, we propose a discrete optimization algorithm to learn discrete hash codes directly, and each bit has a closed-form solution. Extensive experiments on multiple public databases show that our model can seamlessly incorporate attributes and achieve promising performance.

A Visual Saliency-Based Approach for Content-Based Image Retrieval

During the past two decades an enormous amount of visual information has been generated; as a result, content-based image retrieval (CBIR) has received considerable attention. In CBIR the image is used as a query to find the most similar images. One of the biggest challenges in CBIR system is to fill up the “semantic gap,” which is the gap between low-level visual features and the high-level semantic concepts of an image. In this paper, the authors have proposed a saliency-based CBIR system that utilizes the semantic information of image and users search intention. In the proposed model, firstly a significant region is identified with the help of method structured matrix decomposition (SMD) using high-level priors that highlight the prominent area of the image. After that, a two-dimensional principal component analysis (2DPCA) is used as a feature, which is compact and effectively used for fast recognition. Experiment results are validated on different image dataset having an extensive collection of semantic classifications.

Knowledge Graph Driven Approach to Represent Video Streams for Spatiotemporal Event Pattern Matching in Complex Event Processing

Complex Event Processing (CEP) is an event processing paradigm to perform real-time analytics over streaming data and match high-level event patterns. Presently, CEP is limited to process structured data stream. Video streams are complicated due to their unstructured data model and limit CEP systems to perform matching over them. This work introduces a graph-based structure for continuous evolving video streams, which enables the CEP system to query complex video event patterns. We propose the Video Event Knowledge Graph (VEKG), a graph-driven representation of video data. VEKG models video objects as nodes and their relationship interaction as edges over time and space. It creates a semantic knowledge representation of video data derived from the detection of high-level semantic concepts from the video using an ensemble of deep learning models. A CEP-based state optimization — VEKG-Time Aggregated Graph (VEKG-TAG) — is proposed over VEKG representation for faster event detection. VEKG-TAG is a spatiotemporal graph aggregation method that provides a summarized view of the VEKG graph over a given time length. We defined a set of nine event pattern rules for two domains (Activity Recognition and Traffic Management), which act as a query and applied over VEKG graphs to discover complex event patterns. To show the efficacy of our approach, we performed extensive experiments over 801 video clips across 10 datasets. The proposed VEKG approach was compared with other state-of-the-art methods and was able to detect complex event patterns over videos with [Formula: see text]-Score ranging from 0.44 to 0.90. In the given experiments, the optimized VEKG-TAG was able to reduce 99% and 93% of VEKG nodes and edges, respectively, with 5.19[Formula: see text] faster search time, achieving sub-second median latency of 4–20[Formula: see text]ms.

Three-dimensional convolutional neural network model for tree species classification using airborne hyperspectral images

Airborne hyperspectral remote sensing data with both rich spectral and spatial features can effectively improve the classification accuracy of vegetation species. However, the spectral data of hundreds of bands brings about problems such as dimensional explosion, which poses a huge challenge for hyperspectral remote sensing classification based on classical parameters models. Deep learning methods have been used for remotely sensed images classification in recent years, but the popular HSI datasets including Kennedy Space Center, Indian Pines, Pavia University scene and Salinas scene, have low spatial resolution, significant differences between categories, and regular boundaries. When applied to the classification of forestry tree species, the accuracy often decreases because the spectral response of different plants of the same family and genus are very similar, especially under the fragmented species distribution, complex topography and the occluded canopy. So we collect new data sets, selected Gaofeng State Owned Forest Farm in Guangxi province in south China as the research area and adopted the airborne hyperspectral data obtained by the LiCHy system of the Chinese Academy of Forestry to explore an improved three-dimensional convolutional neural network(3D-CNN) model for tree species classification. The proposed model uses raw data as input without dimension reduction or feature screening, and simultaneously extracts spectral and spatial features. After the 3D convolutional layer, the captured high-level semantic concept is a joint spatial spectral feature representation, so we can turn it into a one-dimensional feature as a new input to learn a more abstract level of expression. The widely used earlystop method is also used to prevent overfitting. The proposed model is a lightweight, generalized, and fast convergence classification model, by which the short-time and large-area of multiple tree species classification with high-precision can be realized. The result shows that the 3D-1D CNN model can shorten the training time of the 3D CNN model by 60% and achieve a classification accuracy of 93.14% within 50 ha in 6.37 min, which provides a basis for the classification of tree species, the mapping of forest form and the inventory of forest resources.

Implementing Relevance Feedback for Content-Based Medical Image Retrieval

Content-based image medical retrieval (CBMIR) is a technique for retrieving medical images on the basis of automatically derived image features such as colour, texture and shape. There are many applications of CBMIR, such as teaching, research, diagnosis and electronic patient records. The retrieval performance of a CBMIR system depends mainly on the representation of image features, which researchers have studied extensively for decades. Although a number of methods and approaches have been suggested, it remains one of the most challenging problems in current (CBMIR) studies, largely due to the well-known “semantic gap” issue that exists between machine-captured low-level image features and human-perceived high-level semantic concepts. There have been many techniques proposed to bridge this gap. This study proposes a novel relevance feedback retrieval method (RFRM) for CBMIR. The feedback implemented here is based on voting values performed by each class in the image repository. Here, eighteen using colour moments and GLCM texture features were extracted to represent each image and eight common similarity coefficients were used as similarity measures. After briefly researching using a single random image query, the top images retrieved from each class are used as voters to select the most effective similarity coefficient that will be used for the final searching process. Our proposed method is implemented on the Kvasir dataset, which has 4,000 images divided into eight classes and was recently widely used for gastrointestinal disease detection. Intensive statistical analysis of the results shows that our proposed RFRM method has the best performance for enhancing both recall and precision when it uses any group of similarity coefficients.