Semantic Gap Problem Research Articles

Scale Enhancement Network for Object Detection in Aerial Images

The main challenge for object detection in aerial images is small object detection. Most existing methods use feature fusion strategies to enhance small object features in shallow layers but ignore the problem of inconsistent small object local region responses between feature layers, namely the semantic gap, which may lead to underutilization of small object information in multiple feature layers. To lift the above limitations, we propose a scale enhancement module that adaptively passes valuable small object features in different feature layers to shallow layers to alleviate the semantic gap problem. In particular, the module includes the novel fine-coarse self-attention mechanism, which captures global contextual information by performing strong interaction of pixel-level information at the local scale and weak interaction of region-level information at the global scale. In addition, the anchor assignment strategy based on the Intersection over Union (IoU) metric is not favorable for small objects as the IoU metric for small objects has a lower tolerance for position deviation compared to large ones. For this reason, we design the dynamic anchor assignment strategy with a scale-insensitive metric to assign adequate anchors to small objects. Extensive experiments on three aerial datasets demonstrate the effectiveness and adaptability of our method.

Knowledge-Based Reasoning Network for Relation Detection.

With the rapid growth of large-scale knowledge bases (KBs), knowledge base question answering (KBQA) has attracted increasing attention recently. Relation detection plays an important role in the KBQA system, which finds a compatible answer by analyzing the semantics of questions and querying and reasoning with multiple KB triples. Significant progress has been made by deep neural networks. However, existing methods often concern on detecting single-hop relation without path reasoning, and a few of these methods exploit the multihop relation reasoning, which involves the answer reasoning from the noisy and abundant relational paths in the KB. Meanwhile, the relatedness between question and answer candidates has received little attention and remains unsolved. This article proposes a novel knowledge-based reasoning network (KRN) for relation detection, including both single-hop relation and multihop relation. To address the semantic gap problem in question-answer interaction, we first learn attentive question representations according to the influence of answer aspects. Then, we learn the single-hop relation sequence through different levels of abstraction and adopt the KB entity and structure information to denoise the multihop relation detection task. Finally, we adopt a Siamese network to measure the similarity between question representation and relation representation for both single-hop and multihop relation KBQA tasks. We conduct experiments on two well-known benchmarks, SimpleQuestions and WebQSP, and the results show the superiority of our approach over the state-of-the-art models for both single-hop and multihop relation detection. Our model also achieves a significant improvement over existing methods on KBQA end task. Further analysis demonstrates the robustness and the applicability of the proposed approach.

Semantic-gap-oriented feature selection in hierarchical classification learning

Semantic gap is common issues in classification tasks. Different from traditional classification tasks, the semantic gap problem can introduce serious error propagation for hierarchical classification. In this paper, we propose a hierarchical feature selection method with label enhancement (HFSLE) to solve this challenge, which estimates the label distribution and selects subsets of features for each subtask. Firstly, HFSLE combines the label enhancement and hierarchical feature selection process so that the label enhancement benefits from the feedback of the loss function. Then, the label enhancement is constrained by the consistency of the parent-child distribution to obtain a plausible label distribution. Finally, feature selection is optimized based on intra-class consistency hierarchical regularization terms, which unifies feature subsets with hierarchical semantic descriptions to narrow the semantic gap. HFSLE gives a new perspective on the hierarchical feature selection based on label distribution rather than the original logical label. To demonstrate the superiority of the method, we compared it with six well-established feature selection methods on eight hierarchical data sets, and analyzed the convergence and robustness of the method. Extensive experiments demonstrate that the proposed method can perform favorably against advanced hierarchical feature selection approaches.

MM-FOOD: a high-dimensional index structure for efficiently querying content and concept of multimedia data

The semantic query problem is commonly called the semantic gap and is one of the significant problems in multimedia data retrieval. In this study, we focus on multimedia data retrieval by combining semantic information with data content to solve the semantic gap problem effectively. The main idea behind the combination of low-level content descriptors and the concept of multimedia data is to represent the content information with the semantic information by adding a low-level content descriptor as a new dimension to the index structure. This new dimension is represented by constructing an array index structure that uses a fuzzy clustering algorithm. Thus, a new high-dimensional index structure, named MM-FOOD, supporting querying of multimedia data, including fuzzy querying, is presented in this paper. This proposed index structure’s construction and query algorithms are explained throughout this paper. Our experiments show that our indexing mechanism is considerably efficient compared to the basic indexing approach, which stores low-level content and semantic concept descriptors in separate structures when the data size is large.

Shrinking the Semantic Gap: Spatial Pooling of Local Moment Invariants for Copy-Move Forgery Detection

Copy-move forgery is a manipulation of copying and pasting specific patches from and to an image, with potentially illegal or unethical uses. Recent advances in the forensic methods for copy-move forgery have shown increasing success in detection accuracy and robustness. However, for images with high self-similarity or strong signal corruption, the existing algorithms often exhibit inefficient processes and unreliable results. This is mainly due to the inherent semantic gap between low-level visual representation and high-level semantic concept. In this paper, we present a very first study of trying to mitigate the semantic gap problem in copy-move forgery detection, with spatial pooling of local moment invariants for midlevel image representation. Our detection method expands the traditional works on two aspects: 1) we introduce the bag-of-visual-words model into this field for the first time, may meaning a new perspective of forensic study; 2) we propose a word-to-phrase feature description and matching pipeline, covering the spatial structure and visual saliency information of digital images. Extensive experimental results show the superior performance of our framework over state-of-the-art algorithms in overcoming the related problems caused by the semantic gap.

Wave-Net: A lightweight deep network for retinal vessel segmentation from fundus images

Accurate segmentation of retinal vessels from fundus images is fundamental for the diagnosis of numerous diseases of eye, and an automated vessel segmentation method can effectively help clinicians to make accurate diagnosis for the patients and provide the appropriate treatment schemes. It is important to note that both thick and thin vessels play the key role for disease judgements. Because of complex factors, the precise segmentation of thin vessels is still a great challenge, such as the presence of various lesions, image noise, complex backgrounds and poor contrast in the fundus images. Recently, because of the advantage of context feature representation learning capabilities, deep learning has shown a remarkable segmentation performance on retinal vessels. However, it still has some shortcomings on high-precision retinal vessel extraction due to some factors, such as semantic information loss caused by pooling operations, limited receptive field, etc. To address these problems, this paper proposes a new lightweight segmentation network for precise retinal vessel segmentation, which is called as Wave-Net model on account of the whole shape. To alleviate the influence of semantic information loss problem to thin vessels, to acquire more contexts about micro structures and details, a detail enhancement and denoising block (DED) is proposed to improve the segmentation precision on thin vessels, which replaces the simple skip connections of original U-Net. On the other hand, it could well alleviate the influence of the semantic gap problem. Further, faced with limited receptive field, for multi-scale vessel detection, a multi-scale feature fusion block (MFF) is proposed to fuse cross-scale contexts to achieve higher segmentation accuracy and realize effective processing of local feature maps. Experiments indicate that proposed Wave-Net achieves an excellent performance on retinal vessel segmentation while maintaining a lightweight network design compared to other advanced segmentation methods, and it also has shown a better segmentation ability to thin vessels.

Zero-Shot Learning via Structure-Aligned Generative Adversarial Network.

In this article, we propose a structure-aligned generative adversarial network framework to improve zero-shot learning (ZSL) by mitigating the semantic gap, domain shift, and hubness problem. The proposed framework contains two parts, i.e., a generative adversarial network with a softmax classifier part, and a structure-aligned part. In the first part, the generative adversarial network aims at generating pseudovisual features through the guiding generator and discriminator play the minimax two-player game together. At the same time, the softmax classifier is committed to increasing the interclass distance and reducing intraclass distance. Then, the harmful effect of domain shift and hubness problems can be mitigated. In another part, we introduce a structure-aligned module where the structural consistency between visual space and semantic space is learned. By aligning the structure between visual space and semantic space, the semantic gap between them can be bridged. The performance of classification is improved when the structure-aligned visual-semantic embedding space is transferred to the unseen classes. Our framework reformulates the ZSL as a standard fully supervised classification task using the pseudovisual features of unseen classes. Extensive experiments conducted on five benchmark data sets demonstrate that the proposed framework significantly outperforms state-of-the-art methods in both conventional and generalized settings.

A hybrid CBIR system using novel local tetra angle patterns and color moment features

The exponential growth of communal media platforms including Facebook and Twitter, and the accessibility of low-cost digital capturing devices have generated an enormous number of multimedia content including images. Effective handling of such massive image collection has boosted the development of content-based image retrieval (CBIR) approaches. Researchers have suggested both machine learning and non-learning-based techniques for CBIR. However, machine learning-based methods outperform the non-learning-based methods in the CBIR domain. The CBIR demands the development of reliable descriptors to attain the most appropriate images from the depository and better address the semantic gap problem. To counter these problems, we suggest a novel second-order Local Tetra Angle Patterns (LTAP) to better capture the texture features from the image. LTAPs are computed from adjacent pixels of 0°, 45°, 90°, and 135° using the second-order directional derivatives. Further, we propose a hybrid feature vector by concatenating LTAPs and RGB color features and using the genetic algorithm (GA) to select the finest appropriate features that enhance the image retrieval performance of our system. We employed our hybrid descriptor with the GA to optimize the support vector machine (SVM) for the image classification task and used the Chi-square quadratic distance measure to determine the resemblance between the query image and the images in the repository. Experimental results on three standard datasets including the Corel 1 k, Oxford flower, and CIFAR-10 indicate the effectiveness of the presented system over the contemporary CBIR methods.

R2U++: a multiscale recurrent residual U-Net with dense skip connections for medical image segmentation

U-Net is a widely adopted neural network in the domain of medical image segmentation. Despite its quick embracement by the medical imaging community, its performance suffers on complicated datasets. The problem can be ascribed to its simple feature extracting blocks: encoder/decoder, and the semantic gap between encoder and decoder. Variants of U-Net (such as R2U-Net) have been proposed to address the problem of simple feature extracting blocks by making the network deeper, but it does not deal with the semantic gap problem. On the other hand, another variant UNET++ deals with the semantic gap problem by introducing dense skip connections but has simple feature extraction blocks. To overcome these issues, we propose a new U-Net based medical image segmentation architecture R2U++. In the proposed architecture, the adapted changes from vanilla U-Net are: (1) the plain convolutional backbone is replaced by a deeper recurrent residual convolution block. The increased field of view with these blocks aids in extracting crucial features for segmentation which is proven by improvement in the overall performance of the network. (2) The semantic gap between encoder and decoder is reduced by dense skip pathways. These pathways accumulate features coming from multiple scales and apply concatenation accordingly. The modified architecture has embedded multi-depth models, and an ensemble of outputs taken from varying depths improves the performance on foreground objects appearing at various scales in the images. The performance of R2U++ is evaluated on four distinct medical imaging modalities: electron microscopy, X-rays, fundus, and computed tomography. The average gain achieved in IoU score is 1.5 ± 0.37% and in dice score is 0.9 ± 0.33% over UNET++, whereas, 4.21 ± 2.72 in IoU and 3.47 ± 1.89 in dice score over R2U-Net across different medical imaging segmentation datasets.

Visual Emotion Analysis via Affective Semantic Concept Discovery

With the development of social media, people prefer to express views and share daily life online via visual content, which has led to widespread attention in automatic emotion analysis from images. Capturing the emotions embedded in these social images has always been important yet challenging. In this paper, we propose a visual emotion prediction method that utilizes the affective semantic concepts of an image to predict its emotion. To solve the problems of narrow semantic coverage and low discriminative power of emotions in current semantic concept sets used for visual emotion analysis, we develop a concept selection model to mine emotion-related concepts from social media. Specifically, we propose several selection strategies to build an affective semantic concept set that contains various visual concepts related to emotion conveyance. And they are discovered from affective image datasets and associated tags crawled from websites. To further leverage the discovered affective semantic concepts, we train concept classifiers to predict the concept score of each concept, which are used as the intermediate features to tackle the semantic gap problem for image emotion recognition. Extensive experimental results confirm the validity of the affective semantic concepts and show the improved performance of our method.

An End-to-End Deep Learning Approach for Diagnosis of Parkinson’s Disease Based on Hand Drawing

At present, Parkinson’s disease is mainly subject to the doctor’s subjective assessment of the patient’s symptoms through the unified Parkinson’s disease score scale, which easily leads to a high rate of misdiagnosis. Some wearable devices have been developed to assist in providing accurate diagnostic results. However, this process requires the cooperation of patients and doctors, which is very inconvenient for Parkinson’s patients who have limited mobility. According to reports, Parkinson’s patients will show some writing disabilities in the early stages. Therefore, to provide a non-invasive and convenient diagnosis program, this research is devoted to developing a Parkinson’s disease diagnosis program based on hand-drawn data. This search provides diagnostic solutions for Convolutional Neural Network (CNN) and Residual Neural Network (ResNet). Unlike the traditional hand drawing preprocessing scheme that only retains the trajectory, the color information of the image for the hand drawing was reserved, which allows the subsequent model to learn the stroke pressure, speed, stroke time, and other based on the color information. Compared with CNN, the ResNet can better solve the semantic gap problem, achieve faster convergence speed, and get higher diagnostic accuracy (accuracy = 88.9%). In general, the two diagnostic schemes proposed in this study only require the patient to provide a hand drawing to confirm whether the patient has Parkinson’s disease, which is of great significance for the convenient diagnosis of Parkinson’s disease.

Zero-Shot Image Classification Based on a Learnable Deep Metric.

The supervised model based on deep learning has made great achievements in the field of image classification after training with a large number of labeled samples. However, there are many categories without or only with a few labeled training samples in practice, and some categories even have no training samples at all. The proposed zero-shot learning greatly reduces the dependence on labeled training samples for image classification models. Nevertheless, there are limitations in learning the similarity of visual features and semantic features with a predefined fixed metric (e.g., as Euclidean distance), as well as the problem of semantic gap in the mapping process. To address these problems, a new zero-shot image classification method based on an end-to-end learnable deep metric is proposed in this paper. First, the common space embedding is adopted to map the visual features and semantic features into a common space. Second, an end-to-end learnable deep metric, that is, the relation network is utilized to learn the similarity of visual features and semantic features. Finally, the invisible images are classified, according to the similarity score. Extensive experiments are carried out on four datasets and the results indicate the effectiveness of the proposed method.

Hierarchical feature selection with multi-granularity clustering structure

Hierarchical feature selection addresses the issues caused by the presence of high-dimensional features in multi-category classification systems with hierarchical structures. Granular calculations are made to analyze the hierarchical relationships among categories when selecting the optimal feature subset. However, semantic hierarchy-based feature selection methods are prone to the semantic gap problem, which affects classification accuracy. In this paper, we propose a hierarchical feature selection method with a multi-granularity clustering structure that can effectively alleviate the semantic gap problem. Firstly, a hierarchical structure is constructed via bottom-up multi-granularity clustering based on feature similarities rather than semantic categories. This clustering hierarchy is conducive to solving semantic gap problems in the existing hierarchy. Secondly, the optimal feature subset is selected using the ℓ1,2-norms in each hierarchy’s granularity layer. This joint minimization approach can retain both the granularity layers’ shared features and granularity-specific features. Finally, we execute hierarchical classification according to the granular structure in a coarse to fine sequence. Extensive experiments demonstrate that the proposed method outperforms several state-of-the-art hierarchical feature selection approaches.

Graph Regularized Hierarchical Diffusion Process With Relevance Feedback for Medical Image Retrieval

Befitting from the interpretability and the capacity in capturing the underlying manifold structure, diffusion process (DP) has attracted increasing attention in the field of image retrieval. Within it, hierarchical diffusion process (HDP) has achieved satisfactory results in retrieved performance and complexity. However, the existing hierarchical diffusion process methods only diffuse the affinity values in low-level visual space without considering the high-level semantic information, which cause the problem of semantic gap. To overcome these problems, we propose a Graph Regularized Hierarchical Diffusion Process (GRHDP) method with relevance feedback, and apply it to retrieve medical images. The proposed algorithm firstly establishes a hierarchical structure of the images in medical image database and spreads the affinity values among query images and top-layer images by graph regularization diffusion. Then relevance feedback is introduced to adjust the similarity between query images and retrieved images in top layer, and the affinity values are diffused again according to labeled information of feedback. Finally, the similarity between queries and others in database can be obtained by interpolating the diffused results on the top layer from top to bottom. The experimental results show that our proposed GRHDP with relevance feedback has achieved better retrieval performance than manifold ranking and regularized diffusion process (RDP) when returning top retrieved images.

Directional Magnitude Local Hexadecimal Patterns: A Novel Texture Feature Descriptor for Content-Based Image Retrieval

Social media platforms such as Twitter, Facebook, and Flicker, and the evolution of digital image capturing devices have resulted in the generation of a massive number of images. Thus, we experienced an exponential growth in digital image repositories in the last decade. Content-based image retrieval (CBIR) has been extensively employed to reduce the dependency on textual annotations for image searching. Effective feature descriptor is mandatory to retrieve the most relevant images from the repository. Additionally, CBIR methods often experience the semantic gap problem, which must also be addressed. In this paper, we propose a novel texture descriptor, Directional Magnitude Local Hexadecimal Patterns (DMLHP), based on the texture orientation and magnitude to retrieve the most relevant images. The objective of the proposed feature descriptor is to examine the relationship between the neighboring pixels and their adjacent neighbors based on texture orientation and magnitude. Our DMLHP texture descriptor is capable of capturing the texture and semantic information of the images effectively with the same visual content. Furthermore, the proposed method employs a learning-based approach to lessen the semantic gap problem and to improve the understanding of the contents of query images to retrieve the most relevant images. The presented descriptor provides remarkable results by achieving the average retrieval precision (ARP) of 66%, 92%, 83%, average retrieval recall (ARR) of 66%, 92%, 83%, average retrieval specificity (ARS) of 99%, 99%, 76%, and average retrieval accuracy (ARA) of 98%, 99%, 85% on the AT&T, MIT Vistex, and Brodatz Texture image repositories, respectively. Our experiments reveal that the proposed DMLHP descriptor achieves far better performance, i.e., 95% on AT&T, 92% on BT, and 99% on MIT Vistex, when used with a learning-based approach over a non-learning-based approach (similarity measure). Experimental results show that the proposed texture descriptor outperforms state-of-the-art descriptors such as LNIP, LTriDP, LNDP, LDGP, LEPSEG, and CSLBP for CBIR.

Multilabel Remote Sensing Image Annotation With Multiscale Attention and Label Correlation

Deep-learning-based multilabel image annotation is receiving increasing attention in the field of remote sensing due to the great success of deep networks in single-label remote sensing image classification. Compared with those low-level features, the features extracted by the convolutional neural network (CNN) are more informative and can alleviate the problem of semantic gap. However, the CNN model tends to ignore the smaller objects when objects of different sizes exist in an image. In addition, how to efficiently leverage the correlation among multiple labels to enhance annotation performance remains an open issue. In this article, we propose an end-to-end deep learning framework for multilabel remote sensing image annotation. The framework is composed of a multiscale feature fusion module, a channel-spatial attention learning module, and a label correlation extraction module. The multiscale features from different layers of a CNN model are first fused and refined by using a channel-spatial attention mechanism. Then, the label correlation information is extracted from a label co-occurrence matrix and embedded into the multiscale attentive features to increase the discriminative ability of the resulting image features. The experiments on two benchmark datasets demonstrate the superiority of the proposed method in comparison with the state-of-the-art methods.

Multimodal Wireless Situational Awareness‐Based Tourism Service Scene

Community platforms featuring user sharing and self‐expression in social media generate big data on tourism resources, which, if fully utilized in a smart tourism system driven by high‐tech and new technologies, will bring new life to the field of smart tourism research and will play an important role in the development of Internet+ tourism. However, tourism data in social media has the following characteristics: diversity, redundancy, heterogeneity, and intelligence. To address the characteristics of tourism data in social media, this thesis focuses on the following challenges: it is difficult to efficiently obtain tourism visualization information (text and images) in social media; it is difficult to effectively utilize tourism multimodal heterogeneous information; it is difficult to properly retrieve multimedia entity information of tourism attractions; and it is difficult to reasonably construct tourism personalized recommendation models. In this paper, an image search reordering method based on a hybrid feature graph model is proposed to realize the rapid acquisition of high‐quality Internet images from the web using hybrid visual features and graph models, thus providing data security for the analysis of social media‐based tourism images. To address the shortcomings of current search engines for image retrieval, visual information is used to bridge the problem of semantic gap between text‐based search and images. To address the limitation of single visual features, we use latent semantic analysis to fuse multiple visual features to obtain hybrid features, which not only combine multiple single features but also preserve the potential relationship between these features. To address the shortcomings of the reordering methods based on classification and clustering, a reordering framework based on the graph model is used to reorder the images and finally complete the image search reordering based on the hybrid feature graph model. This method can obtain image information in social media with high efficiency and quality and then prepare for the subsequent work of tourism image analysis mining and personalized recommendation.

Low-rank embedded orthogonal subspace learning for zero-shot classification

Zero-shot classification methods have attracted considerable attention in recent years. Existing ZSC methods encounter domain shift, hubness and visual–semantic gap problems. To address these problems, we propose a low-rank embedded orthogonal subspace learning method (LEOSL) for ZSC. Many previous works project visual features to the semantic space. However, they often suffer from the visual–semantic gap problem. To handle this problem, we project the visual representations and semantic representations to the common subspace. To address the domain shift problem, we restrict the mapping functions with a low-rank constraint. To handle the hubness problem, we introduce the class similarity term so that samples of the same class are located near each other, while samples of different classes are located far away. Furthermore, we restrict the shared representations in the subspace with an orthogonal constraint to remove the correlation between samples. The results show the superiority of LEOSL compared to many state-of-the-art methods.

Learning Expression Features via Deep Residual Attention Networks for Facial Expression Recognition From Video Sequences

Facial expression recognition from video sequences is currently an interesting research topic in computer vision, pattern recognition, artificial intelligence, etc. Considering the problem of semantic gap between the extracted hand-designed features in affective videos and subjective emotions, recognizing facial expressions from video sequences is a challenging subject. To tackle this problem, this paper proposes a new method of facial expression recognition from video sequences via deep residual attention network. Firstly, due to the difference in the intensity of emotional representation of each local area in a facial image, deep residual attention networks are employed to learn high-level affective expression features for each frame of facial expression images in video sequences. The used deep residual attention networks integrate deep residual networks with a spatial attention mechanism. Then, average-pooling is performed to produce fixed-length global video-level feature representations. Finally, the global video-level feature representations are utilized as inputs of a multi-layer perceptron to conduct facial expression classification tasks in video sequences. Experimental results on two public video emotional datasets, i.e. BAUM-1s and RML, demonstrate the effectiveness of the proposed method.

Relation classification via knowledge graph enhanced transformer encoder

Relation classification is an important task in natural language processing fields. The goal is to predict predefined relations for the marked nominal pairs in given sentences. State-of-the-art works usually focus on using deep neural networks as classifier to conduct the relation prediction. The rich semantic information of relationships in the triples of existing knowledge graph (KG) can be used as additional supervision for relation classification. However, these relationships were simply used as labels to specify the class of sentences in previous works, and their semantic information was completely ignored. In this paper, a novel approach is proposed for relation classification, which jointly uses information from textual sentences and knowledge graphs. To this end, we introduce a Transformer encoder to measure the semantic similarity between sentences and relation types. Besides, we connect the semantic information of marked nominals in sentences with that of the corresponding entities in knowledge graph to generate the semantic matching information between textual relations and KG relations. The matching information can provide additional supervision for relation classification. Since the words and entities are used interactively with each other in our work, we propose an embedding translating strategy to handle the semantic gap problem between word embeddings and entity embeddings. Experimental results on two widely used datasets, SemEval-2010 Task 8 and TACRED, show that our approach is able to efficiently use the semantic information from the knowledge graph to enhance the performance of the Transformer encoder for relation classification.