Semantic Enhancement Research Articles

Semantic-enhanced graph neural network for named entity recognition in ancient Chinese books

Named entity recognition (NER) plays a crucial role in the extraction and utilization of knowledge of ancient Chinese books. However, the challenges of ancient Chinese NER not only originate from linguistic features such as the use of single characters and short sentences but are also exacerbated by the scarcity of training data. These factors together limit the capability of deep learning models, like BERT-CRF, in capturing the semantic representation of ancient Chinese characters. In this paper, we explore the semantic enhancement of NER in ancient Chinese books through the utilization of external knowledge. We propose a novel model based on Graph Neural Networks that integrates two different forms of external knowledge: dictionary-level and chapter-level information. Through the Graph Attention Mechanism (GAT), these external knowledge are effectively incorporated into the model’s input context. Our model is evaluated on the C_CLUE dataset, showing an improvement of 3.82% over the baseline BAC-CRF model. It also achieves the best score compared to several state-of-the-art dictionary-augmented models.

Cross attention is all you need: relational remote sensing change detection with transformer

ABSTRACT Deep-learning-based change detection methods have received wide attention, thanks to their strong capability in obtaining rich features from images. However, existing AI-based change detection (CD) methods largely rely on three functionality-enhancing modules, that is, semantic enhancement, attention mechanisms, and correspondence enhancement. The stacking of these modules leads to great model complexity. To unify these three modules into a simple pipeline, we introduce relational change detection transformer (RCDT), a novel and simple framework for remote sensing change detection tasks. The proposed RCDT consists of three major components, a weight-sharing Siamese Backbone to obtain bi-temporal features, a relational cross attention module (RCAM) that implements offset cross attention to obtain bi-temporal relation-aware features, and a features constrain module (FCM) to achieve the final refined predictions with high-resolution constraints. Extensive experiments on four different publicly available datasets suggest that our proposed RCDT exhibits superior change detection performance and great trade-off between performance and complexity compared with other competing methods. The theoretical, methodological, and experimental knowledge of this study is expected to benefit future change detection efforts that involve the cross attention mechanism.

A lightweight ground crack rapid detection method based on semantic enhancement

To address the issue of detecting complex-shaped cracks that rely on manual, which may result in high costs and low efficiency, this paper proposed a lightweight ground crack rapid detection method based on semantic enhancement. Firstly, the introduction of the Context Guided Block module enhanced the YOLOv8 backbone network, improving its feature extraction capability. Next, the incorporation of GSConv and VoV-GSCSP was introduced to construct a lightweight yet efficient neck network, facilitating the effective fusion of information from multiple feature maps. Finally, the detection head achieved more precise target localization by optimizing the probability around the labels. The proposed method was validated through experiments on the public dataset RDD-2022. The experimental results demonstrate that our method effectively detects cracks. Compared to YOLOv8, the model parameters have been reduced by 73.5 %, while accuracy, F1 score, and FPS have improved by 6.6 %, 4.3 %, and 116, respectively. Therefore, our proposed method is more lightweight and holds significant application value.

Enhancing text-based knowledge graph completion with zero-shot large language models: A focus on semantic enhancement

The design and development of text-based knowledge graph completion (KGC) methods leveraging textual entity descriptions are at the forefront of research. These methods involve advanced optimization techniques such as soft prompts and contrastive learning to enhance KGC models. The effectiveness of text-based methods largely hinges on the quality and richness of the training data. Large language models (LLMs) can utilize straightforward prompts to alter text data, thereby enabling data augmentation for KGC. Nevertheless, LLMs typically demand substantial computational resources. To address these issues, we introduce a framework termed constrained prompts for KGC (CP-KGC). This CP-KGC framework designs prompts that adapt to different datasets to enhance semantic richness. Additionally, CP-KGC employs a context constraint strategy to effectively identify polysemous entities within KGC datasets. Through extensive experimentation, we have verified the effectiveness of this framework. Even after quantization, the LLM (Qwen-7B-Chat-int4) still enhances the performance of text-based KGC methods.11Code and datasets are available at https://github.com/sjlmg/CP-KGC. This study extends the performance limits of existing models and promotes further integration of KGC with LLMs.

Multi-view and region reasoning semantic enhancement for image-text retrieval

Multi-view and region reasoning semantic enhancement for image-text retrieval

High-level vision task-driven infrared and visible image fusion approach: Progressive Semantic Enhancement based Multi-scale Cross-modality Interactive Network

Infrared-visible image fusion aims to merge data for a more accurate scene representation. Current research focuses mainly on enhancing visual appeal rather than improving performance in high-level vision tasks. To address this gap, we propose the Semantic Enhanced Multi-scale Cross-modality Interactive Image Fusion Network (SeMIFusion). Initially, Multi-scale Cross-modality Feature Fusion (MCFF) module is devised to extract shallow and deep features across different modalities. During feature extraction, Texture Enhancer (TE) and Semantify Enhancer (SE) blocks capture diverse hierarchical features across multi-scale layers, seamlessly integrating into the Semantic Feature Integration (SFI) module for profound semantic information extraction. Furthermore, an Image Scene Reconstruction (ISR) module maintains original image details in fused features, ensuring image fidelity. Additionally, incorporating a visual preservation guiding mask prioritizes retaining visual quality during reconstruction, preventing degradation. Extensive experiments demonstrate our method’s superiority in preserving visual effects and texture details, especially in high-level vision tasks. The source code is available at https://github.com/yyzzttkkjj/SeMIFusion.

Multi-scale Lesion Feature Fusion and Location-Aware for Chest Multi-disease Detection.

Accurately identifying and locating lesions in chest X-rays has the potential to significantly enhance diagnostic efficiency, quality, and interpretability. However, current methods primarily focus on detecting of specific diseases in chest X-rays, disregarding the presence of multiple diseases in a single chest X-ray scan. Moreover, the diversity in lesion locations and attributes introduces complexity in accurately discerning specific traits for each lesion, leading to diminished accuracy when detecting multiple diseases. To address these issues, we propose a novel detection framework that enhances multi-scale lesion feature extraction and fusion, improving lesion position perception and subsequently boosting chest multi-disease detection performance. Initially, we construct a multi-scale lesion feature extraction network to tackle the uniqueness of various lesion features and locations, strengthening the global semantic correlation between lesion features and their positions. Following this, we introduce an instance-aware semantic enhancement network that dynamically amalgamates instance-specific features with high-level semantic representations across various scales. This adaptive integration effectively mitigates the loss of detailed information within lesion regions. Additionally, we perform lesion region feature mapping using candidate boxes to preserve crucial positional information, enhancing the accuracy of chest disease detection across multiple scales. Experimental results on the VinDr-CXR dataset reveal a 6% increment in mean average precision (mAP) and an 8.4% improvement in mean recall (mR) when compared to state-of-the-art baselines. This demonstrates the effectiveness of the model in accurately detecting multiple chest diseases by capturing specific features and location information.

Effective small crack detection based on tunnel crack characteristics and an anchor-free convolutional neural network

Tunnel cracks are thin and narrow linear targets, and their pixel proportions in images are usually very low, less than 6%; therefore, a method is needed to better detect small crack targets. In this study, a crack detection method based on crack characteristics and an anchor-free framework is investigated. First, the characteristics of cracks are analyzed to obtain the real crack texture, interference noise texture, and targets appearing near each crack as the context information for the model to filter and remove noise. We discuss the crack detection performance of anchor-based and anchor-free algorithms. Then, an optimized anchor-free algorithm is proposed in this paper for crack detection. Based on the advantages of YOLOX-x, we add a semantic enhancement module to better use contextual information. The experimental results show that the anchor-free algorithm performs slightly better than other algorithms in crack detection situations. In addition, the proposed method displays better detection performance for slender and inconspicuous cracks, with an average precision of 0.858.

Crash severity analysis: A data-enhanced double layer stacking model using semantic understanding

The crash severity analysis is of significant importance in traffic crash prevention and emergency resource allocation. A range of innovations offers potential traffic crash severity prediction models to improve road safety. However, the semantic information inherent in traffic crash data, which is crucial in enabling a deeper understanding of its underlying factors and impacts, has yet to be fully utilized. Moreover, traffic crash data are commonly characterized by a small sample size, which leads to sample imbalance problem resulting in prediction performance decline. To tackle these problems, we propose a semantic understanding-based data-enhanced double-layer stacking model, named EnLKtreeGBDT, for crash severity prediction. Specifically, to fully leverage the inherent semantic information within traffic crash data and analyze the factors influencing crashes, we design a semantic enhancement module for multi-dimensional feature extraction. This module aims to enhance the understanding of crash semantics and improve prediction accuracy. Then we introduce a data enhancement module that utilizes data denoising and migration techniques to address the challenge of data imbalance, reducing the prediction model's dependence on large sample crash data. Furthermore, we construct a two-layer stacking model that combines multiple linear and nonlinear classifiers. This model is designed to augment the capability of learning linear and nonlinear mixed relationships, thereby improving the accuracy of predicting the severity of crashes on complex urban roads. Experiments on historical datasets of UK road safety crashes validate the effectiveness of the proposed model, and superior performance of prediction precision is achieved compared with the state-of-the-arts. The ablation experiments on both semantic and data enhancement modules further confirm the indispensability of each module in the proposed model.

PointeNet: A lightweight framework for effective and efficient point cloud analysis

The conventional wisdom in point cloud analysis predominantly explores 3D geometries. It is often achieved through the introduction of intricate learnable geometric extractors in the encoder or by deepening networks with repeated blocks. However, these methods contain a significant number of learnable parameters, resulting in substantial computational costs and imposing memory burdens on CPU/GPU. Moreover, they are primarily tailored for object-level point cloud classification and segmentation tasks, with limited extensions to crucial scene-level applications, such as autonomous driving. To this end, we introduce PointeNet, an efficient network designed specifically for point cloud analysis. PointeNet distinguishes itself with its lightweight architecture, low training cost, and plug-and-play capability, while also effectively capturing representative features. The network consists of a Multivariate Geometric Encoding (MGE) module and an optional Distance-aware Semantic Enhancement (DSE) module. MGE employs operations of sampling, grouping, pooling, and multivariate geometric aggregation to lightweightly capture and adaptively aggregate multivariate geometric features, providing a comprehensive depiction of 3D geometries. DSE, designed for real-world autonomous driving scenarios, enhances the semantic perception of point clouds, particularly for distant points. Our method demonstrates flexibility by seamlessly integrating with a classification/segmentation head or embedding into off-the-shelf 3D object detection networks, achieving notable performance improvements at a minimal cost. Extensive experiments on object-level datasets, including ModelNet40, ScanObjectNN, ShapeNetPart, and the scene-level dataset KITTI, demonstrate the superior performance of PointeNet over state-of-the-art methods in point cloud analysis. Notably, PointeNet outperforms PointMLP with significantly fewer parameters on ModelNet40, ScanObjectNN, and ShapeNetPart, and achieves a substantial improvement of over 2% in 3DAPR40 for PointRCNN on KITTI with a minimal parameter cost of 1.4 million. Code is publicly available at https://github.com/lipeng-gu/PointeNet.

Zero-shot discrete hashing with adaptive class correlation for cross-modal retrieval

Zero-shot retrieval aims to transfer knowledge from seen classes to unseen classes by embedding semantic information on class attributes, thus solving the unseen class retrieval problem. However, existing works have focused mainly on unimodal zero-shot retrieval tasks. In this paper, we introduce an efficient method, termed zero-shot discrete hashing with adaptive class correlation (ZSDH-ACC), to speed up cross-modal retrieval. Specifically, this proposed method combines label information with class attribute information to construct a semantic enhancement embedding, in which the problem of class attribute correspondence of multilabel instances can be solved. Furthermore, we learn semantic enhancement embedding to merge more semantic information for feature representation, and its goal is to learn more discriminative hash codes and hash functions. In addition, our proposed method adaptively learns the correlation between class attributes and then embeds more class attribute information into hash codes. Finally, pairwise similarity is used to constrain the learning of hash codes, and thus more discriminative hash codes can be generated. Extensive experimental results on four benchmark multimodal datasets demonstrate that the proposed ZSDH-ACC method can achieve encouraging performance in cross-modal retrieval tasks. The source code of this paper can be obtained from https://github.com/szq0816/ZSDH_ACC.

Semi-supervised semantic labeling of remote sensing images with improved image-level selection retraining

In recent years, image semantic segmentation technology has developed rapidly, but image annotation usually requires a significant amount of human and financial resources, especially for remote sensing image annotation, which can be expensive and sometimes even unaffordable. To address this issue, this paper integrates the idea of curriculum learning into the self-training method and screens reliable pseudo-labels through computing image-level confidence, significantly reducing the confirmation error problem. Furthermore, the semi-supervised model in this paper combines implicit semantic enhancement with strong data augmentation, which can reduce the coupling between the teacher model and the student model’s prediction distribution and enhance the model’s robustness. Finally, the proposed semi-supervised method is experimentally verified using the ISPRS competition dataset and compared with existing state-of-the-art (SOTA) methods. Experimental results show that the proposed semi-supervised segmentation method achieves higher segmentation accuracy compared to self-training methods. Moreover, despite not using iterative training to simplify the training process, the proposed method still yields satisfactory segmentation results.

A novel network for semantic segmentation of landslide areas in remote sensing images with multi-branch and multi-scale fusion

Landslides pose significant risks as natural disasters, highlighting the importance of accurate mapping using remote sensing images for various practical applications. However, due to the challenges arising from incomplete and inaccurate boundary information of foreground landslide polygons, existing methods can only achieve suboptimal performance. To this premise, in this paper, we propose a segmentation network called GMNet that leverages global information extraction and multi-scale feature fusion to enhance the discrimination of landslides from other objects. Specifically, by employing a multi-branch mechanism, our method effectively captures global information, while an improved multi-scale feature fusion technique addresses the issue of varying scales in landslide polygons. Furthermore, semantic enhancement enhances the semantic information of low-level features, bridging the semantic gap and enhancing fusion efficacy. Experimental results demonstrate the effectiveness of our network in segmenting landslide areas accurately within the remote sensing image dataset. Especially, our F1_scores on three benchmarks outperform existing runner-ups by notable margins of 4.81%, 1.72%, and 1.16%, showcasing the value of our method in this domain.

GSENet:Global Semantic Enhancement Network for Lane Detection

Lane detection is the cornerstone of autonomous driving. Although existing methods have achieved promising results, there are still limitations in addressing challenging scenarios such as abnormal weather, occlusion, and curves. These scenarios with low visibility usually require to rely on the broad information of the entire scene provided by global semantics and local texture information to predict the precise position and shape of the lane lines. In this paper, we propose a Global Semantic Enhancement Network for lane detection, which involves a complete set of systems for feature extraction and global features transmission. Traditional methods for global feature extraction usually require deep convolution layer stacks. However, this approach of obtaining global features solely through a larger receptive field not only fails to capture precise global features but also leads to an overly deep model, which results in slow inference speed. To address these challenges, we propose a novel operation called the Global feature Extraction Module (GEM). Additionally, we introduce the Top Layer Auxiliary Module (TLAM) as a channel for feature distillation, which facilitates a bottom-up transmission of global features. Furthermore, we introduce two novel loss functions: the Angle Loss, which account for the angle between predicted and ground truth lanes, and the Generalized Line IoU Loss function that considers the scenarios where significant deviations occur between the prediction of lanes and ground truth in some harsh conditions. The experimental results reveal that the proposed method exhibits remarkable superiority over the current state-of-the-art techniques for lane detection.Our codes are available at:https://github.com/crystal250/GSENet.

A retinal vessel segmentation network with multiple-dimension attention and adaptive feature fusion

The incidence of blinding eye diseases is highly correlated with changes in retinal morphology, and is clinically detected by segmenting retinal structures in fundus images. However, some existing methods have limitations in accurately segmenting thin vessels. In recent years, deep learning has made a splash in the medical image segmentation, but the lack of edge information representation due to repetitive convolution and pooling, limits the final segmentation accuracy. To this end, this paper proposes a pixel-level retinal vessel segmentation network with multiple-dimension attention and adaptive feature fusion. Here, a multiple dimension attention enhancement (MDAE) block is proposed to acquire more local edge information. Meanwhile, a deep guidance fusion (DGF) block and a cross-pooling semantic enhancement (CPSE) block are proposed simultaneously to acquire more global contexts. Further, the predictions of different decoding stages are learned and aggregated by an adaptive weight learner (AWL) unit to obtain the best weights for effective feature fusion. The experimental results on three public fundus image datasets show that proposed network could effectively enhance the segmentation performance on retinal blood vessels. In particular, the proposed method achieves AUC of 98.30%, 98.75%, and 98.71% on the DRIVE, CHASE_DB1, and STARE datasets, respectively, while the F1 score on all three datasets exceeded 83%. The source code of the proposed model is available at https://github.com/gegao310/VesselSeg-Pytorch-master.

Hyperspectral image classification method based on narrowing semantic gap convolutional neural network

ABSTRACT In recent years, numerous hyperspectral image classification methods based on convolutional neural network (CNN) have been proposed in recent years. However, the majority of CNN-based classification models fail to take into account the semantic gap between shallow and deep features, and instead, just sum up or splice these two types of features, which leads to subpar feature fusion. To address the above problems, the paper proposes a hyperspectral image classification method based on Narrowing Semantic Gap Convolutional Neural Network (NSGCNN). We developed the feature fusion sub-network, which consists of Feature Semantic Enhancement Module (FSEM), Spatial Detail Supplement Module (SDCM), and Cross-layer Feature Fusion Module (CFFM). FSEM extracts rich global semantic information based on multi-scale dilated convolution. Reverse attention mechanism is introduced in the SDCM to complement deep-layer features with fine spatial features like edges and textures. The fusing of hierarchical features is then achieved using CFFM. Additionally, we can regulate the model’s parameter count by regulating the number of output feature maps. Experimental results on three benchmark hyperspectral datasets demonstrate that NSGCNN outperforms other state-of-the-art methods in terms of classification performance and significantly outperforms them in terms of model complexity.

Aspect-Level sentiment analysis based on fusion graph double convolutional neural networks

ABSTRACT Aspect-based sentiment analysis (ABSA) aims to analyze the emotional color contained in sentences or documents in more detail by classifying and evaluating different aspects and emotions in the text. However, the current research methods cannot effectively analyze the relationship between aspect words and context and extract grammatical information about sentences. Additionally, the extracted syntactic information is insufficient, and the combination of syntactic and semantic information is inefficient, leaving the model unable to correctly determine aspects' emotional orientations. This paper proposes an aspect-level sentiment analysis based on Fusion Graph Double Convolutional Neural Networks (FGD-GCN) to address these issues. Firstly, FGD-GCN proposes a multi-feature extraction module. Using BERT and bidirectional long-short-term memory models, this module extracts the hidden context between words. In addition, the positional attention module is used to capture important features in sentences, reducing noise and bias. Then, a semantic enhancement module is proposed, which fuses attention-focused information and feature information extracted from graphs to emphasize aspect words and context, and uses CNN model to classify on feature vectors. According to experiment results on three benchmark datasets, the model outperforms previous GCN methods for context-based aspect-level sentiment analysis.

Cross‐modal knowledge learning with scene text for fine‐grained image classification

AbstractScene text in natural images carries additional semantic information to aid in image classification. Existing methods lack full consideration of the deep understanding of the text and the visual text relationship, which results in the difficult to judge the semantic accuracy and the relevance of the visual text. This paper proposes image classification based on Cross modal Knowledge Learning of Scene Text (CKLST) method. CKLST consists of three stages: cross‐modal scene text recognition, text semantic enhancement, and visual‐text feature alignment. In the first stage, multi‐attention is used to extract features layer by layer, and a self‐mask‐based iterative correction strategy is utilized to improve the scene text recognition accuracy. In the second stage, knowledge features are extracted using external knowledge and are fused with text features to enhance text semantic information. In the third stage, CKLST realizes visual‐text feature alignment across attention mechanisms with a similarity matrix, thus the correlation between images and text can be captured to improve the accuracy of the image classification tasks. On Con‐Text dataset, Crowd Activity dataset, Drink Bottle dataset, and Synth Text dataset, CKLST can perform significantly better than other baselines on fine‐grained image classification, with improvements of 3.54%, 5.37%, 3.28%, and 2.81% over the best baseline in mAP, respectively.

Classification of Typical Static Objects in Road Scenes Based on LO-Net

Mobile LiDAR technology is a powerful tool that accurately captures spatial information about typical static objects in road scenes. However, the precise extraction and classification of these objects pose persistent technical challenges. In this paper, we employ a deep learning approach to tackle the point cloud classification problem. Despite the popularity of the PointNet++ network for direct point cloud processing, it encounters issues related to insufficient feature learning and low accuracy. To address these limitations, we introduce a novel layer-wise optimization network, LO-Net. Initially, LO-Net utilizes the set abstraction module from PointNet++ to extract initial local features. It further enhances these features through the edge convolution capabilities of GraphConv and optimizes them using the “Unite_module” for semantic enhancement. Finally, it employs a point cloud spatial pyramid joint pooling module, developed by the authors, for the multiscale pooling of final low-level local features. Combining three layers of local features, LO-Net sends them to the fully connected layer for accurate point cloud classification. Considering real-world scenarios, road scene data often consist of incomplete point cloud data due to factors such as occlusion. In contrast, models in public datasets are typically more complete but may not accurately reflect real-world conditions. To bridge this gap, we transformed road point cloud data collected by mobile LiDAR into a dataset suitable for network training. This dataset encompasses nine common road scene features; hence, we named it the Road9 dataset and conducted classification research based on this dataset. The experimental analysis demonstrates that the proposed algorithm model yielded favorable results on the public datasets ModelNet40, ModelNet10, and the Sydney Urban Objects Dataset, achieving accuracies of 91.2%, 94.2%, and 79.5%, respectively. On the custom road scene dataset, Road9, the algorithm model proposed in this paper demonstrated outstanding classification performance, achieving a classification accuracy of 98.5%.

Adaptive Semantic-Enhanced Transformer for Image Captioning.

In the research on image captioning, rich semantic information is very important for generating critical caption words as guiding information. However, semantic information from offline object detectors involves many semantic objects that do not appear in the caption, thereby bringing noise into the decoding process. To produce more accurate semantic guiding information and further optimize the decoding process, we propose an end-to-end adaptive semantic-enhanced transformer (AS-Transformer) model for image captioning. For semantic enhancement information extraction, we propose a constrained weaklysupervised learning (CWSL) module, which reconstructs the semantic object's probability distribution detected by the multiple instances learning (MIL) through a joint loss function. These strengthened semantic objects from the reconstructed probability distribution can better depict the semantic meaning of images. Also, for semantic enhancement decoding, we propose an adaptive gated mechanism (AGM) module to adjust the attention between visual and semantic information adaptively for the more accurate generation of caption words. Through the joint control of the CWSL module and AGM module, our proposed model constructs a complete adaptive enhancement mechanism from encoding to decoding and obtains visual context that is more suitable for captions. Experiments on the public Microsoft Common Objects in COntext (MSCOCO) and Flickr30K datasets illustrate that our proposed AS-Transformer can adaptively obtain effective semantic information and adjust the attention weights between semantic and visual information automatically, which achieves more accurate captions compared with semantic enhancement methods and outperforms state-of-the-art methods.