Semantic Enhancement Research Articles

HHSE: heterogeneous graph neural network via higher-order semantic enhancement

HHSE: heterogeneous graph neural network via higher-order semantic enhancement

A Semantic Enhancement Framework for Multimodal Sarcasm Detection

Sarcasm represents a language form where a discrepancy lies between the literal meanings and implied intention. Sarcasm detection is challenging with unimodal text without clearly understanding the context, based on which multimodal information is introduced to benefit detection. However, current approaches only focus on modeling text–image incongruity at the token level and use the incongruity as the key to detection, ignoring the significance of the overall multimodal features and textual semantics during processing. Moreover, semantic information from other samples with a similar manner of expression also facilitates sarcasm detection. In this work, a semantic enhancement framework is proposed to address image–text congruity by modeling textual and visual information at the multi-scale and multi-span token level. The efficacy of textual semantics in multimodal sarcasm detection is pronounced. Aiming to bridge the cross-modal semantic gap, semantic enhancement is performed by using a multiple contrastive learning strategy. Experiments were conducted on a benchmark dataset. Our model outperforms the latest baseline by 1.87% in terms of the F1-score and 1% in terms of accuracy.

Semantic enhancement and multi-level alignment network for cross-modal retrieval

Semantic enhancement and multi-level alignment network for cross-modal retrieval

Improving entity linking by combining semantic entity embeddings and cross-attention encoder

Entity linking is an important task for information retrieval and knowledge graph construction. Most existing methods use a bi-encoder structure to encode mentions and entities in the same space, and learn contextual features for entity linking. However, this type of system still faces some problems: (1) the entity embedding part of the model only learns from the local context of the target entity, which is too unique for entity linking model to learn the context commonality of information; (2) the entity disambiguation part only uses similarity calculation once to determine the target entity, resulting in insufficient interaction between the mentions and candidate entities, and ineffective recall of real entities. We propose a new entity linking model based on graph neural network. Different from other bi-encoder retrieval systems, this paper introduces a fine-grained semantic enhancement information into the entity embedding part of the bi-encoder to reduce the specificity of the model. Then, the cross-attention encoder is used to re-rank the target mention and each candidate entity after the entity retrieval model. Experimental results show that although the model is not optimal in inference speed, it outperforms all baseline methods on the AIDA-CoNLL dataset, and has good generalization effects on four datasets in different fields such as MSNBC and ACE2004.

USER: Unified Semantic Enhancement With Momentum Contrast for Image-Text Retrieval.

As a fundamental and challenging task in bridging language and vision domains, Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality, and its key challenge is to measure the semantic similarity across different modalities. Although significant progress has been achieved, existing approaches typically suffer from two major limitations: (1) It hurts the accuracy of the representation by directly exploiting the bottom-up attention based region-level features where each region is equally treated. (2) It limits the scale of negative sample pairs by employing the mini-batch based end-to-end training mechanism. To address these limitations, we propose a Unified Semantic Enhancement Momentum Contrastive Learning (USER) method for ITR. Specifically, we delicately design two simple but effective Global representation based Semantic Enhancement (GSE) modules. One learns the global representation via the self-attention algorithm, noted as Self-Guided Enhancement (SGE) module. The other module benefits from the pre-trained CLIP module, which provides a novel scheme to exploit and transfer the knowledge from an off-the-shelf model, noted as CLIP-Guided Enhancement (CGE) module. Moreover, we incorporate the training mechanism of MoCo into ITR, in which two dynamic queues are employed to enrich and enlarge the scale of negative sample pairs. Meanwhile, a Unified Training Objective (UTO) is developed to learn from mini-batch based and dynamic queue based samples. Extensive experiments on the benchmark MSCOCO and Flickr30K datasets demonstrate the superiority of both retrieval accuracy and inference efficiency. For instance, compared with the existing best method NAAF, the metric R@1 of our USER on the MSCOCO 5K Testing set is improved by 5% and 2.4% on caption retrieval and image retrieval without any external knowledge or pre-trained model while enjoying over 60 times faster inference speed. Our source code will be released at https://github.com/zhangy0822/USER.

Enhancing Relational Triple Extraction in Specific Domains: Semantic Enhancement and Synergy of Large Language Models and Small Pre-Trained Language Models

Enhancing Relational Triple Extraction in Specific Domains: Semantic Enhancement and Synergy of Large Language Models and Small Pre-Trained Language Models

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

Iterative Edge Enhancing Framework for Building Change Detection

The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Building Change Detection</i> (BCD) task serves urban planning through monitoring the landuse. However, due to the complexity of remote sensing images and high foreground-background similarity, it leads to inaccurate detection of building edge regions. Existing methods deal with this problem by fusing features of different layers. But the fusing operation cannot separate details information from overall information of buildings, resulting in inaccurate detection of building edge area. To address the above challenges, we propose an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Iterative Edge Enhancing Framework</i> (IEEF). The IEEF alleviates the building edge detection difficulty by densely implementing Detail Semantic Enhancement Module in decoding part. This module takes differential features between adjacent scales to explicitly represent the building edge information. Simultaneously, to deal with the classes imbalance problem, a Density-Guided Sampling method dedicated for change detection is proposed to increase the proportion of positive samples during training. Our proposed method achieves state-of-the-art performance on the LEVIR-CD dataset and the WHU dataset, and obtains accurate changed building edges.

Weakening the Dominant Role of Text: CMOSI Dataset and Multimodal Semantic Enhancement Network.

Multimodal sentiment analysis (MSA) is important for quickly and accurately understanding people's attitudes and opinions about an event. However, existing sentiment analysis methods suffer from the dominant contribution of text modality in the dataset; this is called text dominance. In this context, we emphasize that weakening the dominant role of text modality is important for MSA tasks. To solve the above two problems, from the perspective of datasets, we first propose the Chinese multimodal opinion-level sentiment intensity (CMOSI) dataset. Three different versions of the dataset were constructed: manually proofreading subtitles, generating subtitles using machine speech transcription, and generating subtitles using human cross-language translation. The latter two versions radically weaken the dominant role of the textual model. We randomly collected 144 real videos from the Bilibili video site and manually edited 2557 clips containing emotions from them. From the perspective of network modeling, we propose a multimodal semantic enhancement network (MSEN) based on a multiheaded attention mechanism by taking advantage of the multiple versions of the CMOSI dataset. Experiments with our proposed CMOSI show that the network performs best with the text-unweakened version of the dataset. The loss of performance is minimal on both versions of the text-weakened dataset, indicating that our network can fully exploit the latent semantics in nontext patterns. In addition, we conducted model generalization experiments with MSEN on MOSI, MOSEI, and CH-SIMS datasets, and the results show that our approach is also very competitive and has good cross-language robustness.

On Exploring Shape and Semantic Enhancements for RGB-X Semantic Segmentation

On Exploring Shape and Semantic Enhancements for RGB-X Semantic Segmentation

Web Service Recommendation via Combining Topic-aware Heterogeneous Graph Representation and Interactive Semantic Enhancement

Web Service Recommendation via Combining Topic-aware Heterogeneous Graph Representation and Interactive Semantic Enhancement

Negative-Sensitive Framework With Semantic Enhancement for Composed Image Retrieval

Negative-Sensitive Framework With Semantic Enhancement for Composed Image Retrieval

Semantic enhancement based adaptive geometric encoding network for low overlap point cloud registration

The presence of partial or low overlaps in real point cloud pairs poses significant challenges to obtain robust registration. There is an absence of a unified framework that localizes reliable overlapping regions and correspondences. This work proposes an adaptive point cloud geometric encoding network based on semantic enhancement that generates overlap information with geometric-and-semantic consistency in various scenarios. This enables the network to further ensure distinctive geometric representation when the low overlap point cloud pairs have scarce differences in the geometric structures. In addition, our study includes experiments conducted on a self-created semantic-assisted dataset designed to facilitate in-depth research on point cloud registration. The experimental results clearly demonstrate the efficacy and overall improvements of the registration performance, particularly highlighting the potential of our approach in addressing the challenges associated with low overlap scenarios that have repetitive geometry patterns.

An Exploration of the Connotation Characteristics of Data Civics and Its Effects in the Digital Era

Abstract In this paper, the variational self-encoder is dissected, and the depth-embedded variational self-clustering model based on it is constructed, and the loss function and optimization function are designed. By enhancing the information of textual semantic representation, the semantic enhancement module is constructed based on the deep variational inference algorithm, the network structure of the semantic enhancement module is designed, and the SEVAE model of textual clustering based on the semantic embedding of DataSmart is constructed by combining textual clustering layer and dataSmart connotation. On this basis, the comparative experiments of the SEVAE model are constructed to analyze the convergence and clustering performance of the constructed model, explore the connotative features of Data Sensemaking, and analyze the effect of Data Sensemaking through empirical evidence. The results show that the clustering assignment process converges at the 20th iteration round, which significantly improves the feature representation, divides the 8483 texts into 3 groups, and the Data Thinking has the characteristics of Data Thinking, Data Information, and Data Quantification, with a positive, positive effect of P&lt;0.1 for all the control indicators. Data Thinking has the power to promote patriotic sentiment, patriotism, and social cognition.

Identifying the cargo types of road freight with semi-supervised trajectory semantic enhancement

Identifying road freight cargo types is crucial for regional economic interaction and transportation optimization. Existing methods primarily rely on manual labeling and the rule, neither of which can achieve automated semantic enhancement of large-scale road freight trajectories. Consequently, this study proposes a semi-supervised trajectory semantic enhancement method for identifying cargo types based on trajectory feature extraction and point-of-interest (POI) association. The raw trajectories are segmented and enriched with the closest POIs. The sample labeling method with POI semantic enhancement is then proposed using company registration information. Finally, the spatiotemporal and sequential features of labeled freight trips are extracted to build a self-training semi-supervised model for identifying the cargo type of road freight. Experimental studies on real trajectory data demonstrate superior accuracy and robustness compared to existing methods, with accuracy and F1 values reaching 81.4 and 0.77%, respectively. The proposed sample labeling method improves representativeness and universality, increasing accuracy by 7.8–14.4% and F1 value by 8.5–34.5% compared to the rule-based method. The semi-supervised model improves accuracy by 8.9% and F1 value by 29.1% compared to the supervised model when only 10.0% of samples were labeled. This method enables automatic and full-sample cargo type identification in real-world large-scale transportation systems.

Low-Resolution Steel Surface Defects Classification Network Based on Autocorrelation Semantic Enhancement

Aiming at the problems of low-resolution steel surface defects imaging, such as defect type confusion, feature blurring, and low classification accuracy, this paper proposes an autocorrelation semantic enhancement network (ASENet) for the classification of steel surface defects. It mainly consists of a backbone network and an autocorrelation semantic enhancement module (ASE), in which the autocorrelation semantic enhancement module consists of three main learnable modules: the CS attention module, the autocorrelation computation module, and the contextual feature awareness module. Specifically, we first use the backbone network to extract the basic features of the image and then use the designed CS attention module to enhance the basic features. In addition, to capture different aspects of semantic objects, we use the autocorrelation module to compute the correlation between neighborhoods and contextualize the basic and augmented features to enhance the recognizability of the features. Experimental results show that our method produces significant results, and the classification accuracy reaches 96.24% on the NEU-CLS-64 dataset. Compared with ViT-B/16, Swin_t, ResNet50, Mobilenet_v3_small, Densenet121, Efficientnet_b2, and baseline, the accuracy is 9.43%, 5.15%, 4.87%, 3.34%, 3.28%, 3.01%, and 2.72% higher, respectively.

A Named Entity Recognition Approach for Electronic Medical Records Using BERT Semantic Enhancement and BiLSTM

Aiming at the problems of missing local context features, single word vector representation, and low entity recognition accuracy, a method for e-medical recording with named entity recognition, which is based on BERT and model fusion, is proposed. First, with the model of BERT for pre-training, the preceding and following contextual information is fused for the enhancement of word semantic representation and alleviation of the problem of polysemy; second, the network of bi-directional long-short term memory is for obtaining the sequence feature matrix, generation of optimal sequence in global sense achieved through the conditional random field model; finally, data enhancement is used to alleviate the class imbalance and improve the model ability in generalization. Results of the experiments find model proposal measured by F1 on CCKS21 data set reaches 0.8548, which is 0.51% and 0.08% higher than models with ID-CNNs-CRF and multi-task RNN. This demonstrates the excellent performance of the method proposed in this paper in improving named entity recognition.

FSSDD: Few‐shot steel defect detection based on multi‐scale semantic enhancement representation and mask category information mapping

AbstractSteel defect detection is important for industry production as it is tied to the product quality and production efficiency. However, previous steel defect detection methods based on deep convolutional neural networks heavily rely on large‐scale data for training and tend to have poor generalization ability for a novel defect category. In this paper, a novel few‐shot steel defect detection model based on multi‐scale semantic enhancement representation and mask category information mapping is introduced, where only a few annotated samples are acquired for the novel defect category. More concretely, three main components are built: an information‐guidance enhanced multi‐head detector is proposed to improve the representation of information in meta‐feature maps, a mask category representation module is designed to enhance the category feature representation of the mask region in the support set, and a novel multi‐scale category edge loss function is designed to assist the generation of category reweighting vector. Extensive experiments on the North‐east University few‐shot steel defect data set demonstrate that the proposed method significantly outperforms the state‐of‐the‐art methods and verify its effectiveness through ablation studies.

Multi-view semantic enhancement model for few-shot knowledge graph completion

In recent years, few-shot knowledge graph completion (FKGC) has gained popularity as a solution to the long-tail distribution problem of real-world knowledge graphs (KGs). The previous knowledge graph completion (KGC) models obtain triple representation merely relying on the structure view, ignoring the valuable semantic knowledge. In this paper, we propose a multi-view framework for few-shot relation learning to address the issue. Specifically, based the structural information of the graph obtained using the structure view, we add the text view and the commonsense view. The text view employs text descriptions of entities and relations to obtain a richer semantic representation. The commonsense view performs high-quality negative sampling based on complex relations. Moreover, commonsense semantic constraints are invoked to suppress the overfitting caused by the complexity of the relation matrix. Extensive experiments show that our model outperforms state-of-the-art FKGC methods on the frequently-used benchmark datasets FB15k237-One and NELL-One.

TTH-Net: Two-Stage Transformer–CNN Hybrid Network for Leaf Vein Segmentation

Leaf vein segmentation is crucial in species classification and smart agriculture. The existing methods combine manual features and machine learning techniques to segment coarse leaf veins. However, the extraction of the intricate patterns is time consuming. To address the issues, we propose a coarse-to-fine two-stage hybrid network termed TTH-Net, which combines a transformer and CNN to accurately extract veins. Specifically, the proposed TTH-Net consists of two stages and a cross-stage semantic enhancement module. The first stage utilizes the Vision Transformer (base version) to extract globally high-level feature representations. Based on these features, the second stage identifies fine-grained vein features via CNN. To enhance the interaction between the two stages, a cross-stage semantic enhancement module is designed to integrate the strengths of the transformer and CNN, which also improves the segmentation accuracy of the decoder. Extensive experiments on the public dataset LVN are conducted, and the results prove that TTH-Net has significant advantages over other methods in leaf vein segmentation.