Semantic Structural Information Research Articles

RPEPL: Tibetan Sentiment Analysis Based on Relative Position Encoding and Prompt Learning

Sentiment analysis is a critical task for natural language processing. Much research has been done for high-resource languages such as English and Chinese. However, Tibetan is an extremely low-resource language with less reference information. According to the practical demands, this paper proposes a Tibetan sentiment analysis method based on relative position encoding and prompt learning, the method is abbreviated as RPEPL. First, Word information is introduced to syllable sequences by converting the directed acyclic lattice into a squashed structure. Second, a relative position encoding is used to encode the position information of syllables and words. Then the association relations and semantic information of tokens are identified by leveraging the multi-attention. Finally, the sentiment category of Tibetan sentence is obtained through a prompt learning framework. Experimental results demonstrate that RPEPL significantly outperforms the baseline methods on the TUSA dataset and TNEC dataset. Additionally, Traditional recurrent neural networks cannot perform large-scale parallel computation and convolutional neural networks have difficulty modeling long-distance dependencies in Tibetan text both of which are resolved using RPEPL. Furthermore, the use of a multi-attention not only enriches the association relations between syllables and words, but also enhances the understanding of sentence semantic and syntactic structure information, and improves the performance of Tibetan sentiment analysis.

Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detection

Code Language Models (codeLMs) and Graph Neural Networks (GNNs) are widely used in code vulnerability detection. However, a critical yet often overlooked issue is that GNNs primarily rely on aggregating information from adjacent nodes, limiting structural information transfer to single-layer updates. In code graphs, nodes and relationships typically require cross-layer information propagation to fully capture complex program logic and potential vulnerability patterns. Furthermore, while some studies utilize codeLMs to supplement GNNs with code semantic information, existing integration methods have not fully explored the potential of their collaborative effects.To address these challenges, we introduce Vul-LMGNNs that integrates pre-trained CodeLMs with GNNs, leveraging knowledge distillation to facilitate cross-layer propagation of both code semantic knowledge and structural information. Specifically, Vul-LMGNNs utilizes Code Property Graphs (CPGs) to incorporate code syntax, control flow, and data dependencies, while employing gated GNNs to extract structural information in the CPG. To achieve cross-layer information transmission, we implement an online knowledge distillation (KD) program that enables a single student GNN to acquire structural information extracted from a simultaneously trained counterpart through an alternating training procedure. Additionally, we leverage pre-trained CodeLMs to extract semantic features from code sequences. Finally, we propose an ”implicit-explicit” joint training framework to better leverage the strengths of both CodeLMs and GNNs. In the implicit phase, we utilize CodeLMs to initialize the node embeddings of each student GNN. Through online knowledge distillation, we facilitate the propagation of both code semantics and structural information across layers. In the explicit phase, we perform linear interpolation between the CodeLM and the distilled GNN to learn a late fusion model. The proposed method, evaluated across four real-world vulnerability datasets, demonstrated superior performance compared to 17 state-of-the-art approaches. Our source code can be accessed via GitHub: https://github.com/Vul-LMGNN/vul-LMGGNN.

Structural link prediction model with multi-view text semantic feature extraction

The exponential expansion of information has made text feature extraction based on simple semantic information insufficient for the multidimensional recognition of textual data. In this study, we construct a text semantic structure graph based on various perspectives and introduce weight coefficients and node clustering coefficients of co-occurrence granularity to enhance the link prediction model, in order to comprehensively capture the structural information of the text. Firstly, we jointly build the semantic structure graph based on three proposed perspectives (i.e., scene semantics, text weight, and graph structure), and propose a candidate keyword set in conjunction with an information probability retrieval model. Subsequently, we propose weight coefficients of co-occurrence granularity and node clustering coefficients to improve the link prediction model based on the semantic structure graph, enabling a more comprehensive acquisition of textual structural information. Experimental results demonstrate that our research method can reveal potential correlations and obtain more complete semantic structure information, while the WPAA evaluation index validates the effectiveness of our model.

MMHCA: Multi-feature representations based on multi-scale hierarchical contextual aggregation for UAV-view geo-localization

In global navigation satellite system denial environment, cross-view geo-localization based on image retrieval presents an exceedingly critical visual localization solution for Unmanned Aerial Vehicle (UAV) systems. The essence of cross-view geo-localization resides in matching images containing the same geographical targets from disparate platforms, such as UAV-view and satellite-view images. However, images of the same geographical targets may suffer from occlusions and geometric distortions due to variations in the capturing platform, view, and timing. The existing methods predominantly extract features by segmenting feature maps, which overlook the holistic semantic distribution and structural information of objects, resulting in loss of image information. To address these challenges, dilated neighborhood attention Transformer is employed as the feature extraction backbone, and Multi-feature representations based on Multi-scale Hierarchical Contextual Aggregation (MMHCA) is proposed. In the proposed MMHCA method, the multi-scale hierarchical contextual aggregation method is utilized to extract contextual information from local to global across various granularity levels, establishing feature associations of contextual information with global and local information in the image. Subsequently, the multi-feature representations method is utilized to obtain rich discriminative feature information, bolstering the robustness of model in scenarios characterized by positional shifts, varying distances, and scale ambiguities. Comprehensive experiments conducted on the extensively utilized University-1652 and SUES-200 benchmarks indicate that the MMHCA method surpasses the existing techniques, showing outstanding results in UAV localization and navigation.

URINet: Unsupervised point cloud rotation invariant representation learning via semantic and structural reasoning

In recent years, many rotation-invariant networks have been proposed to alleviate the interference caused by point cloud arbitrary rotations. These networks have demonstrated powerful representation learning capabilities. However, most of those methods rely on costly manually annotated supervision for model training. Moreover, they fail to reason the structural relations and lose global information. To address these issues, we present an unsupervised method for achieving comprehensive rotation invariant representations without human annotation. Specifically, we propose a novel encoder–decoder architecture named URINet, which learns a point cloud representation by combining local semantic and global structural information, and then reconstructs the input without rotation perturbation. In detail, the encoder is a two-branch network where the graph convolution based structural branch models the relationships among local regions to learn global structural knowledge and the semantic branch learns rotation invariant local semantic features. The two branches derive complementary information and explore the point clouds comprehensively. Furthermore, to avoid the self-reconstruction ambiguity brought by uncertain poses, a bidirectional alignment is proposed to measure the quality of reconstruction results without orientation knowledge. Extensive experiments on downstream tasks show that the proposed method significantly surpasses existing state-of-the-art methods on both synthetic and real-world datasets.

Incomplete multi-view partial multi-label classification via deep semantic structure preservation

Recent advances in multi-view multi-label learning are often hampered by the prevalent challenges of incomplete views and missing labels, common in real-world data due to uncertainties in data collection and manual annotation. These challenges restrict the capacity of the model to fully utilize the diverse semantic information of each sample, posing significant barriers to effective learning. Despite substantial scholarly efforts, many existing methods inadequately capture the depth of semantic information, focusing primarily on shallow feature extractions that fail to maintain semantic consistency. To address these shortcomings, we propose a novel Deep semantic structure-preserving (SSP) model that effectively tackles both incomplete views and missing labels. SSP innovatively incorporates a graph constraint learning (GCL) scheme to ensure the preservation of semantic structure throughout the feature extraction process across different views. Additionally, the SSP integrates a pseudo-labeling self-paced learning (PSL) strategy to address the often-overlooked issue of missing labels, enhancing the classification accuracy while preserving the distribution structure of data. The SSP model creates a unified framework that synergistically employs GCL and PSL to maintain the integrity of semantic structural information during both feature extraction and classification phases. Extensive evaluations across five real datasets demonstrate that the SSP method outperforms existing approaches, including lrMMC, MVL-IV, MvEL, iMSF, iMvWL, NAIML, and DD-IMvMLC-net. It effectively mitigates the impacts of data incompleteness and enhances semantic representation fidelity.

An attention mechanism and residual network based knowledge graph-enhanced recommender system

Recommender systems enhanced by a knowledge graph (KG) have attained widespread popularity and attention in recent years. However, traditional KG-based recommender systems encounter the challenge of gradient explosion as the network depth increases. Additionally, the abundance of unreliable paths in a KG has a detrimental impact on feature representation learning. In this article, we propose a KG-enhanced recommender system based on residual network and attention mechanism, which can capture high-order connectivity and long-range dependencies of the KG. Specifically, a resource allocation approach is employed to calculate the resource amount, which is subsequently utilized to evaluate the path reliability of the KG. After completing path extraction, we employ an attention mechanism to capture semantic correlations and structural information. To leverage the KG for enhancing recommender systems, we design a deep residual network with shortcut connections, effectively amalgamating advanced and abstract features using deep neural networks. The introduction of shortcut connections not only facilitates the fitting of residual mappings but also mitigates potential issues such as gradient explosion and convergence difficulties due to excessive network depth. Extensive experiments conducted on three standard datasets over baseline methods have demonstrated the superiority of our proposed recommender system.

Hierarchical Semantic-Guided Contextual Structure-Aware Network for Spectral Satellite Image Dehazing

Haze or cloud always shrouds satellite images, obscuring valuable geographic information for military surveillance, natural calamity surveillance and mineral resource exploration. Satellite image dehazing (SID) provides the possibility for better applications of satellite images. Most of the existing dehazing methods are tailored for natural images and are not very effective for satellite images with non-homogeneous haze since the semantic structure information and inconsistent attenuation are not fully considered. To tackle this problem, this study proposes a hierarchical semantic-guided contextual structure-aware network (SCSNet) for spectral satellite image dehazing. Specifically, a hybrid CNN–Transformer architecture integrated with a hierarchical semantic guidance (HSG) module is presented to learn semantic structure information by synergetically complementing local representation from non-local features. Furthermore, a cross-layer fusion (CLF) module is specially designed to replace the traditional skip connection during the feature decoding stage so as to reinforce the attention to the spatial regions and feature channels with more serious attenuation. The results on the SateHaze1k, RS-Haze, and RSID datasets demonstrated that the proposed SCSNet can achieve effective dehazing and outperforms existing state-of-the-art methods.

A Multi-View Temporal Knowledge Graph Reasoning Framework with Interpretable Logic Rules and Feature Fusion

A temporal knowledge graph represents temporal information between entities in a multi-relational graph. Its reasoning aims to infer and predict potential links among entities. Predicting time-aware entities is a challenging task due to significant differences in entity appearances over time, such as different timestamps and frequencies. Current embedding-based similarity-matching methods have been introduced for predicting temporal facts. However, they lack deterministic logical explainability and cannot model the dynamic evolution of entities over time. To address these challenges, we propose a novel framework for temporal knowledge graph reasoning based on multi-view feature fusion (MVFF). First, MVFF extracts logical rules and uses the Gumbel-Softmax trick to sample high-quality rules. Second, it integrates logical rules, temporal quadruples, and factual triples to capture semantic features, temporal information, and structural information to solve link prediction tasks. Through experiments on four benchmark datasets, we show that MVFF outperforms state-of-the-art methods, providing not only better performance but also interpretable results.

Supervised Hierarchical Online Hashing for Cross-modal Retrieval

Online cross-modal hashing has gained attention for its adaptability in processing streaming data. However, existing methods only define the hard similarity between data using labels. This results in poor retrieval performance, as they fail to exploit the semantic structure information of labels and miss the high-quality hash codes guided by the hierarchical relevance between labels. In addition, they ignore the bit-flipping problem, which leads to sub-optimal cross-modal retrieval performance. To address these issues, we propose Supervised Hierarchical Online Hashing (SHOH) for cross-modal retrieval. Our approach acquires hierarchical similarity via cross-layer affiliation of labels and explores its application to online hashing. We design a hierarchical similarity learning method in the online learning framework, which includes virtual center learning and hierarchical similarity embedding. Labels with soft similarity bridge the label hierarchy and cross-modal hash embedding. Furthermore, we propose a Weighted Retrieval Strategy (WRS) to mitigate the impact caused by bit-flipping errors. Extensive experiments and verification on hierarchical and non-hierarchical datasets demonstrate that SHOH preserves accurate inter-class distances and achieves performance improvements compared to state-of-the-art methods. The source code is available at https://github.com/HUST-IDSM-AI/SHOH .

Mutual Dual-Task Generator With Adaptive Attention Fusion for Image Inpainting

Image segmentation can reveal the semantic structure information in an image, which is helpful guidance information for image inpainting. Notably, it can help mitigate the artifacts on the boundaries of different semantic regions during the inpainting process. Existing semantic guidance-based image inpainting provides one-way guidance from the semantic segmentation task to the image inpainting task. There is no feedback from the inpainting results to adjust the guidance process, which causes inferior performance. To tackle this issue, this work proposes mutual dual-task generators to establish the interaction between image segmentation and image inpainting tasks. Thus, semantic segmentation guides image inpainting and also receives feedback from image inpainting. These two processes interact with each other and progressively improve the inpainting quality. The mutual dual-task generator consists of a shared encoder and mutual decoders with the bidirectional Cross-domain Feature DeNormalization (CFDN) module inside, which hierarchically models the Segmentation-guided image Texture (ST) generation and Texture-guided semantic Segmentation (TS) generation. At the end of mutual decoders, an Adaptive Attention Fusion (AAF) module is proposed to augment the texture and semantic class affinity between pixels, further refining the inpainted results. Experimental results demonstrate that the proposed mutual dual-task generator pipeline achieves superior inpainting performances over the state of the arts on three public datasets.

A Malware Detection Framework Based on Semantic Information of Behavioral Features

As the amount of malware has grown rapidly in recent years, it has become the most dominant attack method in network security. Learning execution behavior, especially Application Programming Interface (API) call sequences, has been shown to be effective for malware detection. However, it is troublesome in practice to adequate mining of API call features. Among the current research methods, most of them only analyze single features or inadequately analyze the features, ignoring the analysis of structural and semantic features, which results in information loss and thus affects the accuracy. In order to deal with the problems mentioned above, we propose a novel method of malware detection based on semantic information of behavioral features. First, we preprocess the sequence of API function calls to reduce redundant information. Then, we obtain a vectorized representation of the API call sequence by word embedding model, and encode the API call name by analyzing it to characterize the API name’s semantic structure information and statistical information. Finally, a malware detector consisting of CNN and bidirectional GRU, which can better understand the local and global features between API calls, is used for detection. We evaluate the proposed model in a publicly available dataset provided by a third party. The experimental results show that the proposed method outperforms the baseline method. With this combined neural network architecture, our proposed model attains detection accuracy of 0.9828 and an F1-Score of 0.9827.

LAST: LAtent Space-constrained Transformers for Automatic Surgical Phase Recognition and Tool Presence Detection.

When developing context-aware systems, automatic surgical phase recognition and tool presence detection are two essential tasks. There exist previous attempts to develop methods for both tasks but majority of the existing methods utilize a frame-level loss function (e.g., cross-entropy) which does not fully leverage the underlying semantic structure of a surgery, leading to sub-optimal results. In this paper, we propose multi-task learning-based, LAtent Space-constrained Transformers, referred as LAST, for automatic surgical phase recognition and tool presence detection. Our design features a two-branch transformer architecture with a novel and generic way to leverage video-level semantic information during network training. This is done by learning a non-linear compact presentation of the underlying semantic structure information of surgical videos through a transformer variational autoencoder (VAE) and by encouraging models to follow the learned statistical distributions. In other words, LAST is of structure-aware and favors predictions that lie on the extracted low dimensional data manifold. Validated on two public datasets of the cholecystectomy surgery, i.e., the Cholec80 dataset and the M2cai16 dataset, our method achieves better results than other state-of-the-art methods. Specifically, on the Cholec80 dataset, our method achieves an average accuracy of 93.12±4.71%, an average precision of 89.25±5.49%, an average recall of 90.10±5.45% and an average Jaccard of 81.11 ±7.62% for phase recognition, and an average mAP of 95.15±3.87% for tool presence detection. Similar superior performance is also observed when LAST is applied to the M2cai16 dataset.

IRC-CLVul: Cross-Programming-Language Vulnerability Detection with Intermediate Representations and Combined Features

The most severe problem in cross-programming languages is feature extraction due to different tokens in different programming languages. To solve this problem, we propose a cross-programming-language vulnerability detection method in this paper, IRC-CLVul, based on intermediate representation and combined features. Specifically, we first converted programs in different programming languages into a unified LLVM intermediate representation (LLVM-IR) to provide a classification basis for different programming languages. Afterwards, we extracted the code sequences and control flow graphs of the samples, used the semantic model to extract the program semantic information and graph structure information, and concatenated them into semantic vectors. Finally, we used Random Forest to learn the concatenated semantic vectors and obtained the classification results. We conducted experiments on 85,811 samples from the Juliet test suite in C, C++, and Java. The results show that our method improved the accuracy by 7% compared with the two baseline algorithms, and the F1 score showed a 12% increase.

SIRN: An iterative reasoning network for transmission lines based on scene prior knowledge

High-accuracy positioning and identification of transmission line components is the premise for their status detection and fault diagnosis. However, due to the limitations of imbalance object scale and distribution in aerial images, the problem of detecting dense-tiny objects still needs to be solved. Considering the prior knowledge of fixed connection scenes, we proposed a novel method named scene iterative reasoning network (SIRN) based on self-adaptive clustering and local scene knowledge. It consists of the coarse-grained detector (CGD), scene self-adaptive clustering (SSAC) subnetwork, and scene structure iterative reasoning (SSIR) subnetwork. The CGD was proposed to obtain global coarse detection results. Then, the SSAC subnetwork utilized unsupervised self-adaptive clustering to obtain accurate dense regions based on the CGD results. Finally, the SSIR subnetwork was designed to extract and fuse the scene semantic structure information of local regions for realizing accurate dense-tiny object detection. In summary, the SIRN model utilizes an iterative inference strategy to detect dense regions accurately. By leveraging scene structure information, the model effectively transforms the challenge of detecting dense objects into an advantage, resulting in precise detection. The SIRN model is compatible with single-stage and two-stage object detection models, achieving a notable mean average precision (mAP) improvement ranging from 4.4% to 12.2% compared to the baseline models. Compared to other state-of-the-art object detection models, SIRN demonstrates significant advantages in accuracy and error rate metrics. In addition, qualitative and quantitative experiments show that the SIRN model considerably enhanced the detection of dense-tiny objects. The code is available at https://github.com/CharmingWang/SIRN.

Continual Pre-Training of Language Models for Concept Prerequisite Learning with Graph Neural Networks

Prerequisite chains are crucial to acquiring new knowledge efficiently. Many studies have been devoted to automatically identifying the prerequisite relationships between concepts from educational data. Though effective to some extent, these methods have neglected two key factors: most works have failed to utilize domain-related knowledge to enhance pre-trained language models, thus making the textual representation of concepts less effective; they also ignore the fusion of semantic information and structural information formed by existing prerequisites. We propose a two-stage concept prerequisite learning model (TCPL), to integrate the above factors. In the first stage, we designed two continual pre-training tasks for domain-adaptive and task-specific enhancement, to obtain better textual representation. In the second stage, to leverage the complementary effects of the semantic and structural information, we optimized the encoder of the resource–concept graph and the pre-trained language model simultaneously, with hinge loss as an auxiliary training objective. Extensive experiments conducted on three public datasets demonstrated the effectiveness of the proposed approach. Our proposed model improved by 7.9%, 6.7%, 5.6%, and 8.4% on ACC, F1, AP, and AUC on average, compared to the state-of-the-art methods.

Smart contract vulnerability detection based on a semantic code structure and a self-designed neural network

Smart contracts are riddled with vulnerabilities due to flaws in programming languages and the inexperience of developers, causing damage. Nonetheless, the current research on smart contract vulnerability detection is insufficient. In this study, we propose a novel approach, namely, Blass, based on a semantic code structure and a self-designed neural network. Blass constructs program slices with complete semantic structure information (CPSs) and uses an abstract syntax tree and a depth-first traversal algorithm to convert CPSs into code chains during the process of CPS vectorization, which increases its ability to express vulnerability features. Blass also uses a self-designed neural network, Bi-LSTM-Att, as the classification model, which introduces an attention mechanism to capture the key features of vulnerabilities and effectively achieve improved smart contract vulnerability detection performance. The CPSs and the Bi-LSTM-Att can improve the vulnerability detection effectiveness of Blass, and Blass can be applied to malicious contract detection with satisfactory precision, recall, and F1 values.

Face Image Completion Method Based on Parsing Features Maps

In this paper, we focus on the facial completion task based on the parsing feature map. In recent years, methods based on deep learning have achieved remarkable results in face image completion. However, since many methods do not consider the semantic structure information of face images, these methods may lead to unreasonable or discontinuous cases in the return results. To solve this problem, we propose a face image completion network based on face parsing features. The network includes two sub-network: face parsing map prediction network and face image completion network. The face parsing map prediction network predicts the parsing features of the missing regions, and the face image completion network uses these features to guide the face image completion network to fill the missing areas. On the open dataset, CelebA-HQ and Flickr Faces High Quality(FFHQ), the validity of this method is verified by qualitative and quantitative analysis.

An edge guided coarse-to-fine generative network for image outpainting

Deep-learning based generative models have achieved outstanding performance in various image processing tasks. This paper introduces a method to address the problem of extrapolating or outpainting visual context. When the input size is a small proportion of the output size, a limited amount of information is present to regenerate a semantically coherent image. This task is challenging because the missing region of the original image may include crucial semantic and spatial structural information, which is difficult to predict from the input. We propose a three-stage edge-guided coarse-to-fine generative network model, consisting of a contextual inference network, structural edge map generator and edge enhanced network, to synthesise semantically consistent output from small picture inputs. Our model adopts a gradual growth inference strategy in the contextual inference network so that the generated image can present a more coherent structure, and this result can support the structural edge map generator to generate a reasonable edge map in a large missing area. Combining the contextual inference network and structural edge map generator outputs enables the edge enhanced network to generate more convincing images. We evaluate our model using four public datasets: CelebA, Places2, Oxford Flower102and CUB200. Our experimental results demonstrate that the proposed image outpainting network can successfully regenerate high-quality images with a large missing region even when some structural features are lost in the input images.

HEAT: Incorporating hierarchical enhanced attention transformation into urban road detection

AbstractRoad detection plays a vital role in automated driving and advanced driver assistance systems. In recent years, mainstream frameworks have suffered from the restrictions of a receptive field and the limitation of modelling long‐range relations. Previous methods fail to segment precise road boundaries when urban roads with similar surface textures are presented. Moreover, road regions are perceived as non‐road areas due to the shadow effect, which affects the completeness of the road in the traffic environment. To this end, for urban road detection, a hierarchical enhanced attention transformation (HEAT) architecture, which holds both fine details (road edges) and global contextual information (road structure) is proposed. The symmetrical data‐fusion residual network fuses visual semantic and spatial structure information. The attention consolidation units model global contextual information at different layers to realize the feature enhancement from coarse to fine. In addition, corresponding local and global features are fused hierarchically in the progressive up‐sampling modules. Comprehensive empirical studies are conducted to compare other mainstream methods in the KITTI and the Cityscapes dataset. HEAT shows highly competitive performance in that confusable areas are correctly distinguished in the presence of obstacles, shadows, and similar road textures. HEAT outperforms state‐of‐the‐art methods in urban road detection.