Extract Semantic Information Research Articles

Parsing of Research Documents into XML Using Formal Grammars

Automatic information extraction of content and style format in paged documents is challenging. It requires the conversion of the original document into a granular level of details for which every document section and content is identifiable. This functionality or tool does not exist for any academic research document yet. In this paper, we present an automated process of parsing research paper documents into XML files using a formal method approach of context-free grammars (CFGs) and regular expressions (REGEXs) definable of a standard template. We created a tool for the algorithms to parse these documents into tree-like structures organized as XML files named research_XML (RX) parser. The RX tool performed the extraction of syntactic structure and semantic information of the document’s contents into XML files. These XML output files are lightweight, analyzable, query-able, and web interoperable. The RX tool has a success rate of 91% when evaluated on fifty varying research documents of 160 average pages and 8,004 total pages. The tool and test data are accessible on GitHub repo. The novelty of our process is specific to applying formal techniques for information extraction in structured multipaged documents and academic research documents thus advancing the research in automatic information extraction.

Multi-level and cross-domain search engine based on graph convolutional neural network

Abstract With the advent of the information age, there is a growing demand for multi-level cross-domain information retrieval by users. However, current technological solutions have not yet overcome the dilemma of automatic adjustment of search scope in cross-domain retrieval. This article proposes a multi-level cross-domain retrieval engine based on graph convolutional neural networks (GCNs). GCN can effectively capture the complex semantic structures in documents, excavate the potential dynamic context of documents by learning the relationships between nodes in the graph structure, adapt to cross-domain retrieval, extract semantic information from different levels and automatically adjust the search scope. To verify the effectiveness of the method, experiments were conducted on the NQ320k dataset and compared with SEAL, BM25, Transformer and Bert models. The results show that the cross-domain retrieval engine based on GCN has significant advantages over other methods, with Recall@100 reaching 97.44%, MRR@100 reaching 76.12% and R-Precision reaching 78.84%. In the future, the performance and applicability of the retrieval engine can be further improved by optimizing the model structure and expanding the training dataset.

Thangka Image—Text Matching Based on Adaptive Pooling Layer and Improved Transformer

Image–text matching is a research hotspot in the multimodal task of integrating image and text processing. In order to solve the difficult problem of associating image and text data in the multimodal knowledge graph of Thangka, we propose an image and text matching method based on the Visual Semantic Embedding (VSE) model. The method introduces an adaptive pooling layer to improve the feature extraction capability of semantic associations between Thangka images and texts. We also improved the traditional Transformer architecture by combining bidirectional residual concatenation and mask attention mechanisms to improve the stability of the matching process and the ability to extract semantic information. In addition, we designed a multi-granularity tag alignment module that maps global and local features of images and text into a common coding space, leveraging inter- and intra-modal semantic associations to improve image and text accuracy. Comparative experiments on the Thangka dataset show that our method achieves significant improvements compared to the VSE baseline method. Specifically, our method improves the recall by 9.4% and 10.5% for image-matching text and text-matching images, respectively. Furthermore, without any large-scale corpus pre-training, our method outperforms all models without pre-training and outperforms two out of four pre-trained models on the Flickr30k public dataset. Also, the execution efficiency of our model is an order of magnitude higher than that of the pre-trained models, which highlights the superior performance and efficiency of our model in the image–text matching task.

Semantic information extraction and search of mineral exploration data using text mining and deep learning methods

Large-scale mineral resource reports offer a wealth of information for geological knowledge mining and mineral explorers’ knowledge discovery. The geological conditions in which mineral deposits develop may be learned a great deal from these mineral exploration reports. Mineral exploration data may be queried and aggregated to effectively mitigate future exploration risks and reduce costs. However, due to the reports being presented in unstructured textual format, it becomes challenging to extract valuable geological data without manually scanning through a vast number of reports. This laborious process poses difficulties for geologists. To address this issue, this paper proposes a system that extracts a set of geologically relevant keywords/keyphrases of each chapter from each mineral exploration reposts using latent Dirichlet allocation (LDA), develops a topic graph, recognizes the geological entity and related relations for constructing a knowledge graph, and uses visualization of those graphs (e.g., topic graphs and knowledge graphs) to explore the contents of the report. The text mining and machine learning technique described here serves as the foundation for future research into incorporating semantic analysis into geological information extraction. The findings of this study show how automated text analysis may help with the quick processing of huge quantities of reports in order to identify target mineral systems and their related geological location and rock mineral composition. The suggested approaches can quickly and reliably convert mineral exploration data (e.g., text, figure, and table) into a structured form, which is a hitherto untouched field in geological knowledge mining.

Wave-Like Class Activation Map With Representation Fusion for Weakly-Supervised Semantic Segmentation

The Class Activation Map (CAM) is widely used to generate pseudo-labels for Weakly Supervised Semantic Segmentation (WSSS), while it does not adequately consider the modeling of foreground-independent information, resulting in prone to false positive pixels. In this paper, we propose a Wave-like Class Activation Map (WaveCAM) from the perspective of representation fusion and dynamic aggregation representation to alleviate the above problem. Specifically, our WaveCAM includes the foreground-aware representation modeling that enhances perception of foreground information, and the foreground-independent representation modeling that enhances perception of foreground-independent information, and a representation-adaptive fusion module that fuses the two representations. Both representations are expressed as wave functions with amplitude and phase to dynamically aggregate representations and extract semantic information after initialization, and they are fused through the adaptive fusion module to obtain an output containing rich semantic information. Extensive experiments on PASCAL VOC 2012 dataset and MS COCO 2014 dataset validate that our WaveCAM can easily embed multi-stage WSSS and end-to-end WSSS, achieving the state-of-the-art performance. The release code is available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Rongtao-Xu/RepresentationLearning/tree/main/WaveCAM-TMM2023</uri> .

Semantic Information Extraction and Multi-Agent Communication Optimization Based on Generative Pre-Trained Transformer

Semantic Information Extraction and Multi-Agent Communication Optimization Based on Generative Pre-Trained Transformer

Adversarial Information Bottleneck.

The information bottleneck (IB) principle has been adopted to explain deep learning in terms of information compression and prediction, which are balanced by a tradeoff hyperparameter. How to optimize the IB principle for better robustness and figure out the effects of compression through the tradeoff hyperparameter are two challenging problems. Previous methods attempted to optimize the IB principle by introducing random noise into learning the representation and achieved the state-of-the-art performance in the nuisance information compression and semantic information extraction. However, their performance on resisting adversarial perturbations is far less impressive. To this end, we propose an adversarial IB (AIB) method without any explicit assumptions about the underlying distribution of the representations, which can be optimized effectively by solving a min-max optimization problem. Numerical experiments on synthetic and real-world datasets demonstrate its effectiveness on learning more invariant representations and mitigating adversarial perturbations compared to several competing IB methods. In addition, we analyze the adversarial robustness of diverse IB methods contrasting with their IB curves and reveal that IB models with the hyperparameter β corresponding to the knee point in the IB curve achieve the best tradeoff between compression and prediction and has the best robustness against various attacks.

Remote sensing image semantic segmentation network based on multi-scale feature enhancement fusion

Semantic segmentation is a crucial method for recognizing and classifying objects in high-resolution remote sensing images (HRRSIs). However, due to the problems of varying target scale and difficulty in determining the edges of small-scale targets in remote sensing images, traditional semantic segmentation models perform poorly. To address this issue, we propose a multi-scale feature enhancement network (MFENet) to improve the segmentation accuracy of small-scale objects in HRRSIs. MFENet considers the differences between objects of different scales and selects more suitable receptive fields to enhance the extraction of multi-scale semantic features. We propose a composite atrous multi-scale feature fusion (CAMFF) module to enhance the extraction of spatial detail and semantic information of features at different scales. In addition, we propose an improved composite atrous spatial pyramid pooling (C-ASPP) module to enhance the network feature extraction capability across multiple scales. We also propose a network structure that combines the C-ASPP module with the efficient channel attention (ECA) module in parallel, which performs better to extract contextual information. Our experimental evaluations on the Potsdam and Vaihingen datasets demonstrate the effectiveness of our Network, It F1 score reaching 93.33% and 94.66% respectively.

Multi-Stage Network with Geometric Semantic Attention for Two-View Correspondence Learning.

The removal of outliers is crucial for establishing correspondence between two images. However, when the proportion of outliers reaches nearly 90%, the task becomes highly challenging. Existing methods face limitations in effectively utilizing geometric transformation consistency (GTC) information and incorporating geometric semantic neighboring information. To address these challenges, we propose a Multi-Stage Geometric Semantic Attention (MSGSA) network. The MSGSA network consists of three key modules: the multi-branch (MB) module, the GTC module, and the geometric semantic attention (GSA) module. The MB module, structured with a multi-branch design, facilitates diverse and robust spatial transformations. The GTC module captures transformation consistency information from the preceding stage. The GSA module categorizes input based on the prior stage's output, enabling efficient extraction of geometric semantic information through a graph-based representation and inter-category information interaction using Transformer. Extensive experiments on the YFCC100M and SUN3D datasets demonstrate that MSGSA outperforms current state-of-the-art methods in outlier removal and camera pose estimation, particularly in scenarios with a high prevalence of outliers. Source code is available at https://github.com/shuyuanlin.

Merge-and-Split Graph Convolutional Network for Skeleton-Based Interaction Recognition.

We introduce an innovative approach to address a significant challenge in interaction recognition, specifically the capture of correlation features between different interaction body parts. These features are often overlooked by traditional graph convolution networks commonly used in interaction recognition tasks. Our solution, the Merge-and-Split Graph Convolutional Network, takes a unique perspective, treating interaction recognition as a global problem. It leverages a Merge-and-Split Graph structure to effectively capture dependencies between interaction body parts. To extract the essential interaction features, we introduce the Merge-and-Split Graph Convolution module, which seamlessly combines the Merge-and-Split Graph with Graph Convolutional Networks. This fusion enables the extraction of rich semantic information between adjacent joint points. In addition, we introduce a Short-term Dependence module designed to extract joint and motion characteristics specific to each type of interaction. Furthermore, to extract correlation features between different hierarchical sets, we present the Hierarchical Guided Attention Module. This module plays a crucial role in highlighting the relevant hierarchical sets that contain essential interaction information. The effectiveness of our proposed model is demonstrated by achieving state-of-the-art performance on 2 widely recognized datasets, namely, the NTU60 and NTU120 interaction datasets. Our model's efficacy is rigorously validated through extensive experiments, and we have made the code available for the research community at https://github.com/wanghq05/MS-GCN/.

MVMM: Multiview Multimodal 3-D Object Detection for Autonomous Driving

Object detection in 3D space is a fundamental technology in the autonomous driving system. Among the published 3D object detection methods, the single-modal methods based on point clouds have been widely studied. One problem exposed by these methods is that point clouds lack color and texture features. The limitation in conveying semantic information often leads to failures in detection. In contrast, the multi-modal methods based on the image and point clouds fusion may solve this problem, but relevant research is not sufficient. In this work, a single-stage multi-view multi-modal 3D object detector (MVMM) is proposed, which can naturally and efficiently extract semantic and geometric information from the image and point clouds. Specifically, the data-level fusion approach of point clouds coloring is used for combining information from the camera and LIDAR. Next, an encoder-decoder backbone is devised to extract features from colored points in the range view. Then, colored points are concatenated with the range view features, voxelized, and fed into the point view bridge for down-sampling. Finally, the down-sampled feature map is used by the bird's eye view backbone and the detection head for generating 3D results based on predefined anchors. According to extensive experiments on the KITTI dataset, MVMM achieves competitive performance while runs at 27 FPS on the 1080 Ti GPU. Particularly, MVMM performs extremely well in difficult scenes (e.g., heavy occlusion and truncation) due to the understanding of fused information.

HMFVC: A Human-Machine Friendly Video Compression Scheme

Recent studies on Video Coding for Machine (VCM) have achieved remarkable results. However, in practical visual-analytic applications, except for transmitting the visual feature for machine analysis, video textures are also mandatory for human monitoring and decision-making. To this end, this paper proposes a human-machine friendly video compression scheme (HMFVC) which can satisfy both human viewing and machine analysis well. First, we propose a learned semantic representation (LSR) method to extract semantic information between temporal neighboring frames. LSR could be utilized in signal reconstruction for human viewing and visual analysis for machine understanding. Second, given the proposed LSR, we design an end-to-end optimized video compression framework to jointly optimize the visual quality for human perception, analysis accuracy for machines, and compression efficiency as well. Finally, an HMFVC codec is developed, which can achieve higher action recognition accuracy and better reconstruction quality than the traditional codecs and learned video compression approaches. Specifically, HMFVC saves 77% bitrate to achieve the same analysis performance with the original videos compared to x265. To our knowledge, HMFVC is the first end-to-end optimized video compression scheme to serve both humans and machines. It is a promising framework for human-machine friendly video compression approaches.

The Strategy of Discriminating False Comments on the Internet by Fusing Probabilistic Topic and Word Vector Models

With the acceleration of the social process of "Internet+", e-commerce has entered an era of rapid development. In response to the subjective judgments in current false information detection methods and the problem of low classification accuracy caused by the failure to extract semantic information hidden in text, this study proposes Latent Dirichlet Allocation -Word vector-Edge Convolutional Neural Network (LW-ECNN) Model. First, the probabilistic topic model is organically combined with the word vector model, and then the edge path structure is proposed to optimize the Convolutional Neural Network (CNN) discriminative classification model. Finally, the data-side optimization module is combined with the edge CNN classifier to form a web-based false comment discrimination model based on the probabilistic topic and word vector models. The results show that the probabilistic topic and word vector-based online false comment discrimination model has good results in detecting false comments, and can provide data reference for false comment detection by false detection-related departments, which is of great practical significance to assist the network environment cleaning work.

Deep learning based retinal vessel segmentation and hypertensive retinopathy quantification using heterogeneous features cross-attention neural network.

Retinal vessels play a pivotal role as biomarkers in the detection of retinal diseases, including hypertensive retinopathy. The manual identification of these retinal vessels is both resource-intensive and time-consuming. The fidelity of vessel segmentation in automated methods directly depends on the fundus images' quality. In instances of sub-optimal image quality, applying deep learning-based methodologies emerges as a more effective approach for precise segmentation. We propose a heterogeneous neural network combining the benefit of local semantic information extraction of convolutional neural network and long-range spatial features mining of transformer network structures. Such cross-attention network structure boosts the model's ability to tackle vessel structures in the retinal images. Experiments on four publicly available datasets demonstrate our model's superior performance on vessel segmentation and the big potential of hypertensive retinopathy quantification.

SDSNet: Building Extraction in High-Resolution Remote Sensing Images Using a Deep Convolutional Network with Cross-Layer Feature Information Interaction Filtering

Building extraction refers to the automatic identification and separation of buildings from the background in remote sensing images. It plays a significant role in urban planning, land management, and disaster monitoring. Deep-learning methods have shown advantages in building extraction, but they still face challenges such as variations in building types, object occlusions, and complex backgrounds. To address these issues, SDSNet, a deep convolutional network that incorporates global multi-scale feature extraction and cross-level feature fusion, is proposed. SDSNet consists of three modules: semantic information extraction (SIE), multi-level merge (MLM), and semantic information fusion (SIF). The SIE module extracts contextual information and improves recognition of multi-scale buildings. The MLM module filters irrelevant details guided by high-level semantic information, aiding in the restoration of edge details for buildings. The SIF module combines filtered detail information with extracted semantic information for refined building extraction. A series of experiments conducted on two distinct public datasets for building extraction consistently demonstrate that SDSNet outperforms the state-of-the-art deep-learning models for building extraction tasks. On the WHU building dataset, the overall accuracy (OA) and intersection over union (IoU) achieved impressive scores of 98.86% and 90.17%, respectively. Meanwhile, on the Massachusetts dataset, SDSNet achieved OA and IoU scores of 94.05% and 71.6%, respectively. SDSNet exhibits a unique advantage in recovering fine details along building edges, enabling automated and intelligent building extraction. This capability effectively supports urban planning, resource management, and disaster monitoring.

EDF-YOLOv5: An Improved Algorithm for Power Transmission Line Defect Detection Based on YOLOv5

Detecting defects in power transmission lines through unmanned aerial inspection images is crucial for evaluating the operational status of outdoor transmission equipment. This paper presents a defect recognition method called EDF-YOLOv5, which is based on the YOLOv5s, to enhance detection accuracy. Firstly, the EN-SPPFCSPC module is designed to improve the algorithm’s ability to extract information, thereby enhancing the detection performance for small target defects. Secondly, the algorithm incorporates a high-level semantic feature information extraction network, DCNv3C3, which improves its ability to generalize to defects of different shapes. Lastly, a new bounding box loss function, Focal-CIoU, is introduced to enhance the contribution of high-quality samples during training. The experimental results demonstrate that the enhanced algorithm achieves a 2.3% increase in mean average precision (mAP@.5) for power transmission line defect detection, a 0.9% improvement in F1-score, and operates at a detection speed of 117 frames per second. These findings highlight the superior performance of EDF-YOLOv5 in detecting power transmission line defects.

SeMalBERT: Semantic-based malware detection with bidirectional encoder representations from transformers

Machine learning models are widely used for identifying malicious software. However, existing models suffer from issues such as imprecise polysemous representations and a lack of contextual semantic representations, leading to the failure to recognize certain types of malicious software. In this paper, we propose a semantic-based intelligent malware detection model called SeMalBERT for identifying malicious software in Windows systems. Specifically, the model utilizes the API function sequences of malicious software as the learned features. Firstly, BERT is applied to accomplish word representation tasks and extract semantic information from the sequences. Secondly, a hybrid discriminator based on Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) is used to explore the chaining relationships between functions. Lastly, an external attention mechanism is added after the LSTM to enable the model to better focus on key information within the text. Experimental results demonstrate that SeMalBERT outperforms existing malware detection techniques in terms of accuracy, F1 score, and loss function value on a general dataset.

DOE: a dynamic object elimination scheme based on geometric and semantic constraints

In this paper, we propose a dynamic object elimination algorithm that combines semantic and geometric constraints to address the problem of visual SLAM being easily affected by dynamic feature points in dynamic environments. This issue leads to the degradation of localisation accuracy and robustness. Firstly, we employ a lightweight YOLO-Tiny network to enhance both detection accuracy and system speed. Secondly, we integrate the YOLO-Tiny network into the ORB-SLAM3 system to extract semantic information from the images and initiate the elimination of dynamic feature points. Subsequently, we augment this approach by incorporating geometric constraints between neighbouring frames to further eliminate dynamic feature points. Then, the former is supplemented by combining the geometric constraints between neighbouring frames to further eliminate dynamic feature points. Experiments on the TUM dataset demonstrate that the algorithm in this paper can improve the Relative Pose Error (RPE) by up to 95.12% and the Absolute Trajectory Error (ATE) by up to 99.01% in high dynamic sequences compared to ORB-SLAM3. The effectiveness of dynamic feature point elimination is evident, leading to significantly improved localisation accuracy.

Quantized Graph Neural Networks for Image Classification

Researchers have resorted to model quantization to compress and accelerate graph neural networks (GNNs). Nevertheless, several challenges remain: (1) quantization functions overlook outliers in the distribution, leading to increased quantization errors; (2) the reliance on full-precision teacher models results in higher computational and memory overhead. To address these issues, this study introduces a novel framework called quantized graph neural networks for image classification (QGNN-IC), which incorporates a novel quantization function, Pauta quantization (PQ), and two innovative self-distillation methods, attention quantization distillation (AQD) and stochastic quantization distillation (SQD). Specifically, PQ utilizes the statistical characteristics of distribution to effectively eliminate outliers, thereby promoting fine-grained quantization and reducing quantization errors. AQD enhances the semantic information extraction capability by learning from beneficial channels via attention. SQD enhances the quantization robustness through stochastic quantization. AQD and SQD significantly improve the performance of the quantized model with minimal overhead. Extensive experiments show that QGNN-IC not only surpasses existing state-of-the-art quantization methods but also demonstrates robust generalizability.

A time series anomaly detection method based on series-parallel transformers with spatial and temporal association discrepancies

Multivariate time series anomaly detection methods can discover malfunctions in a complex system by detecting anomalies in the monitoring data. Multivariate time series have complex temporal and spatial correlations. Existing methods that explicitly model both correlations only extract semantic information sequentially in a single temporal-spatial or spatial-temporal order. However, different orders have a significant influence on the method's performance. In addition, a static graph structure trained on normal data has difficulties capturing the change of correlations between features when anomalies occur, limiting the method's performance. This paper proposes the Spatial-Temporal Anomaly Transformer (STAT), which extracts semantic information comprehensively by combining both spatial-temporal and temporal-spatial orders. STAT uses the Temporal Anomaly Transformer and the Spatial Anomaly Transformer (SAT) to explicitly compute temporal and spatial association discrepancies to conduct anomaly detection. Since the topological structure is usually unknown, SAT adopts a trainable hypersphere to learn the prior-association between features on normal data and computes the distance between the attention weights and the center of the hypersphere as the spatial association discrepancy. AUC and two recently proposed evaluation metrics are used to compare STAT with various typical methods on public data sets, proving that our method can achieve state-of-the-art performance.