Semantic Feature Fusion Research Articles

TDAD: Self-supervised industrial anomaly detection with a two-stage diffusion model

Visual anomaly detection has emerged as a highly applicable solution in practical industrial manufacturing, owing to its notable effectiveness and efficiency. However, it also presents several challenges and uncertainties. To address the complexity of anomaly types and the high cost associated with data annotation, this paper introduces a self-supervised learning framework called TDAD, based on a two-stage diffusion model. TDAD consists of three key components: anomaly synthesis, image reconstruction, and defect segmentation. It is trained end-to-end, with the goal of improving pixel-level segmentation accuracy of anomalies and reducing false detection rates. By synthesizing anomalies from normal samples, designing a diffusion model-based reconstruction network, and incorporating a multiscale semantic feature fusion module for defect segmentation, TDAD achieves state-of-the-art performance in image-level detection and anomaly localization on challenging and widely used datasets such as MVTec and VisA benchmarks.

Dynamically Enhanced lane detection with multi-scale semantic feature fusion

Lane detection plays a crucial role in autonomous driving technology, and there have been many recent advancements in 3D lane detection methods. One common approach is simplifying the problem by transforming images into Bird's Eye View (BEV) space. However, existing methods still have limitations, particularly in accurately identifying lanes in many complex autonomous driving scenarios, such as slopes and extreme weather conditions. Therefore, this paper proposes a dynamically updated lane detection method called DynamicallyLane. This method utilizes ConvNeXt V2-N deformations as the backbone for feature extraction, employs dynamic learnable encoding and deformable attention mechanisms, achieves precise conversion from the frontal view to BEV through a novel representation of 3D reference points, and utilizes the Enhanced BEV Features module for feature fusion, obtaining richer and more semantically informative feature representations to facilitate the model learning process better. Experimental results demonstrate that DynamicallyLane achieves an F-score of 56.2 % on the OpenLane dataset and performs excellently on the Apollo 3D Synthetic dataset. Our code is available at https://github.com/Tafble/lane_detection.

A multi-scale semantic feature fusion method for remote sensing crop classification

The efficient and accurate acquisition of crop spatial planting structure information is of great significance to ensure national food security. Current crop classification studies have still not achieved optimal results in accurate crop classification identification due to the complex heterogeneous details of clear images and insufficient analysis and utilization of information. This study takes the Yuncheng-Linfen Basin in Shanxi, China as the study area and uses the ReliefF-RFE feature selection algorithm to achieve data dimensionality reduction. Contextual semantic feature aggregation, spatial channel attention mechanism, and hierarchical refinement strategy are used to fuse the multi-scale feature information in the extracted remote sensing images from the top down and bottom up. This study shows that 1) the ReliefF-RFE algorithm can effectively reduce the dimensionality of 224-dimensional multisource features and generate a 31-dimensional optimal feature space subset with strong interpretation to improve the computational efficiency of the model; 2) the multi-scale feature fusion model constructed in this study is better than UNet, ResNet, DeepLabv3+, HRNet and Swin Transformer in classification recognition of winter wheat and corn, with an overall accuracy of 95.76% and 93.44%; 3) A multiple feature fusion strategy designed through ablation experiments was evaluated to evaluate the reasonableness of each module and validate the irreplaceability and superiority of the model for improving crop accuracy in complex and heterogeneous terrain; 4) the multi-scale feature fusion model converts deep abstract features into local detail information by cross-scale contextual aggregation of deep and shallow multi-scale features, it obtains good extraction effect on fragmented small plots, and the reduction of local fine details of features is significantly better than other models. The multi-scale feature fusion classification model constructed in this study fully considers the interpretability of the input features and fuses the semantic features at different scales to eliminate the “semantic gap”, which shows that the model has strong stability and robustness and provides a reference for fast and accurate mapping of Sentinel-2 images of crops in large areas.

CoF-DResNet: Cancer Metastasis Recognition Network based on Dynamic Coordinated Metabolic Attention and Structural Attention.

Cancer metastasis usually means that cancer cells spread to other tissues or organs, and the condition worsens. Identifying whether cancer has metastasized can help doctors infer the progression of a patient's condition and is an essential prerequisite for devising treatment plans. Fluorine 18 fluorodeoxyglucose positron emission tomography/computed tomography ( 18F -FDG PET/CT) is an advanced cancer diagnostic imaging technique that provides both metabolic and structural information. In cancer metastasis recognition tasks, effectively integrating metabolic and structural information stands as a key technology to enhance feature representation and recognition performance. This paper proposes a cancer metastasis identification network based on dynamic coordinated metabolic attention and structural attention to address these challenges. Specifically, metabolic and structural features are extracted by incorporating a dynamic coordinated attention module (DCAM) into two branches of ResNet networks, thereby amalgamating high metabolic spatial information from PET images with texture structure information from CT images, and dynamically adjusting this process through iterations. Next, to improve the efficacy of feature expression, a multi-receptive field feature fusion module (MRFM) is included in order to execute multi-receptive field fusion of semantic features. To validate the effectiveness of our proposed model, experiments were conducted on both a private lung lymph nodes dataset and a public soft tissue sarcomas dataset. The accuracy of our method reached 76.0% and 75.1% for the two datasets, respectively, demonstrating an improvement of 6.8% and 5.6% compared to ResNet, thus affirming the efficacy of our method.

EPSSNet

Fast and accurate segmentation is important for robot judgement, e.g. robot detection, segmentation, and control. Most researchers have focused on deploying lightweight semantic segmentation models into robot services. The problem is that the critical interaction between semantic segmentation and boundaries is ignored. In this chapter, the authors propose a lightweight parallel execution model (EPSSNet) based on semantic flow branch (SFB), edge flow branch (EFB) and self-adapting weighting fusion (SAWF) for mobile robot service projects. The semantic flow branching module is used to obtain accurate object shape features. The boundary constraint module uses multiple convolution and upsampling to distinguish boundary features from semantic features. In order to adaptively fuse boundary features with semantic segmentation features, the SAWF is proposed. It adaptively fuses semantic and boundary features by learning boundary and semantic feature fusion weights. Detailed experimental results on Cityscapes, Pascal VOC 2012 and ADE20k datasets demonstrate the superior performance of our approach.

An Improved SSD Model for Small Size Work-pieces Recognition in Automatic Production Line

<p>Aiming at the problems of slow recognition speed and low recognition accuracy of arbitrarily placed workpiece by machine vision in traditional automated production lines, a workpiece recognition algorithm based on improved SSD is proposed. Firstly, the improved DarkNet53 is used to replace the backbone network in the original SSD network framework, and the network enhancement is used in the backbone network to solve the defect of small target missed detection. Then, channel attention module and deep semantic feature fusion module are added, in order to improve the recognition ability and detection accuracy of the small target features. Lastly, the loss function was optimized, and the problem caused by sample imbalance was solved by changing the weight distribution of positive and negative samples. In the experiment, image datasets of typical bolts, nuts, and connecting plates were constructed for the network training, the experimental results showed that, the recognition accuracy and speed have been optimized and meet the requirements of automatic work-piece detection in actual production, compared with traditional YOLOv4 and the original SSD algorithm in the work-piece recognition task.</p> <p> </p>

A single-stage anchor-free rotating target visual detection algorithm suitable for flexible body vibration displacement measurement

The phenomenon of angular inclination of flexible structures during vibration poses a significant challenge to the applicability of visual vibration measurement methods because the target locked in the captured image will produce unknown geometric deformations such as scale, displacement, and angle in the time domain space, and the horizontal rectangular frame used for matching during target detection will also increase the false detection rate of the target due to the introduction of more background information. Such subtle geometric deformations and false detections can lead to severe fit errors in the displacement curves regressed by the visual vibration measurement algorithm. To effectively improve the accuracy and robustness of vibration image target recognition, this article takes the flexible body captured by a high-speed camera as the target of vibration displacement measurement. It introduces the rotating target detection method based on deep learning into the field of visual vibration measurement, which verifies the feasibility of the deep learning method in flexible body vibration measurement, and based on the deep convolutional neural network framework, a high-precision displacement measurement algorithm based on single-stage anchor-free rotating target detection is proposed. The algorithm in this article first uses the CSPDarknet backbone network to extract multi-scale features of flexible structural image sequences. It then uses PANet to fuse the top-down and bottom-up bidirectional feature maps of the four bridge target feature maps obtained through the backbone network. The shallow and deep information is used for semantic feature fusion and combined with the Coordinate Attention mechanism to achieve target finding and fine positioning on the feature map. Finally, we use the coordinates of the bounding box obtained from the test to regress the position offset of the object’s center point. To verify the accuracy of the algorithm in this article, we conducted experimental validation on the cable-stayed bridge model and the actual bridge and compared the performance with the traditional template matching algorithm, differential optical flow method, and various deep learning algorithms with different localization principles, as well as the displacement signals collected and processed by accelerometers. The experimental results of time-frequency characteristics analysis show that the vibration displacement trajectories regressed by the algorithm in this paper have the best overlap with the displacement measurements collected by the accelerometer, which verifies that the algorithm in this article has good application potential and implementation space in the field of condition monitoring of flexible structural bodies.

A deep learning-based global and segmentation-based semantic feature fusion approach for indoor scene classification

This work proposes a novel approach that uses a semantic segmentation mask to obtain a 2D spatial layout of the segmentation-categories across the scene, designated by segmentation-based semantic features (SSFs). These features represent, per segmentation-category, the pixel count, as well as the 2D average position and respective standard deviation values. Moreover, a two-branch network, GS2F2App, that exploits CNN-based global features extracted from RGB images and the segmentation-based features extracted from the proposed SSFs, is also proposed. GS2F2App was evaluated in two indoor scene benchmark datasets: the SUN RGB-D and the NYU Depth V2, achieving state-of-the-art results on both datasets.

RETRACTION: Digital Media Art Creation Based on Virtual Reality and Semantic Feature Fusion

RETRACTION: Digital Media Art Creation Based on Virtual Reality and Semantic Feature Fusion

Surface defect segmentation of magnetic tiles based on cross self-attention module

The detection of magnetic tile quality is an essential link before the assembly of permanent magnet motor. In order to meet the high standard of magnetic tile surface defect detection and realize the rapid and automatic segmentation of magnetic tile defects, a magnetic tile surface defect segmentation algorithm based on cross self-attention model (CSAM) is proposed. It adopts high-low level semantic feature fusion method to build the dependency relationship between the deep and shallow features. Multiple auxiliary loss functions are used to constrain the network and reduce the noise in the deep features. In addition, an image enhancement method is also designed to solve the problem of insufficient annotated data. The experimental results show that the network can achieve 79.6% mIoU and 98.5% PA, which can meet the high standard requirements of magnetic tile manufacturing.

An Iterative Closest Point Method for Lidar Odometry with Fused Semantic Features

Lidar sensors play a pivotal role in a multitude of remote sensing domains, finding extensive applications in various sectors, including robotics, unmanned aerial vehicles (UAVs), autonomous driving, and 3D reconstruction, among others. Their significance and versatility in these areas have made them indispensable tools for a wide range of applications. The accuracy of Lidar odometry (LO), which serves as the front end of SLAM, is crucial. In this paper, we propose a novel iterative closest point (ICP) technique that combines semantic features to improve LO precision. First, the semantic segmentation neural network is used to extract the semantic features from each frame of the point cloud. Then, the obtained semantic features assist in extracting the local geometric features of the point cloud. Also, the residual blocks of the ICP algorithm’s least squares are combined with semantic confidence functions to better predict the exact pose. Compared to LOAM, there is an average improvement of 4 cm in accuracy per 100 m. Experimental results illustrate the superiority of the proposed method and indicate that the fusion of semantic features can robustly improve the precision of LO.

MTTSNet: Military time-sensitive targets stealth network via real-time mask generation

The automatic stealth task of military time-sensitive targets plays a crucial role in maintaining national military security and mastering battlefield dynamics in military applications. We propose a novel Military Time-sensitive Targets Stealth Network via Real-time Mask Generation (MTTSNet). According to our knowledge, this is the first technology to automatically remove military targets in real-time from videos. The critical steps of MTTSNet are as follows: First, we designed a real-time mask generation network based on the encoder-decoder framework, combined with the domain expansion structure, to effectively extract mask images. Specifically, the ASPP structure in the encoder could achieve advanced semantic feature fusion. The decoder stacked high-dimensional information with low-dimensional information to obtain an effective mask layer. Subsequently, the domain expansion module guided the adaptive expansion of mask images. Second, a context adversarial generation network based on gated convolution was constructed to achieve background restoration of mask positions in the original image. In addition, our method worked in an end-to-end manner. A particular semantic segmentation dataset for military time-sensitive targets has been constructed, called the Military Time-sensitive Target Masking Dataset (MTMD). The MTMD dataset experiment successfully demonstrated that this method could create a mask that completely occludes the target and that the target could be hidden in real time using this mask. We demonstrated the concealment performance of our proposed method by comparing it to a number of well-known and highly optimized baselines.

CenterPoint-SE: A Single-Stage Anchor-Free 3-D Object Detection Algorithm With Spatial Awareness Enhancement

Real-time and accurate 3-D object detection is one of the foundational technologies for environmental perception in autonomous vehicles. However, the existing second-stage anchor-based 3-D object detection algorithms have high accuracy, but they are challenging in terms of computation complexity and latency. Due to poor perception of spatial features, the accuracy of the existing single-stage anchor-free detection algorithms with low latency are difficult to be implemented into autonomous vehicles. Therefore, we focus on enhancing the spatial perception ability of the anchor-free detection network based on CenterPoints. In this paper, we propose a single-stage anchor-free 3-D object detector CenterPoint-Space-Enhancement (CenterPoint-SE) algorithm and construct an efficient 3-D backbone network to extract fine-grained spatial geometric features by introducing a spatial attention mechanism and residual structure. At the same time, a powerful spatial semantic feature fusion module, the enhancement of feature fusion (EF-Fusion), is designed. In addition, we add a lightweight IoU prediction branch to improve the algorithm’s perception of various object sizes. Finally, we add a foreground point segmentation auxiliary training branch to enable the 3-D backbone to obtain object boundary features. We use the ONCE dataset to train and validate the proposed model, and the results showed that the proposed CenterPoint-SE achieves 70.33 mAP and an inference speed of 17.15 FPS, outperforming other methods.

SGNet: a fast and accurate semantic segmentation network based on semantic guidance

ABSTRACT In this paper, we design a fast and accurate lightweight semantic segmentation method for mobile robots. Accurate semantic segmentation usually requires obtaining high-resolution feature maps with strong semantic representation. Among the widely utilized methods, dilated convolution is computationally intensive and the feature pyramid accuracy is not as accurate. Inspired by the Flow Alignment Module (FAM) proposed by SFNet, we propose a Semantic Guided Upsampling Module (SGUM) for learning semantic offsets between neighboring level feature maps to solve the semantic misalignment problem and effectively fuse high level semantic features into low-level high-resolution feature maps. The integration of the SGUM into the semantic segmentation network with different backbones shows better performance than other methods on various datasets, showing the effectiveness of the SGUM on semantic feature fusion as well as its generalizability.

CTSARF: A Chinese Text Similarity Analysis Model based on Residual Fusion

Semantic textual similarity analysis is a fundamental task in the field of natural language processing. Both representational and interactive models are extensively employed for this purpose. However, representational models often suffer from issues such as semantic defocus, semantic shift, and low accuracy. In contrast, interactive models can accurately capture the semantic focus of a text and extract ample semantic information, but hindered by high time costs and low retrieval efficiency. To address these challenges, this paper introduces a Chinese Text Similarity Analysis model based on Residual Fusion (CTSARF). This model effectively implements the semantic feature fusion of representational and interactive models through a constant mapping method using residual networks. Therefore CTSARF can achieve both high accuracy and retrieval efficiency simultaneously. In this study, we employ the cosine-based loss function CoSENT and the fine-tuned pre-trained model Mengzi to optimize performance in text similarity analysis tasks. Additionally, we utilize a data augmentation method based on text editing to enhance the robustness of CTSARF when evaluating text similarity on unbalanced datasets. We conducted optimization comparison experiments, benchmark model comparison experiments, and ablation experiments of the CTSARF model on four Chinese datasets. The results demonstrate improvements in accuracy, recall, precision, and F1 scores, thereby highlighting the necessity and superiority of the model fusion method based on residual networks.

Rotating box multi-objective visual tracking algorithm for vibration displacement measurement of large-span flexible bridges

Visual displacement measurement methods for flexible structural bodies like large-span bridges has gained wide popularity in recent years, but practical applications still have some limitations. For instance, when acquiring images of large-span flexible bridges at a distance, the slight angular tilt of the detection target due to irregular vibrations can cause extremely serious misfit errors in the displacement curves returned by the vision measurement algorithm. To improve the reliability of vibration displacement measurement of flexible structural bodies, this paper takes the bridge subjected to external excitation in the acquired image sequence as the object of vibration displacement measurement and uses a designed high-precision displacement measurement algorithm for a single-stage rotating target tracking anchor-free box to track the vibration displacement of the target in the flexible structural body. We first extract multi-scale feature information of bridge model image sequences using the improved YOLOv5-s backbone network and combine the Transformer self-attention mechanism with PANet to perform a top-down and bottom-up bi-directional fusion of target feature maps at three different scales to achieve semantic feature fusion of shallow and deep information. Second, the improved Efficient Decoupled Head performs the detection of rotating target centroid offset and bounding box size. Finally, the detected results are passed into the multi-objective tracking algorithm ByteTrack, which strengthens the spatio-temporal correlation between frames and obtains a better-fitting vibration displacement curve. The validation and comparison of traditional visual measurement methods and deep learning measurement methods on cable-stayed bridge models, small arch bridges, and large span bridges show that the vibration displacement trajectories regressed by the algorithm in this paper have the best fit with the actual vibration displacement trajectories, which also verifies that the algorithm in this paper has good potential for engineering applications and implementation space in the field of condition monitoring of flexible structural bodies.

Vulnerability detection through cross-modal feature enhancement and fusion

Software vulnerability detection is critical to computer security. Most existing vulnerability detection methods use single modal-based vulnerability detection models, which cannot effectively extract cross-modal features. To solve this problem, we propose a new multimodal deep learning based vulnerability detection method through a cross-modal feature enhancement and fusion. Firstly, we utilize a special compilation and debugging method to obtain the alignment relationship between source code statements and assembly instructions, as well as between source code variables and assembly code registers. Based on this alignment relationship and program slicing technology, we propose a cross-slicing method to generate bimodal program slices. Then, we propose a cross-modal feature enhanced code representation learning model to capture the fine-grained semantic correlation between source code and assembly code by using the co-attention mechanisms. Finally, vulnerability detection is achieved by feature level fusion of semantic features captured in fine-grained aligned source code and assembly code. Extensive experiments show that our method improves the performance of vulnerability detection compared with state-of-the-art methods. Specifically, our method achieves an accuracy of 97.4% and an F1-measure of 93.4% on the SARD dataset. An average accuracy of 95.4% and an F1-measure of 89.1% on two real-world software projects (i.e., FFmpeg and OpenSSL) is also achieved by our method, improving over SOTA method 4.5% and 2.9%.

Multi-semantic feature fusion attention network for binary code similarity detection

Binary code similarity detection (BCSD) plays a big role in the process of binary application security test. It can be applied in several fields, such as software plagiarism detection, malware analysis, vulnerability detection. Most research is based on recurrent neural networks, which is difficult to get the overall or long-distance semantic information of functions. Besides, exiting works simply extract high-level semantic features, lacking in-depth investigations on the potential mechanisms for fusing low-level and high-level semantic features. In this paper we propose a multi-semantic feature fusion attention network (MFFA-Net) for BCSD. MFFA-Net contains two critical modules: semantic feature fusion (SFF) and attention feature fusion (AFF). The SFF module concatenates multiple semantic features to represent the semantics of the function, which helps to obtain the overall semantic information of the function. The AFF module is designed to find useful information from various features, which assigns an attention matrix to research the relationship between features. In order to evaluate the proposed method, we made extensive experiments on two datasets. MFFA-Net can achieve a high degree of AUC at 99.6% and 98.3% respectively on the two datasets. The experimental results show that MFFA-Net has better performance for BCSD.

Pre-Trained Joint Model for Intent Classification and Slot Filling with Semantic Feature Fusion

The comprehension of spoken language is a crucial aspect of dialogue systems, encompassing two fundamental tasks: intent classification and slot filling. Currently, the joint modeling approach for these two tasks has emerged as the dominant method in spoken language understanding modeling. However, the existing joint models have limitations in terms of their relevancy and utilization of contextual semantic features between the multiple tasks. To address these limitations, a joint model based on BERT and semantic fusion (JMBSF) is proposed. The model employs pre-trained BERT to extract semantic features and utilizes semantic fusion to associate and integrate this information. The results of experiments on two benchmark datasets, ATIS and Snips, in spoken language comprehension demonstrate that the proposed JMBSF model attains 98.80% and 99.71% intent classification accuracy, 98.25% and 97.24% slot-filling F1-score, and 93.40% and 93.57% sentence accuracy, respectively. These results reveal a significant improvement compared to other joint models. Furthermore, comprehensive ablation studies affirm the effectiveness of each component in the design of JMBSF.

MULTI-SCALE ATTENTION BASED U-NET MODEL FOR LIVER TUMOR SEGMENTATION

It is essential to automatically evaluate the position and size of the liver tumour for radiologists, diagnosis, and the clinical process. Many U-Net-based variants have been suggested in recent years to enhance the segmentation results for medical image segmentation, but they are unable to describe the global spatial and channel relationships among lesion regions. To overcome this issues, we proposed a novel network called Multi-scale Attention UNet (MA-UNet) to address this problem by adding a self-attention mechanism into our approach to adaptively combine local features with their global dependencies. The attention mechanism of the MA-UNet allows it to capture complex contextual dependencies. Position-wise Attention Block and Multi-scale Fusion Attention Block are the two blocks that we have developed. The feature interdependencies in spatial dimensions, which represent the spatial dependencies between pixels in a global view, are modelled using the Position-wise Attention Block. A multi-scale semantic feature fusion attention block is also used to capture the channel dependencies between any feature map. On the MICCAI 2017 LiTS Competition dataset, we assess our methodology. Compared to other cutting-edge methods, the suggested way performs better. The Dice and VOE values of liver tumors segmentation are 0.749 ± 0.08 and 0.21 ± 0.06 respectively. KEYWORDS: Liver tumor segmentation, Attention mechanism, Deep learning.