Fusion Of Spatial Information Research Articles

A joint local spatial and global temporal CNN-Transformer for dynamic facial expression recognition

Unlike conventional video action recognition, Dynamic Facial Expression Recognition (DFER) tasks exhibit minimal spatial movement of objects. Addressing this distinctive attribute, we propose an innovative CNN-Transformer model, named LSGTNet, specifically tailored for DFER tasks. Our LSGTNet comprises three stages, each composed of a spatial CNN (Spa-CNN) and a temporal transformer (T-Former) in sequential order. The Spa-CNN extracts spatial features from images, yielding smaller-sized feature maps to alleviate the computational complexity for subsequent T-Former. The T-Former integrates global temporal information from the same spatial positions across different time frames while retaining the feature map dimensions. The alternating interplay between Spa-CNN and T-Former ensures a continuous fusion of spatial and temporal information, leading our model to excel across various real-world datasets. To the best of our knowledge, this is the first method to address the DFER challenge by focusing on capturing the temporal changes in muscles within local spatial regions. Our method has achieved state-of-the-art results on multiple in-the-wild datasets and datasets under laboratory conditions.

Multi-Scale Fusion Siamese Network Based on Three-Branch Attention Mechanism for High-Resolution Remote Sensing Image Change Detection

Remote sensing image change detection (CD) is an important means in remote sensing data analysis tasks, which can help us understand the surface changes in high-resolution (HR) remote sensing images. Traditional pixel-based and object-based methods are only suitable for low- and medium-resolution images, and are still challenging for complex texture features and detailed image detail processing in HR images. At present, the method based on deep learning has problems such as inconsistent fusion and difficult model training in the combination of the difference feature information of the deep and shallow layers and the attention mechanism, which leads to errors in the distinction between the changing region and the invariant region, edge detection and small target detection. In order to solve the above problems of inconsistent fusions of feature information aggregation and attention mechanisms, and indistinguishable change areas, we propose a multi-scale feature fusion Siamese network based on attention mechanism (ABMFNet). To tackle the issues of inconsistent fusion and alignment difficulties when integrating multi-scale fusion and attention mechanisms, we introduce the attention-based multi-scale feature fusion module (AMFFM). This module not only addresses insufficient feature fusion and connection between different-scale feature layers, but also enables the model to automatically learn and prioritize important features or regions in the image. Additionally, we design the cross-scale fusion module (CFM) and the difference feature enhancement pyramid structure (DEFPN) to assist the AMFFM module in integrating differential information effectively. These modules bridge the spatial disparity between low-level and high-level features, ensuring efficient connection and fusion of spatial difference information. Furthermore, we enhance the representation and inference speed of the feature pyramid by incorporating a feature enhancement module (FEM) into DEFPN. Finally, the BICD dataset proposed by the laboratory and public datasets LEVIR-CD and BCDD are compared and tested. We use F1 score and MIoU values as evaluation metrics. For AMBMFNet, the F1 scores on the three datasets are 77.69%, 81.57%, and 77.91%, respectively, while the MIoU values are 84.65%, 85.84%, and 84.54%, respectively. The experimental results show that ABMFNet has better effectiveness and robustness.

A DFRC Based on Multichannel and Spatial Information Fusion for Multiradar Communication

A DFRC Based on Multichannel and Spatial Information Fusion for Multiradar Communication

Moving Object Detection in Freely Moving Camera via Global Motion Compensation and Local Spatial Information Fusion.

Motion object detection (MOD) with freely moving cameras is a challenging task in computer vision. To extract moving objects, most studies have focused on the difference in motion features between foreground and background, which works well for dynamic scenes with relatively regular movements and variations. However, abrupt illumination changes and occlusions often occur in real-world scenes, and the camera may also pan, tilt, rotate, and jitter, etc., resulting in local irregular variations and global discontinuities in motion features. Such complex and changing scenes bring great difficulty in detecting moving objects. To solve this problem, this paper proposes a new MOD method that effectively leverages local and global visual information for foreground/background segmentation. Specifically, on the global side, to support a wider range of camera motion, the relative inter-frame transformations are optimized to absolute transformations referenced to intermediate frames in a global form after enriching the inter-frame matching pairs. The global transformation is fine-tuned using the spatial transformer network (STN). On the local side, to address the problem of dynamic background scenes, foreground object detection is optimized by utilizing the pixel differences between the current frame and the local background model, as well as the consistency of local spatial variations. Then, the spatial information is combined using optical flow segmentation methods, enhancing the precision of the object information. The experimental results show that our method achieves a detection accuracy improvement of over 1.5% compared with the state-of-the-art methods on the datasets of CDNET2014, FBMS-59, and CBD. It demonstrates significant effectiveness in challenging scenarios such as shadows, abrupt changes in illumination, camera jitter, occlusion, and moving backgrounds.

Design of urban road fault detection system based on artificial neural network and deep learning.

In urban traffic management, the timely detection of road faults plays a crucial role in improving traffic efficiency and safety. However, conventional methods often fail to fully leverage the information from road topology and traffic data. To address this issue, we propose an innovative detection system that combines Artificial Neural Networks (ANNs), specifically Graph Convolutional Networks (GCN), Bidirectional Gated Recurrent Units (BiGRU), and self-attention mechanisms. Our approach begins by representing the road topology as a graph and utilizing GCN to model it. This allows us to learn the relationships between roads and capture their structural dependencies. By doing so, we can effectively incorporate the spatial information provided by the road network. Next, we employ BiGRU to model the historical traffic data, enabling us to capture the temporal dynamics and patterns in the traffic flow. The BiGRU architecture allows for bidirectional processing, which aids in understanding the traffic conditions based on both past and future information. This temporal modeling enhances our system's ability to handle time-varying traffic patterns. To further enhance the feature representations, we leverage self-attention mechanisms. By combining the hidden states of the BiGRU with self-attention, we can assign importance weights to different temporal features, focusing on the most relevant information. This attention mechanism helps to extract salient features from the traffic data. Subsequently, we merge the features learned by GCN from the road topology and BiGRU from the traffic data. This fusion of spatial and temporal information provides a comprehensive representation of the road status. By employing a Multilayer Perceptron (MLP) as a classifier, we can effectively determine whether a road is experiencing a fault. The MLP model is trained using labeled road fault data through supervised learning, optimizing its performance for fault detection. Experimental evaluations of our system demonstrate excellent performance in road fault detection. Compared to traditional methods, our system achieves more accurate fault detection, thereby improving the efficiency of urban traffic management. This is of significant importance for city administrators, as they can promptly identify road faults and take appropriate measures for repair and traffic diversion.

A short note on deep contextual spatial and spectral information fusion for hyperspectral image processing: Case of pork belly properties prediction

AbstractThis study demonstrates a new approach to process hyperspectral images where both the contextual spatial information as well as the spectral information are used to predict sample properties. The deep contextual spatial information is extracted using the deep feature extraction from pretrained resnet‐18 deep learning architecture, while the spectral information was readily available as the average pixel values. To fuse the information in a complementary way, a multiblock modeling approach called sequential orthogonalized partial least squares was used. The sequential model guarantees that the information learned is complementary from spatial and spectral domains. The potential of the approach is demonstrated to predict several physical and chemical properties in pork bellies.

Tomato leaf disease detection based on attention mechanism and multi-scale feature fusion.

When detecting tomato leaf diseases in natural environments, factors such as changes in lighting, occlusion, and the small size of leaf lesions pose challenges to detection accuracy. Therefore, this study proposes a tomato leaf disease detection method based on attention mechanisms and multi-scale feature fusion. Firstly, the Convolutional Block Attention Module (CBAM) is introduced into the backbone feature extraction network to enhance the ability to extract lesion features and suppress the effects of environmental interference. Secondly, shallow feature maps are introduced into the re-parameterized generalized feature pyramid network (RepGFPN), constructing a new multi-scale re-parameterized generalized feature fusion module (BiRepGFPN) to enhance feature fusion expression and improve the localization ability for small lesion features. Finally, the BiRepGFPN replaces the Path Aggregation Feature Pyramid Network (PAFPN) in the YOLOv6 model to achieve effective fusion of deep semantic and shallow spatial information. Experimental results indicate that, when evaluated on the publicly available PlantDoc dataset, the model's mean average precision (mAP) showed improvements of 7.7%, 11.8%, 3.4%, 5.7%, 4.3%, and 2.6% compared to YOLOX, YOLOv5, YOLOv6, YOLOv6-s, YOLOv7, and YOLOv8, respectively. When evaluated on the tomato leaf disease dataset, the model demonstrated a precision of 92.9%, a recall rate of 95.2%, an F1 score of 94.0%, and a mean average precision (mAP) of 93.8%, showing improvements of 2.3%, 4.0%, 3.1%, and 2.7% respectively compared to the baseline model. These results indicate that the proposed detection method possesses significant detection performance and generalization capabilities.

Deep Fusion Module for Video Action Recognition

In video action recognition, effective spatiotemporal modeling is crucial. However, traditional two-stream methods face challenges in integrating spatial information from RGB images and temporary information from optical flow without long-range temporal modelling. To address these limitations, we propose the Deep Fusion Module (DFM), which focuses on the deep fusion of spatial and temporal information and consists of two components. First, we propose an Attention Fusion Module (AFM) to effectively fuse the shallow features obtained from a two-stream network, thereby facilitating the integration of spatial and temporal information. Next, we incorporate a SpatioTemporal Module (STM), comprising a ConvGRU and a 1×1 convolution, to model long-range temporal dependency and fuse spatial-temporal features. Experiments on the UCF101 dataset show that our method achieves 96.5% accuracy, outperforming baseline two-stream models by 0.3%.

HRD-Net: High resolution segmentation network with adaptive learning ability of retinal vessel features

Retinal segmentation is a crucial step in the early warning of human health conditions. However, retinal blood vessels possess complex curvature, irregular distribution, and contain multi-scale fine structures, which make the limited receptive field of regular convolution challenging to process their vascular details efficiently. Additionally, the encoder-decoder based network leads to irreversible spatial information loss because of multiple downsampling, resulting in over-segmentation and missed segmentation of the vessels. For this reason, we develop a high-resolution network based on Deformable Convolution v3, called HRD-Net. By constructing a high-resolution representation, the network allows special attention to be paid to the details of tiny blood vessels. The proposed feature enhancement cascade module based on Deformable Convolution v3 can flexibly adapt and capture the ever-changing morphology and intricate connections of retinal blood vessels, ensuring the continuity of vessel segmentation. In the output phase of the network, the proposed global aggregation module integrates full-resolution feature maps while suppressing redundant features, achieving an effective fusion of high-level semantic information and spatial detail information. In addition, we have re-examined the selection criteria for activation and normalization methods, and also refine the network architectures from a spatial domain perspective to release redundant computational loads. Testing on the DRIVE, STARE, and CHASE_DB1 datasets indicates that HRD-Net, with fewer parameters, outperforms existing segmentation methods on several evaluation metrics such as F1, ACC, SE, SP, AUC, and IOU.

LFSimCC: Spatial fusion lightweight network for human pose estimation

To address the limitations of existing 2D human pose estimation methods in terms of speed and lightweight, we propose a method called Lightweight Fusion SimCC (LFSimCC). LFSimCC incorporates two modules: LiteFNet, which enhances multi-scale spatial information fusion, and LKC-GAU, which improves the modeling capability of spatial information. Specifically, LiteFNet utilizes a combination of self-attention mechanism and novel spatial convolution to enable feature maps to capture richer multi-level global feature representations within the network. On the other hand, LKC-GAU enhances SimCC’s ability to capture spatial relationships between joints by incorporating a large kernel of convolution and a self-attention mechanism. Furthermore, we design a keypoint information fusion loss (IFL) that enhances the model’s sensitivity to information between keypoints in the human body. Experimental results demonstrate that our method is capable of extracting more decisive information and suppressing redundant feature representations, leading to high recognition accuracy and low inference latency.

Enhancing Major Depressive Disorder Diagnosis with Dynamic-Static Fusion Graph Neural Networks.

Major Depressive Disorder (MDD) is a debilitating, complex mental condition with unclear mechanisms hindering diagnostic progress. Research links MDD to abnormal brain connectivity using functional magnetic resonance imaging (fMRI). Yet, existing fMRI-based MDD models suffer from limitations, including neglecting dynamic network traits, lacking interpretability, and struggling with small datasets. We present DSFGNN, a novel graph neural network framework addressing these issues for improved MDD diagnosis. DSFGNN employs a graph isomorphism encoder to model static and dynamic brain networks, achieving effective fusion of temporal and spatial information through a spatiotemporal attention mechanism, thereby enhancing interpretability. Furthermore, we incorporate a causal disentangling module and orthogonal regularization module to augment the model's expressiveness. We evaluate DSFGNN on the Rest-meta-MDD dataset, yielding superior results compared to the best baseline. Besides, extensive ablation studies and interpretability analysis confirm DSFGNN's effectiveness and potential for biomarker discovery.

SwinHCST: a deep learning network architecture for scene classification of remote sensing images based on improved CNN and Transformer

ABSTRACTRemote sensing image scene classification is a fundamental task in intelligent interpretation of remote sensing images. Although Transformers possess a powerful attention mechanism, they require lengthy training procedures to achieve good performance levels. To address this issue, this paper proposes a novel deep learning network model by combining CNN and Swin Transformer named SwinHCST. Firstly, the model uses Weighted Normalized CNN to quickly extract low-level features of the image. Secondly, the Receptive Field Block module facilitates multi-scale information fusion, Thirdly, the Information Fusion Transformer further excavates the deep-level features of the image. Furthermore, this paper has designed a plug-and-play Cross Spatial Information Fusion Block, which is used to encodes dimensional information and extracts global information to enhance information exchange. The scene classification experiments show that the proposed model outperforms other methods on the three selected datasets and can achieve excellent performance without requiring large amounts of data and training. Specifically, the classification accuracy of the proposed method on the three datasets is 93.76%, 93.60%, and 98.10%, which is 1.7% to 3.71% higher than ResNet50 and 3.7% to 5.7% higher than Swin Transformer.

Prediction of production indicators of fractured-vuggy reservoirs based on improved Graph Attention Network

Various storage and seepage spaces exist in fractured-vuggy carbonate reservoirs composed of multi-scale dissolution pores and fractures. The frequent regulation of the working system causes nonlinear and unstable production data along with complex water breakthrough characteristics, which lead to difficulty in real-time prediction. Traditional methods based on the water drive curve and autoregressive machine learning ignore the spatial correlation among production wells and local geological characteristics. To address these problems, we propose a model based on improved attention, the spatiotemporal multi-graph convolutional network (STMCN), for production prediction. The unit production wells are abstracted as directed graph network nodes to establish adjacency, connectivity and correlation graphs to extract the spatial and semantic features from different perspectives through graph convolutional networks. To depict the law of fluid movement, we realize the fusion of spatial and semantic information through dynamic routing. Aiming at the autocorrelation characteristics of production sequences, the model uses the self-attention mechanism to capture the dependencies in production sequences. The gating mechanism is designed to achieve dynamic production prediction by adaptively aggregating the spatiotemporal characteristics. This paper evaluates the predictive performance of our model by two real-world datasets of the Tahe Oilfield. The results show that the mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE) of our proposed model for the prediction of well A2 were 3.19, 4.66 and 0.07, respectively, which were better than the relatively new baseline model.

SFHG-YOLO: A Simple Real-Time Small-Object-Detection Method for Estimating Pineapple Yield from Unmanned Aerial Vehicles.

The counting of pineapple buds relies on target recognition in estimating pineapple yield using unmanned aerial vehicle (UAV) photography. This research proposes the SFHG-YOLO method, with YOLOv5s as the baseline, to address the practical needs of identifying small objects (pineapple buds) in UAV vision and the drawbacks of existing algorithms in terms of real-time performance and accuracy. Field pineapple buds are small objects that may be detected in high density using a lightweight network model. This model enhances spatial attention and adaptive context information fusion to increase detection accuracy and resilience. To construct the lightweight network model, the first step involves utilizing the coordinate attention module and MobileNetV3. Additionally, to fully leverage feature information across various levels and enhance perception skills for tiny objects, we developed both an enhanced spatial attention module and an adaptive context information fusion module. Experiments were conducted to validate the suggested algorithm's performance in detecting small objects. The SFHG-YOLO model exhibited significant gains in assessment measures, achieving mAP@0.5 and mAP@0.5:0.95 improvements of 7.4% and 31%, respectively, when compared to the baseline model YOLOv5s. Considering the model size and computational cost, the findings underscore the superior performance of the suggested technique in detecting high-density small items. This program offers a reliable detection approach for estimating pineapple yield by accurately identifying minute items.

Soil Heavy-Metal Pollution Prediction Methods Based on Two Improved Neural Network Models

Current soil pollution prediction methods need improvement, especially with regard to accuracy in supplementing missing heavy-metal values in soil, and the accuracy and slow convergence speed of methods for predicting heavy-metal content at unknown points. To reduce costs and improve prediction accuracy, this study used two neural network models (SA-FOA-BP and SE-GCN) to supplement missing heavy-metal values and efficiently predict heavy-metal content in soil. The SA-FOA-BP model combines simulated annealing and fruit fly algorithms to optimize the parameter search method in traditional BP neural networks and improve prediction of missing heavy-metal values in soil. A spatial information fusion graph convolutional network prediction model (SE-GCN) constructs a spatial information encoder that can perceive spatial context information, and embeds it with spatial autocorrelation used for auxiliary learning to predict the heavy-metal content in soil. From the experimental results, the SE-GCN model demonstrates improved performance in terms of evaluation indicators compared with other models. Application analysis of the two improved neural network models was conducted; application scenarios and suitability were analyzed, showing that these models have practical application value for soil pollution prediction.

Multi-object detection for crowded road scene based on ML-AFP of YOLOv5

Aiming at the problem of multi-object detection such as target occlusion and tiny targets in road scenes, this paper proposes an improved YOLOv5 multi-object detection model based on ML-AFP (multi-level aggregation feature perception) mechanism. Since tiny targets such as non-motor vehicle and pedestrians are not easily detected, this paper adds a micro target detection layer and a double head mechanism to improve the detection ability of tiny targets. Varifocal loss is used to achieve a more accurate ranking in the process of non-maximum suppression to solve the problem of target occlusion, and this paper also proposes a ML-AFP mechanism. The adaptive fusion of spatial feature information at different scales improves the expression ability of network model features, and improves the detection accuracy of the model as a whole. Our experimental results on multiple challenging datasets such as KITTI, BDD100K, and show that the accuracy, recall rate and mAP value of the proposed model are greatly improved, which solves the problem of multi-object detection in crowded road scenes.

Prediction of Au-Polymetallic Deposits Based on Spatial Multi-Layer Information Fusion by Random Forest Model in the Central Kunlun Area of Xinjiang, China

In recent years, there has been a growing emphasis on combining intelligent prospecting algorithms, such as random forest, with extensive geological and mineral data for the purpose of quantitatively predicting exploration geochemistry. This approach holds significant importance for enhancing the accuracy of target delineation. The central Kunlun area in Xinjiang possesses highly favorable ore-forming geological conditions, offering excellent prospects for mineral exploration. However, the depletion of shallow deposits coupled with a decade-long gap in geological exploration have presented increasing challenges in the quest to discover substantial metal resources. Consequently, there is now a severe shortage of reserve assets in the region, prompting an urgent need for the implementation of new theories, methods, and technologies in mineral resource investigation and evaluation efforts. The researchers used geological and regional geochemical data to construct a random forest metallogenic discriminant model for predicting the mineralization of gold polymetallic minerals in the central Kunlun area of Xinjiang and delineating the metallogenic target area. Two different sampling methods were compared to quantitatively predict gold polymetallic mineral resources. The results indicate that the selected training samples offer higher prediction accuracy and reliability by fully capturing the complex information of the original data. The random forest model using select training samples has valuable applications in metallogenic prospect prediction and potential division due to its ability to consider the actual exploration cost and identify small areas with high potential and a high proportion of ore. This study significantly improves prediction accuracy, reduces exploration risk, and expands the use of machine learning algorithms in mathematical geology in the central Kunlun area of Xinjiang.

An Intelligent Detection Method for Small and Weak Objects in Space

In the case of a boom in space resource development, space debris will increase dramatically and cause serious problems for the spacecraft in orbit. To address this problem, a novel context sensing-YOLOv5 (CS-YOLOv5) is proposed for small and weak space object detection, which could realize the extraction of local context information and the enhancement and fusion of spatial information. To enhance the expression ability of feature information and the identification ability of the network, we propose the cross-layer context fusion module (CCFM) through multiple branches in parallel to learn the context information of different scales. At the same time, to map the small-scale features sequentially to the features of the previous layer, we design the adaptive weighting module (AWM) to assist the CCFM in further enhancing the expression of features. Additionally, to solve the problem that the spatial information of small objects is easily lost, we designed the spatial information enhancement module (SIEM) to adaptively learn the weak spatial information of small objects that need to be protected. To further enhance the generalization ability of CS-YOLOv5, we propose a contrast mosaic data augmentation to enrich the diversity of the sample. Extensive experiments are conducted on self-built datasets, which strongly prove the effectiveness of our method in space object detection.

An Adaptive Multisensor Fault Diagnosis Method for High-Speed Train Bogie

High-speed train bogies are the critical components of high-speed trains, which can play the role of traction, braking, and buffering. In long-term train service, bogies are prone to wear and aging. Currently, most studies on fault diagnosis methods are aimed at single equipment identification. It is challenging to accurately diagnose the faults of such coupled multi-equipment combination systems as bogies. Multiple devices on the bogie have implied correlations in space, and fully exploiting their spatial features enhances the fault diagnosis accuracy. This paper proposes a new bogie fault diagnosis method based on the directional graph of train bogie: RS-GAT model. The model uses the Residual-Squeeze Network (RS-Net) to construct the framework of the model and use the Graph Attention Network (GAT) for spatial information fusion and feature extraction to identify bogie faults. Using six datasets collected under the operation of high-speed trains, experimental results demonstrate that the proposed approach has better effectiveness than the RS-Net class model and single-layer graph class model, with diagnosis accuracy near 96%. Ablation experiments and comparisons between RS-GAT and RS-GCN verify the effectiveness of RS-Net framework and GAT model in fault classification. In addition, the RS-GAT model is found to have strong robustness when different models are analyzed using small-scale data sets.

Research on Real-time and High-precision Positioning Method of Ground Target through UAV Stereo Vision and Spatial Information Fusion

Background: Positioning accuracy is the most important index of the reconnaissance positioning system. Positioning accuracy involves many factors, such as the position, attitude and motion state of the flight platform, the pointing accuracy of the stable platform, and various coordinate transformations. A reasonable fusion strategy can guarantee stable positioning accuracy, so it is very important to study the precise fusion of machine vision, 3D geographic data and UAV airborne positioning information to realize the optimal combination of positioning data and complete the accurate and rapid positioning of the ground target. Methods: In this paper, a location model based on stereo vision and spatial information fusion method is proposed. It fully integrates visual information, satellite positioning information and spatial geographic information, greatly improves positioning accuracy, and through a real-time processing algorithm, significantly improves real-time positioning. Results: Through the related experiments, positioning accuracy and real-time ability of positioning could reach about three meters. Conclusion: The proposed real-time and high-precision positioning method of the ground target through UAV stereo vision and spatial information fusion is proposed showing significant improvement compared to other traditional methods.