Fusion Of Spatial Information Research Articles

Research on Land Use and Land Cover Information Extraction Methods for Remote Sensing Images Based on Improved Convolutional Neural Networks

To address the challenges that convolutional neural networks (CNNs) face in extracting small objects and handling class imbalance in remote sensing imagery, this paper proposes a novel spatial contextual information and multiscale feature fusion encoding–decoding network, SCIMF-Net. Firstly, SCIMF-Net employs an improved ResNeXt-101 deep backbone network, significantly enhancing the extraction capability of small object features. Next, a novel PMFF module is designed to effectively promote the fusion of features at different scales, deepening the model’s understanding of global and local spatial contextual information. Finally, introducing a weighted joint loss function improves the SCIMF-Net model’s performance in extracting LULC information under class imbalance conditions. Experimental results show that compared to other CNNs such as Res-FCN, U-Net, SE-U-Net, and U-Net++, SCIMF-Net improves PA by 0.68%, 0.54%, 1.61%, and 3.39%, respectively; MPA by 2.96%, 4.51%, 2.37%, and 3.45%, respectively; and MIOU by 3.27%, 4.89%, 4.2%, and 5.68%, respectively. Detailed comparisons of locally visualized LULC information extraction results indicate that SCIMF-Net can accurately extract information from imbalanced classes and small objects.

A graph-guided transformer based on dual-stream perception for hyperspectral image classification

ABSTRACT The excellent capabilities of Transformers and Graph Neural Networks (GNNs) in modelling long-range dependencies and handling irregular data have led to their widespread application in hyperspectral image (HSI) classification tasks. However, the Graph Transformer combining both advantages is rarely used in this field and has some limitations. Current Graph Transformers consider interactions between all nodes within the graph, adding complexity and introducing unnecessary information from noisy nodes. Moreover, the rich spectral information in HSIs is often ignored, and there is a lack of effective fusion of spatial information. In this paper, we propose a dual-stream graph-guided Transformer for HSI classification. In spatial dimension, superpixels are utilized to guide spatial graph generation, capturing global topological dependencies and local details effectively in HSIs. In terms of spectrum, we innovatively construct a spectral graph based on spectral channels and adopt a contribution score-based strategy to adaptively filter out irrelevant edges, achieving sparsity while preserving spectral context relationships. Experimental results demonstrate the significant competitive advantage of our method in HSI classification tasks on three public datasets. The code is available at https://github.com/youngboy03/GTDPNet.

CP-RDM: a new object detection algorithm for casting and pouring robots

Abstract Automating the casting sector heavily relies on pivotal technology for object detection in pouring robots. A sophisticated algorithm designed to identify and locate target pouring holes in intricate casting workshops is crucial for advancing the intelligence of the casting process. However, the workshop environment for pouring is generally challenging, with uneven lighting, varying sizes of pouring holes, and significant occlusion in the target area, all impacting the accuracy of target detection tasks.
To overcome these challenges, this paper proposes enhancing the YOLOv8s algorithm for object detection in pouring robots. Firstly, to address the issue of different scales in pouring holes, a Multi-Scale Residual Channel and Spatial Information Fusion Module (MRCS) is designed to aggregate channel and spatial information, thereby enhancing the feature extraction capability of the model. The proposed enhancement is validated on the Pascal VOC dataset. Secondly, a SimAM attention mechanism is added at the end of the backbone network to focus the object detection network more on the positional region of the pouring hole. Importantly, this addition does not introduce extra parameters or computational burden to the model.
Finally, in the detection part of the model, the detection head from the RT-DETR model is introduced. This combination of real-time detection capability from YOLO and deep feature extraction capability from RT-DETR enhances the detection accuracy of the model while ensuring real-time performance. Experimental results on the updated pouring hole dataset reveal that, with only a slight increase in parameters, the proposed model achieves a 2.5% and 3.5% improvement in mAP@0.5 and F1-Score, respectively, compared to the baseline algorithm YOLOv8s. Precision (P) is enhanced by 1.8%, recall (R) by 3.5%, and PFS reaches 110, meeting the requirements for real-time pouring in pouring robots.

Domain knowledge-enhanced multi-spatial multi-temporal [formula omitted] forecasting with integrated monitoring and reanalysis data

Accurate air quality forecasting is crucial for public health, environmental monitoring and protection, and urban planning. However, existing methods fail to effectively utilize multi-scale information, both spatially and temporally. There is a lack of integration between individual monitoring stations and city-wide scales. Temporally, the periodic nature of air quality variations is often overlooked or inadequately considered. To overcome these limitations, we conduct a thorough analysis of the data and tasks, integrating spatio-temporal multi-scale domain knowledge. We present a novel Multi-spatial Multi-temporal air quality forecasting method based on Graph Convolutional Networks and Gated Recurrent Units (M2G2), bridging the gap in air quality forecasting across spatial and temporal scales. The proposed framework consists of two modules: Multi-scale Spatial GCN (MS-GCN) for spatial information fusion and Multi-scale Temporal GRU (MT-GRU) for temporal information integration. In the spatial dimension, the MS-GCN module employs a bidirectional learnable structure and a residual structure, enabling comprehensive information exchange between individual monitoring stations and the city-scale graph. Regarding the temporal dimension, the MT-GRU module adaptively combines information from different temporal scales through parallel hidden states. Leveraging meteorological indicators and four air quality indicators, we present comprehensive comparative analyses and ablation experiments, showcasing the higher accuracy of M2G2 in comparison to nine currently available advanced approaches across all aspects. The improvements of M2G2 over the second-best method on RMSE of 72-h future predictions are as follows: PM2.5: 6%∼10%; PM10: 5%∼7%; NO2: 5%∼16%; O3: 6%∼9%. Furthermore, we demonstrate the effectiveness of each module of M2G2 by ablation study. We conduct a sensitivity analysis of air quality and meteorological data, finding that the introduction of O3 adversely impacts the prediction accuracy of PM2.5.

Hyperspectral Rock Classification Method Based on Spatial-Spectral Multidimensional Feature Fusion

The issues of the same material with different spectra and the same spectra for different materials pose challenges in hyperspectral rock classification. This paper proposes a multidimensional feature network based on 2-D convolutional neural networks (2-D CNNs) and recurrent neural networks (RNNs) for achieving deep combined extraction and fusion of spatial information, such as the rock shape and texture, with spectral information. Experiments are conducted on a hyperspectral rock image dataset obtained by scanning 81 common igneous and metamorphic rock samples using the HySpex hyperspectral sensor imaging system to validate the effectiveness of the proposed network model. The results show that the model achieved an overall classification accuracy of 97.925% and an average classification accuracy of 97.956% on this dataset, surpassing the performances of existing models in the field of rock classification.

A coordinated adaptive multiscale enhanced spatio-temporal fusion network for multi-lead electrocardiogram arrhythmia detection

The multi-lead electrocardiogram (ECG) is widely utilized in clinical diagnosis and monitoring of cardiac conditions. The advancement of deep learning has led to the emergence of automated multi-lead ECG diagnostic networks, which have become essential in the fields of biomedical engineering and clinical cardiac disease diagnosis. Intelligent ECG diagnosis techniques encompass Recurrent Neural Networks (RNN), Transformers, and Convolutional Neural Networks (CNN). While CNN is capable of extracting local spatial information from images, it lacks the ability to learn global spatial features and temporal memory features. Conversely, RNN relies on time and can retain significant sequential features. However, they are not proficient in extracting lengthy dependencies of sequence data in practical scenarios. The self-attention mechanism in the Transformer model has the capability of global feature extraction, but it does not adequately prioritize local features and cannot extract spatial and channel features. This paper proposes STFAC-ECGNet, a model that incorporates CAMV-RNN block, CBMV-CNN block, and TSEF block to enhance the performance of the model by integrating the strengths of CNN, RNN, and Transformer. The CAMV-RNN block incorporates a coordinated adaptive simplified self-attention module that adaptively carries out global sequence feature retention and enhances spatial–temporal information. The CBMV-CNN block integrates spatial and channel attentional mechanism modules in a skip connection, enabling the fusion of spatial and channel information. The TSEF block implements enhanced multi-scale fusion of image spatial and sequence temporal features. In this study, comprehensive experiments were conducted using the PTB-XL large publicly available ECG dataset and the China Physiological Signal Challenge 2018 (CPSC2018) database. The results indicate that STFAC-ECGNet surpasses other cutting-edge techniques in multiple tasks, showcasing robustness and generalization.

PCB plug-in solder joint defect detection method based on coordinated attention-guided information fusion

Printed Circuit Boards (PCBs) are the foundational component of electronic devices, and the detection of PCB defects is essential for ensuring the quality control of electronic products. Aiming at the problem that the existing PCB plug-in solder defect detection algorithms cannot meet the requirements of high precision, low false alarm rate, and high speed at the same time, this paper proposes a method based on spatial convolution pooling and information fusion. Firstly, on the basis of YOLOv3, an attention-guided pyramid structure is used to fuse context information, and multiple convolutions of different size are used to explore richer high-level semantic information; Secondly, a coordinated attention network structure is introduced to calibrate the fused pyramidal feature information, highlighting the important feature channels, and reducing the adverse impact of redundant parameters generated by feature fusion; Finally, the ASPP (Atrous Spatial Pyramid Pooling) structure is implemented in the original Darknet53 backbone feature extraction network to acquire multi-scale feature information of the detection targets. With these improvements, the average detection accuracy of the enhanced network has been elevated from 94.45 to 96.43%. This experiments shows that the improved network is more suitable for PCB plug-in solder defect detection applications.

Enhancing Major Depressive Disorder Diagnosis With Dynamic-Static Fusion Graph Neural Networks.

Major Depressive Disorder (MDD) is a debilitating, complex mental condition with unclear mechanisms hindering diagnostic progress. Research links MDD to abnormal brain connectivity using functional magnetic resonance imaging (fMRI). Yet, existing fMRI-based MDD models suffer from limitations, including neglecting dynamic network traits, lacking interpretability, and struggling with small datasets. We present DSFGNN, a novel graph neural network framework addressing these issues for improved MDD diagnosis. DSFGNN employs a graph isomorphism encoder to model static and dynamic brain networks, achieving effective fusion of temporal and spatial information through a spatiotemporal attention mechanism, thereby enhancing interpretability. Furthermore, we incorporate a causal disentangling module and orthogonal regularization module to augment the model's expressiveness. We evaluate DSFGNN on the Rest-meta-MDD dataset, yielding superior results compared to the best baseline. Besides, extensive ablation studies and interpretability analysis confirm DSFGNN's effectiveness and potential for biomarker discovery.

Identification of chrysanthemum variety via hyperspectral imaging and wavelength selection based on multitask particle swarm optimization

Chrysanthemum, a widely favored flower tea, contains numerous phytochemicals for health benefits. Due to the different geographical origins and processing technics, its variety has a direct influence on the phytochemical content and pharmacological effect. Accordingly, an accurate identification for chrysanthemum varieties is significant for quality detection and market supervision. In this study, the hyperspectral imaging (HSI) combined with chemometrics methods was exploited to identify the chrysanthemum varieties. First, to alleviate the problem of easily trapping into local optimum in traditional spectral variable selection methods, the multi-tasking particle swarm optimization (MTPSO) was developed to select the key wavelengths by dividing hundreds of variables into low-dimensional subtasks. Second, to enrich the feature information, the spatial texture and color features contained in hyperspectral images were extracted and applied to chrysanthemum identification for the first time. Finally, an ensemble learning model, extreme gradient boosting (XGBoost), was constructed to conduct the chrysanthemum variety classification due to its strong generalization ability. Experimental results showed that the proposed MTPSO achieved the identification accuracy of 96.89%, and increased by 1.11–5.91% than classical spectral feature selection methods. Furthermore, after the involvement of spatial image information, the classification accuracy using spatial–spectral features was improved further, and reached 98.39%. Overall, this study highlights that the feature fusion of key wavelengths and spatial information is more effective for chrysanthemum variety identification, and can also provide technical reference for other HSI-related applications.

An Overview of Behavioral Recognition

Human behavior recognition has become a popular research topic in the field of computer vision. With the introduction of deep learning and attention mechanisms, this field has been further promoted. However, issues such as dataset acquisition and preprocessing operations on multimodal datasets, modeling of long time information in videos, and fusion of temporal and spatial information still exist. In this paper, we first outline some video action recognition datasets and related preprocessing techniques, including frame extraction, optical flow extraction, and skeletal feature acquisition. Then, the relevant models are classified and parsed according to their characteristics and the types of input data modalities. In addition, we evaluate the performance of the models on several benchmark datasets to gain a deeper understanding of the model development process. Finally, we summarized the current challenges faced in the field of video behavior recognition, including model timeliness, data set subjectivity and effective fusion of multi-modal features, and proposed possible future improvement directions in order to provide more ideas and methods for subsequent research.

A joint local spatial and global temporal CNN-Transformer for dynamic facial expression recognition

Unlike conventional video action recognition, Dynamic Facial Expression Recognition (DFER) tasks exhibit minimal spatial movement of objects. Addressing this distinctive attribute, we propose an innovative CNN-Transformer model, named LSGTNet, specifically tailored for DFER tasks. Our LSGTNet comprises three stages, each composed of a spatial CNN (Spa-CNN) and a temporal transformer (T-Former) in sequential order. The Spa-CNN extracts spatial features from images, yielding smaller-sized feature maps to alleviate the computational complexity for subsequent T-Former. The T-Former integrates global temporal information from the same spatial positions across different time frames while retaining the feature map dimensions. The alternating interplay between Spa-CNN and T-Former ensures a continuous fusion of spatial and temporal information, leading our model to excel across various real-world datasets. To the best of our knowledge, this is the first method to address the DFER challenge by focusing on capturing the temporal changes in muscles within local spatial regions. Our method has achieved state-of-the-art results on multiple in-the-wild datasets and datasets under laboratory conditions.

Multi-Scale Fusion Siamese Network Based on Three-Branch Attention Mechanism for High-Resolution Remote Sensing Image Change Detection

Remote sensing image change detection (CD) is an important means in remote sensing data analysis tasks, which can help us understand the surface changes in high-resolution (HR) remote sensing images. Traditional pixel-based and object-based methods are only suitable for low- and medium-resolution images, and are still challenging for complex texture features and detailed image detail processing in HR images. At present, the method based on deep learning has problems such as inconsistent fusion and difficult model training in the combination of the difference feature information of the deep and shallow layers and the attention mechanism, which leads to errors in the distinction between the changing region and the invariant region, edge detection and small target detection. In order to solve the above problems of inconsistent fusions of feature information aggregation and attention mechanisms, and indistinguishable change areas, we propose a multi-scale feature fusion Siamese network based on attention mechanism (ABMFNet). To tackle the issues of inconsistent fusion and alignment difficulties when integrating multi-scale fusion and attention mechanisms, we introduce the attention-based multi-scale feature fusion module (AMFFM). This module not only addresses insufficient feature fusion and connection between different-scale feature layers, but also enables the model to automatically learn and prioritize important features or regions in the image. Additionally, we design the cross-scale fusion module (CFM) and the difference feature enhancement pyramid structure (DEFPN) to assist the AMFFM module in integrating differential information effectively. These modules bridge the spatial disparity between low-level and high-level features, ensuring efficient connection and fusion of spatial difference information. Furthermore, we enhance the representation and inference speed of the feature pyramid by incorporating a feature enhancement module (FEM) into DEFPN. Finally, the BICD dataset proposed by the laboratory and public datasets LEVIR-CD and BCDD are compared and tested. We use F1 score and MIoU values as evaluation metrics. For AMBMFNet, the F1 scores on the three datasets are 77.69%, 81.57%, and 77.91%, respectively, while the MIoU values are 84.65%, 85.84%, and 84.54%, respectively. The experimental results show that ABMFNet has better effectiveness and robustness.

A DFRC Based on Multichannel and Spatial Information Fusion for Multiradar Communication

A DFRC Based on Multichannel and Spatial Information Fusion for Multiradar Communication

Moving Object Detection in Freely Moving Camera via Global Motion Compensation and Local Spatial Information Fusion.

Motion object detection (MOD) with freely moving cameras is a challenging task in computer vision. To extract moving objects, most studies have focused on the difference in motion features between foreground and background, which works well for dynamic scenes with relatively regular movements and variations. However, abrupt illumination changes and occlusions often occur in real-world scenes, and the camera may also pan, tilt, rotate, and jitter, etc., resulting in local irregular variations and global discontinuities in motion features. Such complex and changing scenes bring great difficulty in detecting moving objects. To solve this problem, this paper proposes a new MOD method that effectively leverages local and global visual information for foreground/background segmentation. Specifically, on the global side, to support a wider range of camera motion, the relative inter-frame transformations are optimized to absolute transformations referenced to intermediate frames in a global form after enriching the inter-frame matching pairs. The global transformation is fine-tuned using the spatial transformer network (STN). On the local side, to address the problem of dynamic background scenes, foreground object detection is optimized by utilizing the pixel differences between the current frame and the local background model, as well as the consistency of local spatial variations. Then, the spatial information is combined using optical flow segmentation methods, enhancing the precision of the object information. The experimental results show that our method achieves a detection accuracy improvement of over 1.5% compared with the state-of-the-art methods on the datasets of CDNET2014, FBMS-59, and CBD. It demonstrates significant effectiveness in challenging scenarios such as shadows, abrupt changes in illumination, camera jitter, occlusion, and moving backgrounds.

Design of urban road fault detection system based on artificial neural network and deep learning.

In urban traffic management, the timely detection of road faults plays a crucial role in improving traffic efficiency and safety. However, conventional methods often fail to fully leverage the information from road topology and traffic data. To address this issue, we propose an innovative detection system that combines Artificial Neural Networks (ANNs), specifically Graph Convolutional Networks (GCN), Bidirectional Gated Recurrent Units (BiGRU), and self-attention mechanisms. Our approach begins by representing the road topology as a graph and utilizing GCN to model it. This allows us to learn the relationships between roads and capture their structural dependencies. By doing so, we can effectively incorporate the spatial information provided by the road network. Next, we employ BiGRU to model the historical traffic data, enabling us to capture the temporal dynamics and patterns in the traffic flow. The BiGRU architecture allows for bidirectional processing, which aids in understanding the traffic conditions based on both past and future information. This temporal modeling enhances our system's ability to handle time-varying traffic patterns. To further enhance the feature representations, we leverage self-attention mechanisms. By combining the hidden states of the BiGRU with self-attention, we can assign importance weights to different temporal features, focusing on the most relevant information. This attention mechanism helps to extract salient features from the traffic data. Subsequently, we merge the features learned by GCN from the road topology and BiGRU from the traffic data. This fusion of spatial and temporal information provides a comprehensive representation of the road status. By employing a Multilayer Perceptron (MLP) as a classifier, we can effectively determine whether a road is experiencing a fault. The MLP model is trained using labeled road fault data through supervised learning, optimizing its performance for fault detection. Experimental evaluations of our system demonstrate excellent performance in road fault detection. Compared to traditional methods, our system achieves more accurate fault detection, thereby improving the efficiency of urban traffic management. This is of significant importance for city administrators, as they can promptly identify road faults and take appropriate measures for repair and traffic diversion.

A short note on deep contextual spatial and spectral information fusion for hyperspectral image processing: Case of pork belly properties prediction

AbstractThis study demonstrates a new approach to process hyperspectral images where both the contextual spatial information as well as the spectral information are used to predict sample properties. The deep contextual spatial information is extracted using the deep feature extraction from pretrained resnet‐18 deep learning architecture, while the spectral information was readily available as the average pixel values. To fuse the information in a complementary way, a multiblock modeling approach called sequential orthogonalized partial least squares was used. The sequential model guarantees that the information learned is complementary from spatial and spectral domains. The potential of the approach is demonstrated to predict several physical and chemical properties in pork bellies.

Tomato leaf disease detection based on attention mechanism and multi-scale feature fusion.

When detecting tomato leaf diseases in natural environments, factors such as changes in lighting, occlusion, and the small size of leaf lesions pose challenges to detection accuracy. Therefore, this study proposes a tomato leaf disease detection method based on attention mechanisms and multi-scale feature fusion. Firstly, the Convolutional Block Attention Module (CBAM) is introduced into the backbone feature extraction network to enhance the ability to extract lesion features and suppress the effects of environmental interference. Secondly, shallow feature maps are introduced into the re-parameterized generalized feature pyramid network (RepGFPN), constructing a new multi-scale re-parameterized generalized feature fusion module (BiRepGFPN) to enhance feature fusion expression and improve the localization ability for small lesion features. Finally, the BiRepGFPN replaces the Path Aggregation Feature Pyramid Network (PAFPN) in the YOLOv6 model to achieve effective fusion of deep semantic and shallow spatial information. Experimental results indicate that, when evaluated on the publicly available PlantDoc dataset, the model's mean average precision (mAP) showed improvements of 7.7%, 11.8%, 3.4%, 5.7%, 4.3%, and 2.6% compared to YOLOX, YOLOv5, YOLOv6, YOLOv6-s, YOLOv7, and YOLOv8, respectively. When evaluated on the tomato leaf disease dataset, the model demonstrated a precision of 92.9%, a recall rate of 95.2%, an F1 score of 94.0%, and a mean average precision (mAP) of 93.8%, showing improvements of 2.3%, 4.0%, 3.1%, and 2.7% respectively compared to the baseline model. These results indicate that the proposed detection method possesses significant detection performance and generalization capabilities.

Deep Fusion Module for Video Action Recognition

In video action recognition, effective spatiotemporal modeling is crucial. However, traditional two-stream methods face challenges in integrating spatial information from RGB images and temporary information from optical flow without long-range temporal modelling. To address these limitations, we propose the Deep Fusion Module (DFM), which focuses on the deep fusion of spatial and temporal information and consists of two components. First, we propose an Attention Fusion Module (AFM) to effectively fuse the shallow features obtained from a two-stream network, thereby facilitating the integration of spatial and temporal information. Next, we incorporate a SpatioTemporal Module (STM), comprising a ConvGRU and a 1×1 convolution, to model long-range temporal dependency and fuse spatial-temporal features. Experiments on the UCF101 dataset show that our method achieves 96.5% accuracy, outperforming baseline two-stream models by 0.3%.

HRD-Net: High resolution segmentation network with adaptive learning ability of retinal vessel features

Retinal segmentation is a crucial step in the early warning of human health conditions. However, retinal blood vessels possess complex curvature, irregular distribution, and contain multi-scale fine structures, which make the limited receptive field of regular convolution challenging to process their vascular details efficiently. Additionally, the encoder-decoder based network leads to irreversible spatial information loss because of multiple downsampling, resulting in over-segmentation and missed segmentation of the vessels. For this reason, we develop a high-resolution network based on Deformable Convolution v3, called HRD-Net. By constructing a high-resolution representation, the network allows special attention to be paid to the details of tiny blood vessels. The proposed feature enhancement cascade module based on Deformable Convolution v3 can flexibly adapt and capture the ever-changing morphology and intricate connections of retinal blood vessels, ensuring the continuity of vessel segmentation. In the output phase of the network, the proposed global aggregation module integrates full-resolution feature maps while suppressing redundant features, achieving an effective fusion of high-level semantic information and spatial detail information. In addition, we have re-examined the selection criteria for activation and normalization methods, and also refine the network architectures from a spatial domain perspective to release redundant computational loads. Testing on the DRIVE, STARE, and CHASE_DB1 datasets indicates that HRD-Net, with fewer parameters, outperforms existing segmentation methods on several evaluation metrics such as F1, ACC, SE, SP, AUC, and IOU.

LFSimCC: Spatial fusion lightweight network for human pose estimation

To address the limitations of existing 2D human pose estimation methods in terms of speed and lightweight, we propose a method called Lightweight Fusion SimCC (LFSimCC). LFSimCC incorporates two modules: LiteFNet, which enhances multi-scale spatial information fusion, and LKC-GAU, which improves the modeling capability of spatial information. Specifically, LiteFNet utilizes a combination of self-attention mechanism and novel spatial convolution to enable feature maps to capture richer multi-level global feature representations within the network. On the other hand, LKC-GAU enhances SimCC’s ability to capture spatial relationships between joints by incorporating a large kernel of convolution and a self-attention mechanism. Furthermore, we design a keypoint information fusion loss (IFL) that enhances the model’s sensitivity to information between keypoints in the human body. Experimental results demonstrate that our method is capable of extracting more decisive information and suppressing redundant feature representations, leading to high recognition accuracy and low inference latency.