Local Spatial Features Research Articles

HIGF-Net: Hierarchical information-guided fusion network for polyp segmentation based on transformer and convolution feature learning

Polyp segmentation plays a role in image analysis during colonoscopy screening, thus improving the diagnostic efficiency of early colorectal cancer. However, due to the variable shape and size characteristics of polyps, small difference between lesion area and background, and interference of image acquisition conditions, existing segmentation methods have the phenomenon of missing polyp and rough boundary division. To overcome the above challenges, we propose a multi-level fusion network called HIGF-Net, which uses hierarchical guidance strategy to aggregate rich information to produce reliable segmentation results. Specifically, our HIGF-Net excavates deep global semantic information and shallow local spatial features of images together with Transformer encoder and CNN encoder. Then, Double-stream structure is used to transmit polyp shape properties between feature layers at different depths. The module calibrates the position and shape of polyps in different sizes to improve the model's efficient use of the rich polyp features. In addition, Separate Refinement module refines the polyp profile in the uncertain region to highlight the difference between the polyp and the background. Finally, in order to adapt to diverse collection environments, Hierarchical Pyramid Fusion module merges the features of multiple layers with different representational capabilities. We evaluate the learning and generalization abilities of HIGF-Net on five datasets using six evaluation metrics, including Kvasir-SEG, CVC-ClinicDB, ETIS, CVC-300, and CVC-ColonDB. Experimental results show that the proposed model is effective in polyp feature mining and lesion identification, and its segmentation performance is better than ten excellent models.

Uncertainty-guided joint attention and contextual relation network for person re-identification

Due to the influence of factors such as camera angle and pose changes, some salient local features are often suppressed in person re-identification tasks. Moreover, many existing person re-identification methods do not consider the relation between features. To address these issues, this paper proposes two novel approaches: (1) To solve the problem of being confused and misidentified when local features of different individuals have similar attributes, we design a contextual relation network that focuses on establishing the relationship between local features and contextual features, so that all local features of the same person both contain contextual information. (2) To fully and correctly express key local features, we propose an uncertainty-guided joint attention module. The module focuses on the joint representation of individual pixels and local spatial features to enhance the credibility of local features. Finally, our method achieves competitive performance on four widely recognized datasets compared with state-of-the-art methods.

Space Target Material Identification Based on Graph Convolutional Neural Network

Under complex illumination conditions, the spectral data distributions of a given material appear inconsistent in the hyperspectral images of the space target, making it difficult to achieve accurate material identification using only spectral features and local spatial features. Aiming at this problem, a material identification method based on an improved graph convolutional neural network is proposed. Superpixel segmentation is conducted on the hyperspectral images to build the multiscale joint topological graph of the space target global structure. Based on this, topological graphs containing the global spatial features and spectral features of each pixel are generated, and the pixel neighborhoods containing the local spatial features and spectral features are collected to form material identification datasets that include both of these. Then, the graph convolutional neural network (GCN) and the three-dimensional convolutional neural network (3-D CNN) are combined into one model using strategies of addition, element-wise multiplication, or concatenation, and the model is trained by the datasets to fuse and learn the three features. For the simulated data and the measured data, the overall accuracy of the proposed method can be kept at 85–90%, and their kappa coefficients remain around 0.8. This proves that the proposed method can improve the material identification performance under complex illumination conditions with high accuracy and strong robustness.

Spectral-Spatial Attention Rotation-Invariant Classification Network for Airborne Hyperspectral Images

An airborne hyperspectral imaging system is typically equipped on an aircraft or unmanned aerial vehicle (UAV) to capture ground scenes from an overlooking perspective. Due to the rotation of the aircraft or UAV, the same region of land cover may be imaged from different viewing angles. While humans can accurately recognize the same objects from different viewing angles, classification methods based on spectral-spatial features for airborne hyperspectral images exhibit significant errors. The existing methods primarily involve incorporating image or feature rotation angles into the network to improve its accuracy in classifying rotated images. However, these methods introduce additional parameters that need to be manually determined, which may not be optimal for all applications. This paper presents a spectral-spatial attention rotation-invariant classification network for the airborne hyperspectral image to address this issue. The proposed method does not require the introduction of additional rotation angle parameters. There are three modules in the proposed framework: the band selection module, the local spatial feature enhancement module, and the lightweight feature enhancement module. The band selection module suppresses redundant spectral channels, while the local spatial feature enhancement module generates a multi-angle parallel feature encoding network to improve the discrimination of the center pixel. The multi-angle parallel feature encoding network also learns the position relationship between each pixel, thus maintaining rotation invariance. The lightweight feature enhancement module is the last layer of the framework, which enhances important features and suppresses insignificance features. At the same time, a dynamically weighted cross-entropy loss is utilized as the loss function. This loss function adjusts the model’s sensitivity for samples with different categories according to the output in the training epoch. The proposed method is evaluated on five airborne hyperspectral image datasets covering urban and agricultural regions. Compared with other state-of-the-art classification algorithms, the method achieves the best classification accuracy and is capable of effectively extracting rotation-invariant features for urban and rural areas.

Multi-scale receptive fields: Graph attention neural network for hyperspectral image classification

Hyperspectral image (HSI) classification has attracted wide attention in many fields. Applying Graph Neural Network (GNN) to HSI classification is one of the research frontiers, which has improved the HSI classification accuracy greatly. However, GNN-based methods have not been widely applied due to their time-consuming, inefficient information description as well as poor anti-noise robustness. To overcome the deficiencies, a novel multi-scale receptive fields graph attention neural network (MRGAT) is proposed for HSI classification in this paper. In this network, a superpixel segment method is adopted to abstract the original HSI local spatial features. A two-layer one-dimensional convolution neural network (1D CNN) spectral transformer mechanism, is designed to extract the spectral features of superpixels, with which the spectral features can be acquired automatically. Furthermore, graph edges are introduced into Graph Attention Network (GAT) to acquire the local semantic feature of the graph. Moreover, inspired by the transformer network, we design a novel multi-scale receptive field GAT to extract the local-global adjacent node-features and edges-features. Finally, a graph attention network and a softMax function are utilized for multi-receptive feature fusion and pixel-label predicting. On Pavia University, Salinas, and Houston 2013 datasets, the overall accuracies (OAs) of our MRGAT are 71.76%, 82.61%, and 63.82%, respectively. Moreover, the performances with limited labeled samples indicates that the MRGAT contains superior adaptability. Compared with the competitive classifiers, MRGAT achieves high classification efficiency verified by training time comparison experiment.

EEG emotion recognition using attention-based convolutional transformer neural network

EEG-based emotion recognition has become an important task in affective computing and intelligent interaction. However, how to effectively combine the spatial, spectral, and temporal distinguishable information of EEG signals to achieve better emotion recognition performance is still a challenge. In this paper, we propose a novel attention-based convolutional transformer neural network (ACTNN), which effectively integrates the crucial spatial, spectral, and temporal information of EEG signals, and cascades convolutional neural network and transformer in a new way for emotion recognition task. We first organized EEG signals into spatial–spectral–temporal representations. To enhance the distinguishability of features, spatial and spectral attention masks are learned for the representation of each time slice. Then, a convolutional module is used to extract local spatial and spectral features. Finally, we concatenate the features of all time slices, and feed them into the transformer-based temporal encoding layer to use multi-head self-attention for global feature awareness. The average recognition accuracy of the proposed ACTNN on two public datasets, namely SEED and SEED-IV, is 98.47% and 91.90% respectively, outperforming the state-of-the-art methods. Besides, to explore the underlying reasoning process of the model and its neuroscience relevance with emotion, we further visualize spatial and spectral attention masks. The attention weight distribution shows that the activities of prefrontal lobe and lateral temporal lobe of the brain, and the gamma band of EEG signals might be more related to human emotion. The proposed ACTNN can be employed as a promising framework for EEG emotion recognition.

FSAU-Net: a network for extracting buildings from remote sensing imagery using feature self-attention

ABSTRACT Convolutional neural networks (CNNs) extract semantic features from images by stacking convolutional operators, which easily causes semantic information loss and leads to hollow and edge inaccuracies in building extraction. Therefore, a features self-attention U-block network (FSAU-Net) is proposed. The network focuses on the target feature self-attention in the coding stage, and features self-attention (FSA) distinguishes buildings from nonbuilding by weighting the extracted features themselves; we introduce spatial attention (SA) in the decoder stage to focus on the spatial locations of features, and SA generates spatial location features through the spatial relationship among the features to highlight the building information area. A jump connection is used to fuse the shallow features generated in the decoder stage with the deep features generated in the encoder stage to reduce the building information loss. We validate the superiority of the method FSAU-Net on the WHU and Inria datasets with 0.3 m resolution and Massachusetts with 1.0 m resolution, experimentally showing IoU of 91.73%, 80.73% and 78.46% and precision of 93.60%, 90.71% and 86.37%, respectively. In addition, we also set up ablation experiments by adding an FSA module, Squeeze-and-Excitation (SE) module and Efficient Channel Attention (ECA) module to UNet and ResNet101, where UNet+FSA improves the IoU values by 3.15%, 2.72% and 1.77% compared to UNet, UNet+SE and UNet+ECA, respectively, and ResNet101+FSA improves the IoU values by 2.06%, 1.17% and 0.9% compared to ResNet101, ResNet101+SE and ResNet101+ECA, respectively, demonstrating the superiority of our proposed FSA module. FSAU-Net improves the IoU values by 3.18%, 2.75% and 1.80% compared to those of UNet, UNet+SE and UNet+ECA, respectively. FSAU-Net has 2.11%, 1.22%, and 0.95% IoU improvements over the IoU values of ResNet101, ResNet101+SE and ResNet101+ECA, respectively, demonstrating the superiority of our proposed FSAU-Net model. The TensorFlow implementation is available at https://github.com/HMH12456/FSAU-Net-master.git.

SOR-TC: Self-attentive octave ResNet with temporal consistency for compressed video action recognition

Modeling and recognizing video activities from videos are key parts of many promising applications such as visual surveillance, human–computer interaction, and video summarization. However, current approaches mainly suffer from two issues: (a) Short-term local and global spatial features are not well represented. The spatial redundancy and dependency have not been well considered on CNN-based action recognition, which may result in a further increase in both memory and computation cost. (b) Long-term temporal consistency is not well captured. The action consistency across multiple clips has been ignored in video-level action recognition approaches. To address these two issues, we propose a Self-Attentive Octave ResNet with Temporal Consistency (SOR-TC) for compressed video action recognition to better capture the short-term and long-term features in video and improve the efficiency and effectiveness of action recognition. In addition, this paper introduces a consistency hypothesis that adjacent clips should predict similar actions. So the consistency loss function is designed to learn the correlation of clips. Finally, extensive experimental results on two benchmark datasets HMDB-51 and UCF-101 verify the effectiveness of our proposed method.

TransFusionNet: Semantic and Spatial Features Fusion Framework for Liver Tumor and Vessel Segmentation Under JetsonTX2.

Liver cancer is one of the most common malignant diseases worldwide. Segmentation and reconstruction of liver tumors and vessels in CT images can provide convenience for physicians in preoperative planning and surgical intervention. In this paper, we introduced a TransFusionNet framework, which consists of a semantic feature extraction module, a local spatial feature extraction module, an edge feature extraction module, and a multi-scale feature fusion module to achieve fine-grained segmentation of liver tumors and vessels. In addition, we applied the transfer learning approach to pre-train using public datasets and then fine-tune the model to further improve the fitting effect. Furthermore, we proposed an intelligent quantization scheme to compress the model weights and achieved high performance inference on JetsonTX2. The TransFusionNet framework achieved mean IoU of 0.854 in vessel segmentation task, and achieved mean IoU of 0.927 in liver tumor segmentation task. When profiling the Computational Performance of the quantized inference, our quantized model achieved 4TFLOPs on Node with NVIDIA RTX3090 and 132GFLOPs on JetsonTX2. This unprecedented segmentation effect solves the accuracy and performance bottleneck of automated segmentation to a certain extent.

A Novel Method for Obstacle Detection in Front of Vehicles Based on the Local Spatial Features of Point Cloud

Obstacle detection is the primary task of the Advanced Driving Assistance System (ADAS). However, it is very difficult to achieve accurate obstacle detection in complex traffic scenes. To this end, this paper proposes an obstacle detection method based on the local spatial features of point clouds. Firstly, the local spatial point cloud of a superpixel is obtained through stereo matching and the SLIC image segmentation algorithm. Then, the probability of the obstacle in the corresponding area is estimated from the spatial feature information of the local plane normal vector and the superpixel point-cloud height, respectively. Finally, the detection results of the two methods are input into the Bayesian framework in the form of probabilities for the final decision. In order to describe the traffic scene efficiently and accurately, the detection results are further transformed into a multi-layer stixel representation. We carried out experiments on the KITTI dataset and compared several obstacle detection methods. The experimental results indicate that the proposed method has advantages in terms of its Pixel-wise True Positive Rate (PTPR) and Pixel-wise False Positive Rate (PFPR), particularly in complex traffic scenes, such as uneven roads.

A Novel Deep Learning Method for Automatic Recognition of Coseismic Landslides

Massive earthquakes generally trigger thousands of coseismic landslides. The automatic recognition of these numerous landslides has provided crucial support for post-earthquake emergency rescue, landslide risk mitigation, and city reconstruction. The automatic recognition of coseismic landslides has always been a difficult problem due to the relatively small size of a landslide and various complicated environmental backgrounds. This work proposes a novel semantic segmentation network, EGCN, to improve the landslide identification accuracy. EGCN conducts coseismic landslide recognition by a recognition index set as the input data, CGBlock as the basic module, and U-Net as the baseline. The CGBlock module can extract the relatively stable global context-dependent features (global context features) and the unstable local features by the GNN Branch and CNN Branch (GNN Branch contains the proposed EISGNN) and integrates them via adaptive weights. This method has four advantages. (1) The recognition indices are established according to the causal mechanism of coseismic landslides. The rationality of the indices guarantees the accuracy of landslide recognition. (2) The module of EISGNN is suggested based on the entropy importance coefficient and GATv2. Owing to the feature aggregation among nodes with high entropy importance, global and useful context dependency can be synthesized and the false alarm of landslide recognition can be reduced. (3) CGBlock automatically integrates context features and local spatial features, and has strong adaptability for the recognition of coseismic landslides located in different environments. (4) Owing to CGBlock being the basic module and U-Net being the baseline, EGCN can integrate the context features and local spatial characteristics at both high and low levels. Thus, the accuracy of landslide recognition can be improved. The meizoseismal region of the Ms 7.0 Jiuzhaigou earthquake is selected as an example to conduct coseismic landslide recognition. The values of the precision indices of Overall Accuracy, mIoU, Kappa, F1-score, Precision, and Recall reached 0.99854, 0.99709, 0.97321, 0.97396, 0.97344, and 0.97422, respectively. The proposed method outperforms the current major deep learning methods.

Uncertainty-guided hierarchical frequency domain Transformer for image restoration

Existing convolutional neural network (CNN)-based and vision Transformer (ViT)-based image restoration methods are usually explored in the spatial domain. However, we employ Fourier analysis to show that these spatial domain models cannot perceive the entire frequency spectrum of images, i.e., mainly focus on either high-frequency (CNN-based models) or low-frequency components (ViT-based models). This intrinsic limitation results in the partial missing of semantic information and the appearance of artifacts. To address this limitation, we propose a novel uncertainty-guided hierarchical frequency domain Transformer named HFDT to effectively learn both high and low-frequency information while perceiving local and global features. Specifically, to aggregate semantic information from various frequency levels, we propose a dual-domain feature interaction mechanism, in which the global frequency information and local spatial features are extracted by corresponding branches. The frequency domain branch adopts the Fast Fourier Transform (FFT) to convert the features from the spatial domain to the frequency domain, where the global low and high-frequency components are learned with Log-linear complexity. Complementarily, an efficient convolution group is employed in the spatial domain branch to capture local high-frequency details. Moreover, we introduce an uncertainty degradation-guided strategy to efficiently represent degraded prior information, rather than simply distinguishing degraded/non-degraded regions in binary form. Our approach achieves competitive results in several degraded scenarios, including rain streaks, raindrops, motion blur, and defocus blur.

Hybrid Multi-Strategy Aquila Optimization with Deep Learning Driven Crop Type Classification on Hyperspectral Images

Hyperspectral imaging instruments could capture detailed spatial information and rich spectral signs of observed scenes. Much spatial information and spectral signatures of hyperspectral images (HSIs) present greater potential for detecting and classifying fine crops. The accurate classification of crop kinds utilizing hyperspectral remote sensing imaging (RSI) has become an indispensable application in the agricultural domain. It is significant for the prediction and growth monitoring of crop yields. Amongst the deep learning (DL) techniques, Convolution Neural Network (CNN) was the best method for classifying HSI for their incredible local contextual modeling ability, enabling spectral and spatial feature extraction. This article designs a Hybrid Multi-Strategy Aquila Optimization with a Deep Learning-Driven Crop Type Classification (HMAODL-CTC) algorithm on HSI. The proposed HMAODL-CTC model mainly intends to categorize different types of crops on HSI. To accomplish this, the presented HMAODL-CTC model initially carries out image preprocessing to improve image quality. In addition, the presented HMAODL-CTC model develops dilated convolutional neural network (CNN) for feature extraction. For hyperparameter tuning of the dilated CNN model, the HMAO algorithm is utilized. Eventually, the presented HMAODL-CTC model uses an extreme learning machine (ELM) model for crop type classification. A comprehensive set of simulations were performed to illustrate the enhanced performance of the presented HMAODL-CTC algorithm. Extensive comparison studies reported the improved performance of the presented HMAODL-CTC algorithm over other compared methods.

Global–Local 3-D Convolutional Transformer Network for Hyperspectral Image Classification

Benefiting from powerful feature extraction capabilities, convolutional neural networks (CNNs) have gained prominence in hyperspectral image (HSI) classification. Nevertheless, with restricted receptive fields of convolution kernels, CNN-based methods fail to learn complex characteristics of long-range sequences. Meanwhile, vision transformer allows us to learn long-range dependencies in a global view, but local region features are ignored. To overcome these limitations, we propose a novel method entitled global-local three-dimensional convolutional transformer network (GTCT), where 3-D convolution is embedded in a dual-branch transformer to simultaneously capture global-local associations in both spectral and spatial domains. In particular, the global-local spectral convolutional transformer (GECT) is designed to exploit global spectral sequence signatures and local spectral relationships between bands. Symmetrically, the global-local spatial convolutional transformer (GACT) is devised to exploit local spatial context features and global interactions among different pixels. In addition, multiscale global-local spectral-spatial information is adaptively fused with trainable weights by the weighted multiscale spectral-spatial feature interaction (WMSFI) module. It is worth noting that a spectral-spatial global attention mechanism (SSGAM) is incorporated into multi-head convolutional attention to further integrate discriminative spectral-spatial information. Extensive experiments on four HSI datasets, including GF-5 and ZY1-02D satellite hyperspectral images, demonstrate the superiority of the proposed GTCT method over other state-of-the-art algorithms with fewer parameters and lower floating-point operations (FLOPs) in practical applications.

A ConvLSTM Neural Network Model for Spatiotemporal Prediction of Mining Area Surface Deformation Based on SBAS-InSAR Monitoring Data

The surface deformation caused by underground mining leads to damage to surface buildings and brings potential safety hazards and property losses. The demand for reliable prediction methods of surface deformation in mining areas is becoming increasingly significant. At present, most prediction methods are based on sampling points; however, these methods neglect to consider local and overall spatial features, and this oversight affects the spatial accuracy of prediction results. The data form of the prediction output is often discontinuous and not intuitive. In order to solve this problem, the spatiotemporal prediction method of surface deformation in mining areas is a very effective proposal. However, few scholars have proposed a solution based on this idea. In this study, a convolution long short-term memory (ConvLSTM) neural network for surface deformation spatiotemporal prediction based on time-series interferometric synthetic aperture radar (InSAR) is proposed to directly predict the overall spatial deformation of the surface in the mining area. First, based on Sentinel-1A images of the Jinchuan Mining Area, Jinchang, Gansu province, China, the time-series InSAR surface deformation data of the study area from January 2018 to October 2020 (81 scenes) are obtained using small baseline subset InSAR (SBAS-InSAR) technology. Because of the large value scale of surface deformation in the mining area, we propose a method to fit the maximum and minimum values of time-series deformation, respectively, and carry out piecewise numerical compression. Then, based on the ConvLSTM neural network layer, construct the spatiotemporal prediction model of time-series InSAR surface deformation. Support vector regression (SVR), multilayer perceptron (MLP) regression, and the gray model [GM (1,1)] are used as benchmark methods. The prediction results of our models are compared with the three benchmark methods. The comparison results show that the prediction effect of the ConvLSTM model with optimal input time steps is significantly better than the benchmark methods in the comprehensive performance of various evaluation metrics, especially for the metrics used to evaluate the error. This shows that the ConvLSTM model has relatively fine spatiotemporal prediction performance for time-series InSAR surface deformation. Based on the InSAR time-series deformation monitoring results of the Jinchuan Mining Area, we carry out the spatiotemporal prediction of surface deformation in the subsequent 50 time steps (600 days). The results agree with the natural deformation development law in band collection statistics and 3-D representation of deformation; moreover, the reliability of numerical spatial distribution is relatively high. This result can be used to intuitively evaluate the overall surface deformation of the mining area in the monitoring range, find potential hazards in time, and take measures to address these hazards quickly. At the same time, this research process also provides a new concept design for such problems.

Spectral Reconstruction From Satellite Multispectral Imagery Using Convolution and Transformer Joint Network

Spectral reconstruction based on satellite multispectral (MS) images can produce high spatial resolution hyperspectral (HS) images at a reasonable cost, significantly expanding the application of satellite-based HS remote sensing. As a challenging ill-posed problem, existing methods have difficulty making full use of local and global information of space and spectra to guide the reconstruction, resulting in limited accuracy in large-scale scenes with complex ground features and severe spectral mixing. In this article, we propose a novel convolution and Transformer joint network (CTJN) to address the challenge of high-accuracy spectral reconstruction in complex scenes. The CTJN is cascaded with shallow feature extraction modules (SFEMs) and deep feature extraction modules (DFEMs), which can explore local spatial features and global spectral features. Besides, a high-frequency Transformer block (HF-TB) is designed to highlight the detailed features of the images to prevent significant high-frequency information loss, which could improve the reconstruction results in regions with drastic feature changes. Moreover, a spatial–spectral recalibration block (SSRB) is proposed to perform explicit constraints on the reconstructed points by exploiting the correlation among neighboring pixels and adjacent spectra. Extensive experimental results on four HS–MS datasets and one MS dataset demonstrate that the proposed CTJN outperforms the state-of-the-art methods in large-scale and small-scale scenes.

Spatiotemporal Co-Attention Hybrid Neural Network for Pedestrian Localization Based on 6D IMU

In this paper, we propose spatiotemporal co-attention hybrid neural network (SC-HNN), a novel hybrid neural network model with both spatial and temporal attention mechanisms for pose-invariant inertial odometry. The main idea is to extract both local and global features from a window of IMU measurements for velocity prediction. SC-HNN leverages the convolutional neural network (CNN) to capture the sectional features and long short-term memory (LSTM) recurrent neural network (RNN) to extract the long-range dependencies. Attention mechanisms are designed and embedded in both CNN and LSTM modules for better model representation. Specifically, in the CNN attention block, the convolved features are refined along both channel and element dimensions. For the LSTM module, softmax scoring is applied to update the weights of the hidden states along the temporal axis. We evaluate SC-HNN on the benchmark with the largest and most natural IMU data, RoNIN. Extensive ablation experiments demonstrate the effectiveness of our SC-HNN model. Compared with the state of the art, the 50th percentile accuracy of SC-HNN is 18.21% higher and the 90th percentile accuracy is 21.15% higher for all the phone holders not appeared in the training set. The real scenario inertial tracking trials in the CUHK campus further prove the superior generalization ability of the SC-HNN model. Note to Practitioners—This paper aims at improving the localization accuracy of deep inertial odometry. We focus on the problem of indoor localization only from the low-cost IMU embedded in the smartphone without any restriction on the phone’s daily use. IMU is a perfect solution for indoor localization because of its low power consumption, high privacy protection, and external infrastructure free. This paper suggests a novel hybrid convolutional and recurrent neural network with a set of carefully designed attention mechanisms to improve the representation ability of deep inertial odometry model. Specifically, the convolutional layer is applied to extract the local spatial features among the 6D IMU signals, following a cascaded channel attention module and element attention module to boost the representation ability of CNN. The complex long-term dependencies are then identified by the LSTM layers. To adaptively capture the temporal features of the multimodal inertial signals, an attention mechanism is applied to weigh the hidden states for the generation of the final features. The effectiveness of the SC-HNN design is validated by extensive ablation studies. To the best of our knowledge, our model is the first HNN fused attention mechanism for inertial tracking. Extensive experiments show that the proposed method outperforms the state of the art.

DJSPNet: Deep Joint Statistical–Spatial Pooling Network for High-Resolution SAR Image Classification

The previous approaches based on statistical features or spatial features have achieved promising performance on pixel-wise high-resolution (HR) synthetic aperture radar (SAR) image classification, but these methods always can not capture local spatial features and global statistical properties efficiently because of the complex spatial structural patterns and statistical nature in SAR patches. Inspired by this, we propose a deep joint statistical-spatial pooling network, named DJSPNet, for HR SAR image classification, which combines a group second-order statistical feature learning (GSFL) block and an efficient feature-fusion style (EFS) into an end-to-end feature learning block. GSFL block is designed with a group second-order feature learning method in two steps, where the first step divides convolutional channels into several semantic groups; The second step collects second-order feature statistics through calculating pairwise feature interactions within each group. EFS models second-order attentional statistics between statistical characteristics and spatial features by polynomial kernel approximation and guides the discriminative feature activations in SAR patches. More specifically, both GSFL and EFS are stacked and plugged into the encoder stage of conventional U-Net for distinguishable feature learning. Experimental results suggest that the proposed DJSPNet gives better classification performance compared with related deep feature learning networks on a real TerraSAR-X dataset.

A Pose Estimation Algorithm for Multimodal Data Fusion

In response to the problem that the previous pose detection systems are not effective under conditions such as severe occlusion or uneven illumination, this paper focuses on the multimodal information fusion pose estimation problem. The main work is to design a multimodal data fusion pose estimation algorithm for the problem of pose estimation in complex scenes such as low-texture targets and poor lighting conditions. The network takes images and point clouds as input and extracts local color and spatial features of the target object using the improved DenseNet and PointNet++ networks, which are combined with a microscopic bit-pose iterative network to achieve end-to-end bit-pose estimation. Excellent detection accuracy was obtained on two benchmark datasets of LineMOD (97.8%) and YCB-Video (95.3%) for pose estimation. The algorithm is able to obtain accurate poses of target objects from complex scenes, providing accurate, real-time and robust relative poses for object tracking in motion and wave compensation.

Attention-Based Multiscale Spatiotemporal Network for Traffic Forecast with Fusion of External Factors

Periodic traffic prediction and analysis is essential for urbanisation and intelligent transportation systems (ITS). However, traffic prediction is challenging due to the nonlinear flow of traffic and its interdependencies on spatiotemporal features. Traffic flow has a long-term dependence on temporal features and a short-term dependence on local and global spatial features. It is strongly influenced by external factors such as weather and points of interest. Existing models consider long-term and short-term predictions in Euclidean space. In this paper, we design an attention-based encoder–decoder with stacked layers of LSTM to analyse multiscale spatiotemporal dependencies in non-Euclidean space to forecast traffic. The attention weights are obtained adaptively and external factors are fused with the output of the decoder to evaluate region-wide traffic predictions. Extensive experiments are conducted to evaluate the performance of the proposed attention-based non-Euclidean spatiotemporal network (ANST) on real-world datasets. The proposed model has improved prediction accuracy over previous methods. The insights obtained from traffic prediction would be beneficial for daily commutation and logistics.