Hierarchical Feature Extraction Research Articles

An Onboard Point Cloud Semantic Segmentation System for Robotic Platforms

Point clouds represent an important way for robots to perceive their environments, and can be acquired by mobile robots with LiDAR sensors or underwater robots with sonar sensors. Hence, real-time semantic segmentation of point clouds with onboard edge devices is essential for robots to apprehend their surroundings. In this paper, we propose an onboard point cloud semantic segmentation system for robotic platforms to overcome the conflict between attaining high accuracy of segmentation results and the limited available computational resources of onboard devices. Our system takes raw a sequence of point clouds as inputs, and outputs semantic segmentation results for each frame as well as a reconstructed semantic map of the environment. At the core of our system is the transformer-based hierarchical feature extraction module and fusion module. The two modules are implemented with sparse tensor technologies to speed up inference. The predictions are accumulated according to Bayes rules to generate a global semantic map. Experimental results on the SemanticKITTI dataset show that our system achieves +2.2% mIoU and 18× speed improvements compared with SOTA methods. Our system is able to process 2.2 M points per second on Jetson AGX Xavier (NVIDIA, Santa Clara, USA), demonstrating its applicability to various robotic platforms.

STFRDN: a residual dense network for remote sensing image spatiotemporal fusion

ABSTRACT Convolutional Neural Networks (CNN) are useful models for spatiotemporal fusion, especially under strong temporal changes. Convolutional layers with receptive fields of various sizes produce hierarchical features that can improve prediction performance. In this study, we aimed to develop a CNN model for spatiotemporal fusion, which can extract hierarchical features and fully use them. The proposed network is composed of residual dense blocks with local dense connections that effectively utilize the hierarchical features. We conducted the experiments in the newly proposed Kansas dataset, which includes Sentinel-2 and Sentinel-3 image pairs with large resolution differences and strong temporal changes. Based on both quantitative and qualitative evaluation, experiments revealed that the STFRDN model outperformed the Flexible Spatiotemporal DAta Fusion (FSDAF) 2.0, Reliable and Adaptive Spatiotemporal Data Fusion (RASDF), Fit-FC, and DMNet methods in all of the test groups. The second major finding was that the STFRDN model was able to produce sufficiently successful predictions against cases with such large resolution differences. To get more accurate prediction results when there is a significant resolution difference, it is important to extract the right features that can model the discrepancy. Loss of spatial detail can be avoided by making full use of hierarchical features without any manipulation after extraction. The results of this study provide insights into the significance of extracting and fully utilizing the hierarchical features.

Spatiotemporal bias of the human gaze toward hierarchical visual features during natural scene viewing

The human gaze is directed at various locations from moment to moment in acquiring information necessary to recognize the external environment at the fine resolution of foveal vision. Previous studies showed that the human gaze is attracted to particular locations in the visual field at a particular time, but it remains unclear what visual features produce such spatiotemporal bias. In this study, we used a deep convolutional neural network model to extract hierarchical visual features from natural scene images and evaluated how much the human gaze is attracted to the visual features in space and time. Eye movement measurement and visual feature analysis using the deep convolutional neural network model showed that the gaze was more strongly attracted to spatial locations containing higher-order visual features than to locations containing lower-order visual features or to locations predicted by conventional saliency. Analysis of the time course of gaze attraction revealed that the bias to higher-order visual features was prominent within a short period after the beginning of observation of the natural scene images. These results demonstrate that higher-order visual features are a strong gaze attractor in both space and time, suggesting that the human visual system uses foveal vision resources to extract information from higher-order visual features with higher spatiotemporal priority.

A hierarchical birdsong feature extraction architecture combining static and dynamic modeling

To conserve bird biodiversity and monitor the distribution of species in the region, it is of tremendous necessity to identify birds by their songs and explore the rich ecological information birdsong contains. The audios recorded in the monitoring area generally have complex background noise, the characteristics of the song are not prominent and the biological spectrum information is not comprehensive, which brings some challenges to the identification of birds. This study proposes a hierarchical birdsong feature extraction architecture combining dynamic and static modeling to cope with complex environments as a modeling context. Firstly, six common speech features were extracted for the characteristics of birdsong. The Pearson correlation coefficient is then used to analyze the correlations between birdsong and human speech, examining the correlations between each feature in the presence and absence of environmental noise interference. Combined with the scatter plot matrix analysis, we conclude that Mel Frequency Cepstral Coefficient (MFCC) is more suitable comparing with other features when dealing with birdsong and can superiorly cope with a complex background noise. Secondly, a feature extraction architecture is built, which integrates static and dynamic modeling to fully explore the contextual relationship, to solve the problem of ignoring the internal structure information of the patch and losing some spatial information in the Transformer-type model. Finally, a hierarchical refinement module is designed to help extract more detailed features, as well as to optimize the computational cost of the Transformer-type model that requires many training data and has high complexity. The performance of the model can be detected with 93.67 % accuracy on the self-built birdsong dataset, 95.19 % accuracy on the public birdsong dataset Birdsdata and 97.02 % accuracy on the public environmental dataset UrbanSound8k.

Automatic Defect Recognition and Localization for Aeroengine Turbine Blades Based on Deep Learning

Radiographic testing is generally used in the quality management of aeroengine turbine blades. Traditional radiographic testing is critically dependent on artificially detecting professional inspectors. Thus, it sometimes tends to be error-prone and time-consuming. In this study, we gave an automatic defect detection method by combining radiographic testing with computer vision. A defect detection algorithm named DBFF-YOLOv4 was introduced for X-ray images of aeroengine turbine blades by employing two backbones to extract hierarchical defect features. In addition, a new concatenation form containing all feature maps was developed which play an important role in the present defect detection framework. Finally, a defect detection and recognition system was established for testing and output of complete turbine blade X-ray images. Meanwhile, nine cropping cycles for one defect, flipping, brightness increasing and decreasing were applied for expansion of training samples and data augmentation. The results found that this defect detection system can obtain a recall rate of 91.87%, a precision rate of 96.7%, and a false detection rate of 7% within the score threshold of 0.5. It was proven that cropping nine times and data augmentation are extremely helpful in improving detection accuracy. This study provides a new way of automatic radiographic testing for turbine blades.

Scattered Train Bolt Point Cloud Segmentation Based on Hierarchical Multi-Scale Feature Learning.

In view of the difficulty of using raw 3D point clouds for component detection in the railway field, this paper designs a point cloud segmentation model based on deep learning together with a point cloud preprocessing mechanism. First, a special preprocessing algorithm is designed to resolve the problems of noise points, acquisition errors, and large data volume in the actual point cloud model of the bolt. The algorithm uses the point cloud adaptive weighted guided filtering for noise smoothing according to the noise characteristics. Then retaining the key points of the point cloud, this algorithm uses the octree to partition the point cloud and carries out iterative farthest point sampling in each partition for obtaining the standard point cloud model. The standard point cloud model is then subjected to hierarchical multi-scale feature extraction to obtain global features, which are combined with local features through a self-attention mechanism, while linear interpolation is used to further expand the perceptual field of local features of the model as a basis for segmentation, and finally the segmentation is completed. Experiments show that the proposed algorithm could deal with the scattered bolt point cloud well, realize the segmentation of train bolt and background, and could achieve high segmentation accuracy, which has important practical significance for train safety detection.

HCFPN: Hierarchical Contextual Feature-Preserved Network for Remote Sensing Scene Classification

Convolutional neural networks (CNNs) have made significant advances in remote sensing scene classification (RSSC) in recent years. Nevertheless, the limitations of the receptive field cause CNNs to suffer from a disadvantage in capturing contextual information. To address this issue, vision transformer (ViT), a novel model that has piqued the interest of academics, is used to extract latent contextual information in remote sensing scene classification. However, when confronted with the challenges of large-scale variations and high interclass similarity in scene classification images, the original ViT has the drawback of ignoring important local features, thereby causing the model’s performance to degrade. Consequently, we propose the hierarchical contextual feature-preserved network (HCFPN) by combining the advantages of CNNs and ViT. First, a hierarchical feature extraction module based on ResNet-34 is utilized to acquire the multilevel convolutional features and high-level semantic features. Second, a contextual feature-preserved module takes advantage of the first two multilevel features to capture abundant long-term contextual features. Then, the captured long-term contextual features are utilized for multiheaded cross-level attention computing to aggregate and explore the correlation of multilevel features. Finally, the multiheaded cross-level attention score and high-level semantic features are classified. Then, a category score average module is proposed to fuse the classification results, whereas a label smoothing approach is utilized prior to calculating the loss to produce discriminative scene representation. In addition, we conduct extensive experiments on two publicly available RSSC datasets. Our proposed HCPFN outperforms most state-of-the-art approaches.

A Novel Lightweight Attention-Discarding Transformer for High-Resolution SAR Image Classification

Vision Transformer (ViT) has been introduced in high-resolution synthetic aperture radar (HR SAR) image classification due to its excellent global feature extraction ability. However, small samples of SAR images make it difficult to fit the ViT with excessive trainable parameters, which easily results in over-fitting in training. Meanwhile, poor capability in capturing local features of ViT limits its accuracy in SAR image classification. To solve these problems, this letter proposes a new Lightweight Attention-Discarding Transformer (LAD Transformer) for the classification of HR SAR images. In the proposed model, the backbone of the advanced Swin transformer is used to model global information and extract hierarchical features. Moreover, the vital feature extraction part of the LAD Transformer completely discards the self-attention mechanism and extracts local features of SAR images by introducing lighter group convolution and channel shuffle (GC-CS Block). In addition, to address the estimation shift caused by consecutive batch normalization (BN) layers, a new composite normalization method consisting of Batch normalization and Layer Normalization (BLN) in GC-CS Block is proposed. The experiments show that the proposed network has fewer parameters and higher classification accuracy on two real HR SAR data.

Pose-Guided Hierarchical Graph Reasoning for 3-D Hand Pose Estimation From a Single Depth Image.

Estimating 3-D hand pose estimation from a single depth image is important for human-computer interaction. Although depth-based 3-D hand pose estimation has made great progress in recent years, it is still difficult to deal with some complex scenes, especially the issues of serious self-occlusion and high self-similarity of fingers. Inspired by the fact that multipart context is critical to alleviate ambiguity, and constraint relations contained in the hand structure are important for the robust estimation, we attempt to explicitly model the correlations between different hand parts. In this article, we propose a pose-guided hierarchical graph convolution (PHG) module, which is embedded into the pixelwise regression framework to enhance the convolutional feature maps by exploring the complex dependencies between different hand parts. Specifically, the PHG module first extracts hierarchical fine-grained node features under the guidance of hand pose and then uses graph convolution to perform hierarchical message passing between nodes according to the hand structure. Finally, the enhanced node features are used to generate dynamic convolution kernels to generate hierarchical structure-aware feature maps. Our method achieves state-of-the-art performance or comparable performance with the state-of-the-art methods on five 3-D hand pose datasets: 1) HANDS 2019; 2) HANDS 2017; 3) NYU; 4) ICVL; and 5) MSRA.

Towards Diversified IoT Image Recognition Services in Mobile Edge Computing

This paper proposes an IoT services framework for different needs in MEC environment, which consists of Hierarchical Discriminative Feature Extraction (HDFE) and Sub-extractor Deployment (Sub-ED) algorithms. We first propose HDFE, which can generate effective extractors by separately generating extractors to avoid mutual interference between data. This is followed by the Sub-ED, which divides the extractor into sub-extractors and deploys them on appropriate MEC platforms. By doing so, the IoT device can connect to the corresponding MEC platform according to its service types, and use the sub-extract on the corresponding MEC platform to extract DFs. Then, the MEC platform uploads the extracted feature data to the cloud server for further processing. Finally, the cloud server sends the processed result back to the IoT device. Experimental results show that the proposed framework improves recognition accuracy by about 6.44% and reduces network traffic by up to 94.97% compared with the state-of-the-art approaches.

Modulation classification analysis of CNN model for wireless communication systems

<abstract><p>Modulation classification (MC) is a critical task in wireless communication systems, enabling the identification of the modulation class in the received signals. In this paper, we analyzed a novel multi-layer convolutional neural network (CNN) to extract hierarchical features directly from the raw baseband samples. Moreover, we compared the training and testing accuracy of the CNN model for various decimation rates, input sample size and the number of convolutional layers. The results showed that the three-layer CNN model provided better classification accuracy with less computation cost. Furthermore, we observed that the MC performance of the proposed CNN model was better than the other deep learning (DL) and cumulant-based models.</p></abstract>

Safeguarding Digital Essence: A Sub-band DCT Neural Watermarking Paradigm Leveraging GRNN and CNN for Unyielding Image Protection and Identification

Image watermarking preserves digital content. This study introduces a new watermarking approach employing Sub-Band Discrete Cosine Transform and Deep neural networks, GRNN and CNN. The method embeds robust, invisible watermarks in greyscale photos and compares the two neural network topologies. The watermark is added using sub-band DCT. Watermark embedding in low-frequency sub-bands resists photo processing. The binary watermark modifies sub-band DCT coefficients to determine embedding intensity, resisting signal deterioration, and assaults. GRNN and CNN neural networks extract watermarks accurately. CNN extracts hierarchical features from images, enabling robust watermark recovery even under distortions, whereas non-parametric GRNN stores the whole training data to create predictions. The watermarking approach is tested on several greyscale photos. PSNR, SSIM, MSE, and NCC measure performance. The watermark tests noise addition, compression, and filtering. Compare GRNN and CNN's watermark extraction strengths and shortcomings to assess image watermarking suitability.

DCM: A Distributed Collaborative Training Method for the Remote Sensing Image Classification

As the number of aero and space remote sensing platforms increases, distributed observation and real-time terminal processing become mainstream in the future. However, most of the training methods for the multi-platform are still limited to centralized structures or independent training based on a single platform, which is inefficient or limited in accuracy. In order to solve this problem, we innovatively propose a distributed collaborative method (DCM) for remote sensing image classification training in this article. First, the proposed training method, which is based on one cloud and several terminals, can aggregate different parameters of the terminal network to the cloud to improve global accuracy. Second, a sample proximity network is designed to process the problem of data heterogeneity on different terminal networks, which further improves the accuracy during the model fusion on the cloud. Third, a multi-layer grouped concatenation module is applied after the model fusion to extract hierarchical features with different categories of remote sensing images. Experimental results on the challenging remote sensing image classification dataset FAIR1M show that the proposed training method has better collaborative learning ability than the centralized-based model or terminal-trained lightweight network under the heterogeneous data.

Exploiting Spatial–Temporal Dynamics for Satellite Image Sequence Prediction

Satellite image sequence prediction is a challenging and significant task. Existing deep learning methods for the task make predictions mainly based on low-level pixel-wise features, which fail to model the sophisticated spatial-temporal features of satellite image sequences and deliver unsatisfactory performance. In this paper, we present a Hierarchical Spatial-Temporal network (HSTnet) for satellite image sequence prediction. With a carefully designed hierarchical feature extraction mechanism, HSTnet can learn effective spatial-temporal features from both pixel level and patch level. In addition, to better capture patch-level spatial-temporal dynamics, a dual-branch Transformer is proposed to model patch-level spatial and temporal features, respectively. Comprehensive experiments on the FY-4A satellite dataset demonstrate the superiority and effectiveness of our proposed method HSTnet over state-of-the-art approaches.

Remaining Useful Life Estimation of Bearing Using Deep Multiscale Window-Based Transformer

Accurate and reliable remaining useful life (RUL) estimation for bearings relies on the extraction of desired degradation features from vibration signals. Now, various feature extractors based on convolutional neural networks have been proposed for RUL prediction and achieved great success. However, these approaches focus on extracting degradation features from data in a single time frame but fail to capture the degradation information across multiple time frames, leading to degenerate estimation performance. In this study, we propose a novel multi-scale feature extractor called deep multi-scale window-based transformer (DMW-Trans) to overcome these limitations. To be more specific, a bunch of feature maps (FMs) are firstly generated with time-frequency maps (TFMs) at multiple time frames. Then, sequential layers equipped with window-based transformer blocks, i.e., WT-Blocks, and patch-merge modules are designed, where the former is used to extract features with multi-head self-attention windows while the latter is employed to fuse features from neighboring areas on TFMs to enlarge the attention fields from layer to layer. Through this hierarchical feature extraction process, multi-scale degradation related features can be effectively extracted. Lastly, multi-layer perceptron is applied to fuse the multi-scale features to estimate RUL. Due to the excellent multi-scale feature extraction capability, the proposed DMW-Trans can provide accurate and reliable RUL prediction results. Extensive experiments on two public run-to-failure bearing datasets demonstrate the superior performance of our proposed method compared to some state-of-the-art approaches.

Accurate determination of homogeneous ignition of single solid fuel particles enabled by machine learning

This work presents an experimental investigation of solid fuel combustion in a laminar flow reactor with high-speed laser diagnostics. The main focus lays on the development of image evaluation approaches for accurate determination of single-particle ignition based on optical measurement data. Homogeneous ignition of individual bituminous coal particles is visualized by simultaneous planar laser-induced fluorescence of OH radicals (OH-LIF) and diffuse backlight-illumination (DBI) techniques at 10kHz. Two coal particle sizes of 90–125μm and 160–200μm are investigated in conventional air and oxy-fuel conditions with increasing oxygen concentrations. A comprehensive experimental data set is established, containing in total 1518 single-particle events and high-quality ground truth labels of ignition delay times. The previously proposed structure and signal (SAS) analysis is evaluated. Errors and uncertainties are evaluated in a sensitivity analysis with variation in the threshold selection, implying inherent restrictions of conventional image processing methods. An improved blob detection approach is applied, enabling feature detection at various scales and higher accuracy in predicting ignition delay times. Moreover, two deep learning architectures, namely residual network (ResNet) and feature pyramid network (FPN) are implemented. Both networks are capable of detecting ignition with significantly higher accuracy and precision than other algorithms in this study. Influences of training data and depth of networks on the prediction performance of a trained model are examined. The current study shows that the hierarchical feature extraction of the convolutions networks clearly facilitates data evaluation for high-speed optical measurements and could be transferred to other solid fuel experiments with similar boundary conditions.

An end‐to‐end steel surface defect detection approach via Swin transformer

AbstractDifferent from most current studies using convolutional neural network (CNN), a deep learning detection method for steel plate surface defects based on Transformer is researched. This paper presents a process, network structure and detection method for steel strip surface defect detection. A Swin Transformer was used to extract hierarchical features in the detection system. Then feature pyramid networks (FPN) was used to fuse the above features to form multi‐scale feature maps, and region proposal network (RPN) was adopted to generate the generate region of interest (ROI) of defects. Finally, the ROI head was used to generate defect category information and its precise location. Here, ablation experiments were conducted to explore the impact of different backbone networks and the number of stages of Swin Transformer and FPN on target detection performance to prove the rationality and efficiency of the network. On the NEU‐DET dataset, the proposed algorithm achieved 81.1% mAP, which is higher than that of classic CNN detection methods such as Faster R‐CNN, SSD, Yolo v3 and RepPoints.

Gated Multi-Attention Feedback Network for Medical Image Super-Resolution

Medical imaging technology plays a crucial role in the diagnosis and treatment of diseases. However, the captured medical images are often in a low resolution (LR) due to the limited imaging condition. Super-resolution (SR) technology is a feasible solution to enhance the resolution of a medical image without increasing the hardware cost. However, the existing SR methods often ignore high-frequency details, which results in blurred edges and an unsatisfying visual perception. In this paper, a gated multi-attention feedback network (GAMA) is proposed for medical image SR. Specifically, a gated multi-feedback network is employed as the backbone to extract hierarchical features. Meanwhile, a layer attention feature extraction (LAFE) module is introduced to refine the feature map. In addition, a channel-space attention reconstruction (CSAR) module is built to enhance the representational ability of the semantic feature map. Furthermore, a gradient variance loss is tailored as the regularization in guiding the model learning to regularize the model in generating a faithful high-resolution image with rich textures and sharp edges. The experiments verify the effectiveness of the proposed GAMA compared with the state-of-the-art approaches.

Integration of RNA molecules data with prior-knowledge driven Joint Deep Semi-Negative Matrix Factorization for heart failure study.

Heart failure (HF) is the main manifestation of cardiovascular disease. Recent studies have shown that various RNA molecules and their complex connections play an essential role in HF’s pathogenesis and pathological progression. This paper aims to mine key RNA molecules associated with HF. We proposed a Prior-knowledge Driven Joint Deep Semi-Negative Matrix Factorization (PD-JDSNMF) model that uses a hierarchical nonlinear feature extraction method that integrates three types of data: mRNA, lncRNA, and miRNA. The PPI information is added to the model as prior knowledge, and the Laplacian constraint is used to help the model resist the noise in the genetic data. We used the PD-JDSNMF algorithm to identify significant co-expression modules. The elements in the module are then subjected to bioinformatics analysis and algorithm performance analysis. The results show that the PD-JDSNMF algorithm can robustly select biomarkers associated with HF. Finally, we built a heart failure diagnostic model based on multiple classifiers and using the Top 13 genes in the significant module, the AUC of the internal test set was up to 0.8714, and the AUC of the external validation set was up to 0.8329, which further confirmed the effectiveness of the PD-JDSNMF algorithm.

Deep Learning of Latent Variable Models for Industrial Process Monitoring

Data-driven process monitoring based on latent variable models are widely employed in industry. This article proposes a novel monitoring framework for latent variable models using hierarchical feature extraction, Bayesian inference, and weighting strategy. We first establish a deep structure to implement hierarchical latent variables extraction, the extracted features are used to construct diverse monitoring statistics. Then, we utilize Bayesian inference and proper weighting strategy to fuse various useful information. In line with the different characteristics of principal component analysis (PCA) and independent component analysis (ICA), we construct a deep PCA-ICA model for process monitoring according to the proposed framework. The deep PCA-ICA model performs hierarchical feature extraction, which can simultaneously extract deep Gaussian information and deep non-Gaussian information. The features extracted by different layers are then transformed to posterior probabilities through Bayesian inference. After that, different posterior probabilities are combined through appropriate weighting strategy to build new probabilistic statistic, which can give more synthetic monitoring results. Moreover, the Bayesian inference and weighting strategy are further used to integrate the advantages of different models by transforming various probabilistic statistics into an overall monitoring index, which can comprehensively indicate the process status. The Tennessee Eastman process is used to validate the superiority of the proposed model over the existing methods. Besides, the extracted features are further analyzed to show the effectiveness and benefits of the deep hierarchical feature extraction structure.