Multi-sensor Features Research Articles

DeployFusion: A Deployable Monocular 3D Object Detection with Multi-Sensor Information Fusion in BEV for Edge Devices.

To address the challenges of suboptimal remote detection and significant computational burden in existing multi-sensor information fusion 3D object detection methods, a novel approach based on Bird's-Eye View (BEV) is proposed. This method utilizes an enhanced lightweight EdgeNeXt feature extraction network, incorporating residual branches to address network degradation caused by the excessive depth of STDA encoding blocks. Meantime, deformable convolution is used to expand the receptive field and reduce computational complexity. The feature fusion module constructs a two-stage fusion network to optimize the fusion and alignment of multi-sensor features. This network aligns image features to supplement environmental information with point cloud features, thereby obtaining the final BEV features. Additionally, a Transformer decoder that emphasizes global spatial cues is employed to process the BEV feature sequence, enabling precise detection of distant small objects. Experimental results demonstrate that this method surpasses the baseline network, with improvements of 4.5% in the NuScenes detection score and 5.5% in average precision for detection objects. Finally, the model is converted and accelerated using TensorRT tools for deployment on mobile devices, achieving an inference time of 138 ms per frame on the Jetson Orin NX embedded platform, thus enabling real-time 3D object detection.

Overbounding Multipath Error in Urban Canyon With LSTM Using Multi-Sensor Features

Overbounding Multipath Error in Urban Canyon With LSTM Using Multi-Sensor Features

Online Detection of Laser Welding Penetration Depth Based on Multi-Sensor Features.

The accurate online detection of laser welding penetration depth has been a critical problem to which the industry has paid the most attention. Aiming at the laser welding process of TC4 titanium alloy, a multi-sensor monitoring system that obtained the keyhole/molten pool images and laser-induced plasma spectrum was built. The influences of laser power on the keyhole/molten pool morphologies and plasma thermo-mechanical characteristics were investigated. The results showed that there were significant correlations among the variations of the keyhole-molten pool, plasma spectrum, and penetration depth. The image features and spectral features were extracted by image processing and dimension-reduction methods, respectively. Moreover, several penetration depth prediction models based on single-sensor features and multi-sensor features were established. The mean square error of the neural network model built by multi-sensor features was 0.0162, which was smaller than that of the model built by single-sensor features. The established high-precision model provided a theoretical basis for real-time feedback control of the penetration depth in the laser welding process.

Multisensor fusion-based digital twin for localized quality prediction in robotic laser-directed energy deposition

Early detection of defects, such as keyhole pores and cracks is crucial in laser-directed energy deposition (L-DED) additive manufacturing (AM) to prevent build failures. However, the complex melt pool behaviour cannot be adequately captured by conventional single-modal process monitoring approaches. This study introduces a multisensor fusion-based digital twin (MFDT) for localized quality prediction in the robotic L-DED process. The data used in multisensor fusion includes features extracted from a coaxial melt pool vision camera, a microphone, and an off-axis short wavelength infrared thermal camera. The key novelty of this work is a spatiotemporal data fusion method that synchronizes multisensor features with the real-time robot motion data to achieve localized quality prediction. Optical microscope (OM) images of the printed part are used to locate defect-free and defective regions (i.e., cracks and keyhole pores), which serve as ground truth labels for training supervised machine learning (ML) models for quality prediction. The trained ML model is then used to generate a virtual quality map that registers quality prediction outcomes within the 3D volume of the printed part, thus eliminating the need of physical inspections by destructive methods. Experiments show that the virtual quality map closely matches the actual quality observed by OM. Compared to traditional single-sensor-based quality prediction, the MFDT has achieved a significantly higher quality prediction accuracy (96%), a higher ROC-AUC score (99%), and a lower false alarm rate (4.4%). As a result, the MFDT is a more reliable method for defect prediction. The proposed MFDT also lays the groundwork for our future development of a self-adaptive hybrid processing strategy that combines machining with AM for defect removal and quality improvement.

A novel collaborative bearing fault diagnosis method based on multi-scale dynamic fusion network under speed fluctuating condition

Improving bearing fault diagnosis accuracy under speed fluctuation is a challenge in engineering applications. With the development of big data processing technology, a new solution, multi-sensor complementary information, has emerged. However, single-scale dimension compression, which is adopted in most multi-sensor data fusion methods, captures only a small amount of valuable information. To deal with this deficiency, a multi-scale dynamic fusion network (MSDFN) is proposed. First, considering the existence of non-stationary features in the fluctuating speed signal, the FReLU function is adopted to activate the features after considering contextual information. Then, multi-sensor features are fused by multiple scales to obtain richer feature information, and fusion features at different scales are weighted by using the attention mechanism. Finally, batch normalization is employed to standardize the variable speed feature distribution. The validity of the MSDFN is proved by conducting fault diagnosis experiments on two bearings under speed fluctuating conditions. Experimental results indicate that the MSDFN is not only effective in identifying various types of fault samples, but also shows higher stability in multiple trials when compared with other methods.

MULTISENSOR FUSION-BASED DIGITAL TWIN IN ADDITIVE MANUFACTURING FOR IN-SITU QUALITY MONITORING AND DEFECT CORRECTION

AbstractEarly detection and correction of defects are critical in additive manufacturing (AM) to avoid build failures. In this paper, we present a multisensor fusion-based digital twin for in-situ quality monitoring and defect correction in a robotic laser-directed energy deposition process. Multisensor fusion sources consist of an acoustic sensor, an infrared thermal camera, a coaxial vision camera, and a laser line scanner. The key novelty and contribution of this work are to develop a spatiotemporal data fusion method that synchronizes and registers the multisensor features within the part's 3D volume. The fused dataset can be used to predict location-specific quality using machine learning. On-the-fly identification of regions requiring material addition or removal is feasible. Robot toolpath and auto-tuned process parameters are generated for defect correction. In contrast to traditional single-sensor-based monitoring, multisensor fusion allows for a more in-depth understanding of underlying process physics, such as pore formation and laser-material interactions. The proposed methods pave the way for self-adaptation AM with higher efficiency, less waste, and cleaner production.

In situ monitoring in laser melt injection based on fusion of infrared thermal and high-speed camera images

Unstable reliability and poor repeatability in the production of metal matrix composite (MMC) parts are primary challenges currently faced by laser melt injection (LMI). In situ monitoring of LMI during the part fabrication process provides a sound solution. This research presents promising results that aim to bridge the knowledge gap by determining an innovative technique to correlate the combined features of infrared thermal and high-speed cameras to state monitoring of LMI and the forming quality identification of MMC. The combined features of the molten pool and spatter were extracted based on the understanding of LMI physical mechanisms and then were selected by the fisher score algorithm with the introduction of weighting factor (WF-FS). The support vector machine (SVM) prediction models based on single-sensor features, multi-sensor features, and fusion features decomposed with different methods were developed to identify the quality of MMC. The results indicated that the multi-sensor features exhibit better classification performance. The empirical mode decomposition (EMD) method can improve classification performance by mining the depth and subtle feature information of multi-source features. The optimal feature subset with 9 EMD features was selected by principal component analysis to balance computational period and classification accuracy. This work provides an effective and novel approach to monitor the LMI process and evaluate the MMC part fabrication quality.

DSFNet: A Distributed Sensors Fusion Network for Action Recognition

Human action recognition (HAR) has become a hot topic in the field of computer vision and pattern recognition due to its wide range of application prospects. In most deep learning and multisensor data-based action recognition works, sequences from sensors are first stacked into 2-D images and then fed into convolutional neural network (CNN) models to extract features. This approach based on data-level fusion learns different sensor data features but loses the uniqueness of different sensor data. In this article, we propose a new deep learning model called distributed sensors fusion network (DSFNet). For the property that acceleration sequences can reflect motion trends, we first learn the features of different sensors and then use the Transformer encoder module to model the dependencies of multisensor actions and extract features. For the property that angular velocity can reflect the direction and velocity of the local pose, we first learn the features of a single sensor in different motion directions in time sequence and then learn the output feature maps of multisensor features in time sequence and extract the features. Finally, the outputs of different data modalities are used for decision-level fusion, which significantly improves the performance of the model. We evaluate the performance of the proposed model on the self-built dataset Changzhou University: a comprehensive multi-modal human action dataset (CZU-MHAD), and the experimental results show that the DSFNet model outperforms the existing methods.

Adaptive Band Extraction Based on Low Rank Approximated Nonnegative Tucker Decomposition for Anti-Friction Bearing Faults Diagnosis Using Measured Vibration Data

Condition monitoring and fault diagnosis are topics of growing interest for improving the reliability of modern industrial systems. As critical structural components, anti-friction bearings often operate under harsh conditions and are contributing factors of system failures. Efforts have been cast on bearing diagnostics under the sensor fusion and machine learning framework, whilst challenges remain open on the identification of incipient faults. In this paper, exploiting multi-way representations and decompositions of measured vibration data, a novel band separation method based on the factorization of spectrogram tensors using the low rank approximated nonnegative Tucker decomposition (LRANTD) is proposed and applied to identify detailed fault signatures from the spectral, temporal, and spatial dimensions, flexible for extracting multi-sensor features and multi-dimensional correlations. With the proposed method, informative frequency bands of the latent vibrational components can be automatically extracted, in accordance with the inherent temporal patterns that can be conveniently fed for spectral analysis and fault discrimination. Furthermore, an improved cross-spectrum can be calculated from multi-channel vibrations via LRANTD with enhanced fault features. Based on the real-world vibration data of the accelerated bearing life tests, detailed experimental studies and thorough comparisons to the conventional benchmarks have verified the effectiveness of the reported diagnostic methodology. The proposed method significantly improves the presence of the bearing frequency peaks distinctly over the background noises in the spectrum and hence improves the bearing defect detection process.

Milling tool wear prediction using multi-sensor feature fusion based on stacked sparse autoencoders

Tool wear prediction was significant for improving processing efficiency, ensuring product quality and reducing tool costs in manufacturing. In this paper, a novel deep learning method based on stacked sparse autoencoders (SSAE) and multi-sensor feature fusion was proposed for milling tool wear prediction. The signal features were extracted in time, frequency and time–frequency domains and the optimum multi-sensor features were determined by correlation analysis, which were input into the SSAE for deep feature learning. Backpropagation neural network (BPNN) was utilized to establish the prediction model of tool wear. Different milling wear experiment datasets were applied to verify the predictive performance of the trained models. The prediction results showed that the proposed model had the minimum root mean square error (RMSE) and maximum coefficient of determination (R2), which outperformed the comparative predictive models. The combination of multi-sensor feature fusion and deep learning method was demonstrated for improving the predictive performance.

Generative Feature Extraction From Sentinel 1 and 2 Data for Prediction of Forest Aboveground Biomass in the Italian Alps

Aboveground biomass (AGB) is an important forest attribute directly linked to the forest carbon pool. The use of satellite remote sensing (RS) data has increased for AGB prediction due to their large footprint and low-cost availability. However, they have been limited due to saturation effect that leads to low prediction precision. In this study, we propose an innovative and dynamic architecture based on generative neural network that extracts target oriented generative features for predicting forest AGB using satellite RS data. These features are more resilient to mixed forest types and geographical conditions as compared to the traditional features and models. The effectiveness of the proposed features was assessed by experiments performed using multispectral (MS), Synthetic aperture Radar (SAR) and combined dual-source (DS) datasets. The proposed model achieved best performance in terms of precision, model agreement and overfitting as compared to the other conventional models for all analyzed datasets. The t-distributed stochastic neighbor embedding (t-SNE) scatterplots of the generative features clearly show one dimension of the feature space associated with the target AGB. Feature importance analysis indicated that the produced generative features were more significant than the conventional analytical features. Also, the model provided a robust framework for homogeneous fusion of multi-sensor features from satellite RS data for predicting AGB.

MMFNet: Multisensor Data and Multiscale Feature Fusion Model for Intelligent Cross-Domain Machinery Fault Diagnosis

Although cross-domain fault diagnosis has received much attention in intelligent mechanical fault diagnosis, most existing methods only achieve knowledge transfer within the data from a single sensor. For a complex industrial system, multiple sensors are usually required to monitor its operating conditions coordinately. In such a situation, the fault diagnosis capability of existing cross-domain methods might be impaired significantly. To address this issue, in this article, a multisensor data and multiscale feature fusion network (MMFNet) is proposed to achieve cross-domain fault diagnosis using multisensor data. More specifically, a multiscale feature extraction module based on the pyramid principle is designed to fuse both deep and shallow features of the complex raw signal. Then, a Transformer-based multiscale features fusion and domain adaptation module is developed to fuse multiscale features and achieve the domain invariant of the sensor data. Finally, the proposed transferable Transformer is used for multisensor features cross-domain fusion. With the developed MMFNet, the features from multiple sensors can be comprehensively captured so that a more accurate fault diagnosis can be achieved. A series of experiments on a complex planetary gearbox demonstrate the effectiveness and practicability of the proposed method for fault diagnosis.

A novel Hot-Flash classification algorithm via multi-sensor features integration.

We aim to evaluate the feasibility and performance of a novel hot flash (HF) classification algorithm based on multisensor features integration using commercial wearable sensors. First, we processed feature sets from wrist-based multi-sensor data (photoplethysmography, motion, temperature, skin conductance and). Then, we classified (Decision Tree) physiological-recorded HFs (N=27) recorded from three menopause women, and we assessed the algorithm performance against gold-standard HF expert evaluation. The results indicated that while skin conductance features alone explain most of the variance (~65%) in HF classification, the multi-sensor approach achieved above 90% sensitivity at 95.6% specificity in HF classification and showed advantages under conditions of signal corruption and different biobehavioral states (sleep vs wake). The proposed new multi-sensor approach showed being promising in HF classification using common commercially-available wearable sensors and target locations.Clinical Relevance- The development of "user-centered" accurate, automatic detection systems for HFs can advance the measurement and treatment of HFs.

Multi-condition identification in milling Ti-6Al-4V thin-walled parts based on sensor fusion

Chatter and tool wear always coexist during the milling of Ti-6Al-4V thin-walled parts, while the boundary characteristics, time-varying, modal coupling, and position dependence characteristics in milling thin-walled parts lead to the variable spatial time–frequency distribution of the signal features related to milling conditions. As a result, the existing milling monitoring methods can only identify single chatter or tool wear while ignoring another condition. Therefore, in order to identify the chatter and tool wear simultaneously, a multi-condition identification method based on sensor fusion is proposed by fusing multi-source heterogeneous data with different spatial time–frequency distributions. Multi-sensor features of the milling process are extracted from sound, acceleration, and cutting bending moment signals. To reduce the complexity of the original feature dataset, an upgraded principal component analysis (UPCA) algorithm is proposed by screening sensor features in specific frequency bands. Moreover, a new signal feature considering energy proportion is extracted to improve the identification accuracy of multiple conditions. The process parameters are designed according to the stability lobe diagram (SLD) to improve the experimental efficiency. The experimental results show that the proposed method can identify the multi-milling condition composed of chatter and tool wear failure. With the help of UPCA and energy proportion, the computational efficiency is also improved, and the identification accuracy of the proposed method reaches 93.75%, which is much higher than the accuracy of classifiers with traditional data processing methods.

A novel method for journal bearing degradation evaluation and remaining useful life prediction under different working conditions

Accurate bearing degradation performance analysis and remaining useful life (RUL) prediction are significant to prevent major accidents and economic losses in industry. Data-driven methods have emerged as reliable algorithms for RUL prediction. These existing approaches assume that the training (source) and testing (target) samples have the same probability distribution. However, the obtained run-to-failure datasets with different work conditions are usually from different domains. The distribution discrepancy between the source and target domain will reduce the accuracy of RUL prediction models when only source domain data in one working condition is trained. To solve the problem, this paper proposes a novel transfer learning method for journal bearing RUL prediction under different work conditions based on the LSTM-DNN network with domain adaptation. The multi-sensor run-to-failure datasets of journal bearings are collected and the extracted multi-sensor features are used for degradation assessment through the fuzzy c-means (FCM) clustering algorithm and the determination of degradation occurrence time (DOT). The multi-sensor feature representations and RUL values after DOT are used to validate the effectiveness of the proposed model. The results show that the proposed method has higher accuracy in journal bearing RUL prediction under different work conditions and outperforms other transfer learning approaches.

Extraction of built-up area using multi-sensor data—A case study based on Google earth engine in Zhejiang Province, China

ABSTRACT Accurate and up-to-date built-up area mapping is of great importance to the science community, decision-makers, and society. Therefore, satellite-based, built-up area (BUA) extraction at medium resolution with supervised classification has been widely carried out. However, the spectral confusion between BUA and bare land (BL) is the primary hindering factor for accurate BUA mapping over large regions. Here we propose a new methodology for the efficient BUA extraction using multi-sensor data under Google Earth Engine cloud computing platform. The proposed method mainly employs intra-annual satellite imagery for water and vegetation masks, and a random-forest machine learning classifier combined with auxiliary data to discriminate between BUA and BL. First, a vegetation mask and water mask are generated using NDVI (normalized differenced vegetation index) max in vegetation growth periods and the annual water-occurrence frequency. Second, to accurately extract BUA from unmasked pixels, consisting of BUA and BL, random-forest-based classification is conducted using multi-sensor features, including temperature, night-time light, backscattering, topography, optical spectra, and NDVI time-series metrics. This approach is applied in Zhejiang Province, China, and an overall accuracy of 92.5% is obtained, which is 3.4% higher than classification with spectral data only. For large-scale BUA mapping, it is feasible to enhance the performance of BUA mapping with multi-temporal and multi-sensor data, which takes full advantage of datasets available in Google Earth Engine.

A Multi-Sensor Fusion Framework Based on Coupled Residual Convolutional Neural Networks

Multi-sensor remote sensing image classification has been considerably improved by deep learning feature extraction and classification networks. In this paper, we propose a novel multi-sensor fusion framework for the fusion of diverse remote sensing data sources. The novelty of this paper is grounded in three important design innovations: 1- a unique adaptation of the coupled residual networks to address multi-sensor data classification; 2- a smart auxiliary training via adjusting the loss function to address classifications with limited samples; and 3- a unique design of the residual blocks to reduce the computational complexity while preserving the discriminative characteristics of multi-sensor features. The proposed classification framework is evaluated using three different remote sensing datasets: the urban Houston university datasets (including Houston 2013 and the training portion of Houston 2018) and the rural Trento dataset. The proposed framework achieves high overall accuracies of 93.57%, 81.20%, and 98.81% on Houston 2013, the training portion of Houston 2018, and Trento datasets, respectively. Additionally, the experimental results demonstrate considerable improvements in classification accuracies compared with the existing state-of-the-art methods.

Rapid Multi-Sensor Feature Fusion Based on Non-Stationary Kernel JADE for the Small-Amplitude Hunting Monitoring of High-Speed Trains

Joint Approximate Diagonalization of Eigen-matrices (JADE) cannot deal with non-stationary data. Therefore, in this paper, a method called Non-stationary Kernel JADE (NKJADE) is proposed, which can extract non-stationary features and fuse multi-sensor features precisely and rapidly. In this method, the non-stationarity of the data is considered and the data from multi-sensor are used to fuse the features efficiently. The method is compared with EEMD-SVD-LTSA and EEMD-JADE using the bearing fault data of CWRU, and the validity of the method is verified. Considering that the vibration signals of high-speed trains are typically non-stationary, it is necessary to utilize a rapid feature fusion method to identify the evolutionary trends of hunting motions quickly before the phenomenon is fully manifested. In this paper, the proposed method is applied to identify the evolutionary trend of hunting motions quickly and accurately. Results verify that the accuracy of this method is much higher than that of the EEMD-JADE and EEMD-SVD-LTSA methods. This method can also be used to fuse multi-sensor features of non-stationary data rapidly.

PI-RCNN: An Efficient Multi-Sensor 3D Object Detector with Point-Based Attentive Cont-Conv Fusion Module

LIDAR point clouds and RGB-images are both extremely essential for 3D object detection. So many state-of-the-art 3D detection algorithms dedicate in fusing these two types of data effectively. However, their fusion methods based on Bird's Eye View (BEV) or voxel format are not accurate. In this paper, we propose a novel fusion approach named Point-based Attentive Cont-conv Fusion(PACF) module, which fuses multi-sensor features directly on 3D points. Except for continuous convolution, we additionally add a Point-Pooling and an Attentive Aggregation to make the fused features more expressive. Moreover, based on the PACF module, we propose a 3D multi-sensor multi-task network called Pointcloud-Image RCNN(PI-RCNN as brief), which handles the image segmentation and 3D object detection tasks. PI-RCNN employs a segmentation sub-network to extract full-resolution semantic feature maps from images and then fuses the multi-sensor features via powerful PACF module. Beneficial from the effectiveness of the PACF module and the expressive semantic features from the segmentation module, PI-RCNN can improve much in 3D object detection. We demonstrate the effectiveness of the PACF module and PI-RCNN on the KITTI 3D Detection benchmark, and our method can achieve state-of-the-art on the metric of 3D AP.

Real-time monitoring of high-power disk laser welding statuses based on deep learning framework

The laser welding quality is determined by its welding statuses, and online welding statuses are depicted by the real-time signals captured from the welding process. A multiple-sensor system is designed to obtain information as comprehensive as possible for welding statuses monitoring. The multiple-sensor system includes an auxiliary illumination visual sensor system, an ultraviolet and visible band visual sensor system, a spectrometer and two photodiodes. The signals captured by different sensors are analyzed via signal or digital image processing algorithms, and distinct features are extracted from these signals to depict the online welding statuses. A deep learning framework based on stacked sparse autoencoder (SSAE) is established to model the relationship between the multi-sensor features and their corresponding welding statuses, and Genetic algorithm (GA) is applied to optimize the parameters of the SSAE framework (SSAE-GA). The proposed framework achieves higher accuracy and stronger robustness in monitoring welding status by comparing with the backpropagation neural network, support vector machine and random forest. Three new experiments with different welding parameters are implemented to validate the effectiveness and generalization of our proposed method. This study provides a novel and accurate method for high-power disk laser welding status monitoring.