Multi-level Information Fusion Research Articles

A vibration-based Machine Learning type Structural Health Monitoring methodology for populations of composite aerostructures under uncertainty

A robust to uncertainty Machine Learning (ML) based Structural Health Monitoring methodology for populations of composite aerostructures is postulated. The methodology is founded upon a number of unsupervised ML algorithms for damage detection and a supervised counterpart for damage characterization. Damage detection is specifically based on two types of Healthy Subspace representations: A Multiple Model (MM) and a varying radii Hyper-Sphere (HS) type. Both are built upon response-only vibration acceleration and/or strain signals at properly selected sensor locations. Based on them, Multiple Input Single Output (MISO) Transmittance Function AutoRegressive with eXogenous (TF-ARX) excitation data driven models representing the partial structural dynamics are obtained. Decision making is then based on the model parameter vector that may be transformed and reduced via Principal Component Analysis (PCA). Damage detection is achieved via multi-level information fusion using acceleration and/or strain sensors. Damage characterization, referring to damage type, location, and level determination, is achieved via a hierarchical cosine similarity based algorithm. The methodology is successfully assessed via hundreds of experiments using a population of small-scale composite coupons for the detection and characterization of Delamination and Impact damage under material/manufacturing, temperature, excitation, and experimental uncertainty.

Development and experimental validation of a prototype system for Machine Learning based SHM in composite aerostructures

The development and experimental validation of a Machine Learning (ML) vibration based Structural Health Monitoring (SHM) prototype system for composite aerostructures operating under uncertainty is presented. Following initial training, it is intended to operate autonomously, in almost real-time, without interrupting normal operation, relying on a limited number of acceleration and strain measurements from properly selected locations on the aerostructure. It is equipped with a multi-level information fusion methodology comprising multiple sensor technologies and diagnostic algorithms for optimized SHM performance in terms of damage detection and characterization. Through its flexible software, the system provides multiple options of ML based SHM methods targeting wide applicability to various types of structures. The system is validated based on numerous experiments with healthy and damaged full-scale composite bonded spars operating under multiple uncertainty factors. Its diagnostic performance indicates that the current progress in hardware and robust vibration-based Machine Learning algorithms may lead to compact, effective, and low cost SHM systems for composite aerostructures under normal operating conditions.

ScMIC: A Deep Multi-Level Information Fusion Framework for Clustering Single-Cell Multi-Omics Data.

Cell type identification is a crucial step towards the study of cellular heterogeneity and biological processes. Advances in single-cell sequencing technology have enabled the development of a variety of clustering methods for cell type identification. However, most of existing methods are designed for clustering single omic data such as single-cell RNA-sequencing (scRNA-seq) data. The accumulation of single-cell multi-omics data provides a great opportunity to integrate different omics data for cell clustering, but also raise new computational challenges for existing methods. How to integrate multi-omics data and leverage their consensus and complementary information to improve the accuracy of cell clustering still remains a challenge. In this study, we propose a new deep multi-level information fusion framework, named scMIC, for clustering single-cell multi-omics data. Our model can integrate the attribute information of cells and the potential structural relationship among cells from local and global levels, and reduce redundant information between different omics from cell and feature levels, leading to more discriminative representations. Moreover, the proposed multiple collaborative supervised clustering strategy is able to guide the learning process of the core encoding part by learning the high-confidence target distribution, which facilitates the interaction between the clustering part and the representation learning part, as well as the information exchange between omics, and finally obtain more robust clustering results. Experiments on seven single-cell multi-omics datasets show the superiority of scMIC over existing state-of-the-art methods.

An Evolutionary Game-Theoretic Approach to Unmanned Aerial Vehicle Network Target Assignment in Three-Dimensional Scenarios

Target assignment has been a hot topic of research in the academic and industrial communities for swarms of multiple unmanned aerial vehicle (multi-UAVs). Traditional methods mainly focus on cooperative target assignment in planes, and they ignore three-dimensional scenarios for the multi-UAV network target assignment problem. This paper proposes a method for target assignment in three-dimensional scenarios based on evolutionary game theory to achieve cooperative targeting for multi-UAVs, significantly improving operational efficiency and achieving maximum utility. Firstly, we construct an evolutionary game model including game participants, a tactical strategy space, a payoff matrix, and a strategy selection probability space. Then, a multi-level information fusion algorithm is designed to evaluate the overall attack effectiveness of multi-UAVs against multiple targets. The replicator equation is leveraged to obtain the evolutionarily stable strategy (ESS) and dynamically update the optimal strategy. Finally, a typical scenario analysis and an effectiveness experiment are carried out on the RflySim platform to analyze the calculation process and verify the effectiveness of the proposed method. The results show that the proposed method can effectively provide a target assignment solution for multi-UAVs.

Multi-sensor and multi-level information fusion model for compressor blade crack detection

In order to complete the early crack detection of compressor blades under complex working conditions, a multi-sensor and multi-level information fusion model is proposed. Firstly, based on the designed correlated degree of cyclostationarity (CDCS) fusion rule, data-level fusion of multi-sensor signals is performed. The fused signal has more obvious fault features and lower redundancy. Then, based on the proposed multi-scale attention module (MSAM), a dual-branch one-dimensional convolutional neural network (1D-CNN) is designed to extract the features from the fused signal, and the dynamic feature fusion module is used to complete the feature fusion of the two branches, so as to obtain more rich and complete features. Finally, the fused features are input into the softmax classifier to complete the blade crack detection. The effectiveness and superiority of the proposed method are verified by a series of ablation experiments and comparison experiments under cross-speed conditions.

Multi-dimension and Multi-level Information Fusion for Facial Expression Recognition

Multi-dimension and Multi-level Information Fusion for Facial Expression Recognition

SANet: A novel segmented attention mechanism and multi-level information fusion network for 6D object pose estimation

Reliably and rapidly estimating the 6D position of an object is a critical challenge when using Internet of Things (IoT) technologies for monitoring. Nowadays, the prevalent 6D pose estimation architecture is based on a two-stage technique, which requires a significant amount of time for both training and deploying the algorithm in actual applications. Additionally, the majority of approaches include intricate high-low level features in the network that greatly influence training but contribute less to testing. To enable more accurate 6D object pose estimation while shortening the deployment time, we used a single-stage end-to-end algorithm to design the network. In this paper, we propose SANet, which is composed of a segmented attention module and a multi-level information fusion module. Specifically, by extracting high-level semantic information from images before fusing them to the decoder, and by removing redundant information using the multi-level information fusion module, the feature fusion complexity of the model is reduced by extracting high level features. In addition, the segmented attention module can suppress unreliable information to enhance network learning of channel and spatial information, enabling the network to more accurately understand the geometric aspects of the object. Extensive experiments on LM and LMO datasets demonstrate that our method outperforms state-of-the-art baselines, ranking 1st in both speed and accuracy.

Using CNN with Multi-Level Information Fusion for Image Denoising

Deep convolutional neural networks (CNN) with hierarchical architectures have obtained good results for image denoising. However, in some cases where the noise level is unknown and the image background is complex, it is challenging to obtain robust information through CNN. In this paper, we present a multi-level information fusion CNN (MLIFCNN) in image denoising containing a fine information extraction block (FIEB), a multi-level information interaction block (MIIB), a coarse information refinement block (CIRB), and a reconstruction block (RB). In order to adapt to more complex image backgrounds, FIEB uses parallel group convolution to extract wide-channel information. To enhance the robustness of the obtained information, a MIIB uses residual operations to act in two sub-networks for implementing the interaction of wide and deep information to adapt to the distribution of different noise levels. To enhance the stability of the training denoiser, CIRB stacks common and group convolutions to refine the obtained information. Finally, RB uses a residual operation to act in a single convolution in order to obtain the resultant clean image. Experimental results show that our method is better than many other excellent methods, both in terms of quantitative and qualitative aspects.

Deep-Learning-Based Context-Aware Multi-Level Information Fusion Systems for Indoor Mobile Robots Safe Navigation.

Hazardous object detection (escalators, stairs, glass doors, etc.) and avoidance are critical functional safety modules for autonomous mobile cleaning robots. Conventional object detectors have less accuracy for detecting low-feature hazardous objects and have miss detection, and the false classification ratio is high when the object is under occlusion. Miss detection or false classification of hazardous objects poses an operational safety issue for mobile robots. This work presents a deep-learning-based context-aware multi-level information fusion framework for autonomous mobile cleaning robots to detect and avoid hazardous objects with a higher confidence level, even if the object is under occlusion. First, the image-level-contextual-encoding module was proposed and incorporated with the Faster RCNN ResNet 50 object detector model to improve the low-featured and occluded hazardous object detection in an indoor environment. Further, a safe-distance-estimation function was proposed to avoid hazardous objects. It computes the distance of the hazardous object from the robot's position and steers the robot into a safer zone using detection results and object depth data. The proposed framework was trained with a custom image dataset using fine-tuning techniques and tested in real-time with an in-house-developed mobile cleaning robot, BELUGA. The experimental results show that the proposed algorithm detected the low-featured and occluded hazardous object with a higher confidence level than the conventional object detector and scored an average detection accuracy of 88.71%.

Semantic-Structure-Aware Multi-Level Information Fusion for Robust Global Orientation Optimization of Autonomous Mobile Robots.

Multi-camera-based simultaneous localization and mapping (SLAM) has been widely applied in various mobile robots under uncertain or unknown environments to accomplish tasks autonomously. However, the conventional purely data-driven feature extraction methods cannot utilize the rich semantic information in the environment, which leads to the performance of the SLAM system being susceptible to various interferences. In this work, we present a semantic-aware multi-level information fusion scheme for robust global orientation estimation. Specifically, a visual semantic perception system based on the synthesized surround view image is proposed for the multi-eye surround vision system widely used in mobile robots, which is used to obtain the visual semantic information required for SLAM tasks. The original multi-eye image was first transformed to the synthesized surround view image, and the passable space was extracted with the help of the semantic segmentation network model as a mask for feature extraction; moreover, the hybrid edge information was extracted to effectively eliminate the distorted edges by further using the distortion characteristics of the reverse perspective projection process. Then, the hybrid semantic information was used for robust global orientation estimation; thus, better localization performance was obtained. The experiments on an intelligent vehicle, which was used for automated valet parking both in indoor and outdoor scenes, showed that the proposed hybrid multi-level information fusion method achieved at least a 10-percent improvement in comparison with other edge segmentation methods, the average orientation estimation error being between 1 and 2 degrees, much smaller than other methods, and the trajectory drift value of the proposed method was much smaller than that of other methods.

Multiview Feature Learning and Multilevel Information Fusion for Joint Classification of Hyperspectral and LiDAR Data

Multiview Feature Learning and Multilevel Information Fusion for Joint Classification of Hyperspectral and LiDAR Data

DG-ECG: Multi-stream deep graph learning for the recognition of disease-altered patterns in electrocardiogram

Representation learning of electrocardiogram (ECG) has been an active research field for the automated detection of cardiac disease. In addition to extracting time and frequency domain features of ECG, an increasing amount of studies have adapted deep neural networks for the recognition of disease-altered ECG patterns. However, many deep learning models are deployed as blackboxes without fully exploring disease-pertinent information hidden in the signal. This, as a result, diminishes the efficacy and interpretability of the model and impedes applications in clinical practice. To address this problem, we develop a new multi-stream deep graph learning of ECG (DG-ECG) framework, which integrates multi-stream graph neural networks to uncover disease-altered ECG patterns from multifold perspectives (e.g., the morphology and rhythm of ECG signals). In each stream, visibility graphs are modeled to transform signal patterns into graph topological features, which are then mined by graph convolution. In addition, attention mechanisms are integrated into DG-ECG for multi-level information fusion to enhance its detection power and interpretability. Experimental results have demonstrated that the developed DG-ECG is better capable of gleaning disease-pertinent information from multi-channel ECG signals compared to benchmark models. The developed framework is extendable and suited for the pattern recognition of various cardiac disorders.

Multi-level information fusion with motion constraints: Key to achieve high-precision gait analysis using low-cost inertial sensors

Gait can reflect locomotion and physical condition and thus is used to assess people's health. Traditional high-precision gait analysis devices are expensive and are limited to laboratory conditions, so the trend is to use wearable devices to analyze gait. The existing wearable inertial-based gait-analysis methods have stride-length and foot-clearance estimation accuracy of 2.0 cm to 5.0 cm (in root mean square) and have limitations. They have not considered the regularity of the movement between the toe and the heel, the varying dual-feet distance due to foot dynamics, and the diversity of sensors used. Involving these factors, this paper achieves an improved gait-analysis system that provides stride-length and foot-clearance accuracy of 1.5 cm and 1.0 cm, respectively. Such accuracy is state-of-the-art for low-cost inertial systems and is even competitive with those from visual-sensor-based gait-analysis systems. A key to the proposed method is a new multi-level information fusion architecture and the extraction of human-walking constraints. The information-fusion architecture involves data fusion from the sensor, single-foot, and dual-foot levels. Two gait-characteristic-based motion constraints are presented to achieve such fusion, including the toe-heel constant distance constraint and the dual-foot flexible distance constraint. To implement these constraints, a constrained Kalman filter is constructed. The corresponding hardware system has been designed using multiple dollar-level inertial measurement units.

What happens next? Combining enhanced multilevel script learning and dual fusion strategies for script event prediction

Script event prediction (SEP), aiming at predicting next event from context event sequences (i.e., scripts), has played an important role in many real-world applications such as government decision-making. While most of the existing research only depend on the top-level event prediction, they ignore the influence of other bottom levels or other relationship modeling manners. In this paper, we focus on the problem of SEP via multilevel script learning where the goal of is to explore a multistage, multiprediction and multilevel information fusion model for SEP. This is challenging in (1) simultaneously modeling of the multilevel event relationship semantic information and (2) effectively designing multilevel information fusion strategies. In this paper, we propose a new script event prediction model based on Enhanced Multilevel script learning and Dual Fusion strategies, named EMDF-Net. Specifically, EMDF-Net designs the multilevel (event/chain/segment level) script learning to model both temporal and casual information as well as the rich structural relevance via neural stacking of self-attention mechanism and graph neural networks. Then it proposes dual fusion strategies to fully integrate different-level information by nonlinear feature composition and weighted score fusion. Finally, a deep supervision strategy is utilized to end-to-end train the whole model and provide a good initialization for information fusion. Experimental results on the popular NYT corpus demonstrate the effectiveness and superiority of EMDF-Net.

Diagnosing Diabetic Retinopathy in OCTA Images Based on Multilevel Information Fusion Using a Deep Learning Framework.

Objective As an extension of optical coherence tomography (OCT), optical coherence tomographic angiography (OCTA) provides information on the blood flow status at the microlevel and is sensitive to changes in the fundus vessels. However, due to the distinct imaging mechanism of OCTA, existing models, which are primarily used for analyzing fundus images, do not work well on OCTA images. Effectively extracting and analyzing the information in OCTA images remains challenging. To this end, a deep learning framework that fuses multilevel information in OCTA images is proposed in this study. The effectiveness of the proposed model was demonstrated in the task of diabetic retinopathy (DR) classification. Method First, a U-Net-based segmentation model was proposed to label the boundaries of large retinal vessels and the foveal avascular zone (FAZ) in OCTA images. Then, we designed an isolated concatenated block (ICB) structure to extract and fuse information from the original OCTA images and segmentation results at different fusion levels. Results The experiments were conducted on 301 OCTA images. Of these images, 244 were labeled by ophthalmologists as normal images, and 57 were labeled as DR images. An accuracy of 93.1% and a mean intersection over union (mIOU) of 77.1% were achieved using the proposed large vessel and FAZ segmentation model. In the ablation experiment with 6-fold validation, the proposed deep learning framework that combines the proposed isolated and concatenated convolution process significantly improved the DR diagnosis accuracy. Moreover, inputting the merged images of the original OCTA images and segmentation results further improved the model performance. Finally, a DR diagnosis accuracy of 88.1% (95%CI ± 3.6%) and an area under the curve (AUC) of 0.92 were achieved using our proposed classification model, which significantly outperforms the state-of-the-art classification models. As a comparison, an accuracy of 83.7 (95%CI ± 1.5%) and AUC of 0.76 were obtained using EfficientNet. Significance. The visualization results show that the FAZ and the vascular region close to the FAZ provide more information for the model than the farther surrounding area. Furthermore, this study demonstrates that a clinically sophisticated designed deep learning model is not only able to effectively assist in the diagnosis but also help to locate new indicators for certain illnesses.

A novel methodology for modal parameter identification of arch dam based on multi-level information fusion

Operational modal analysis plays an important role in the structural health monitoring and safety diagnosis of arch dam. However, due to background noise, it is difficult to accurately extract the effective characteristics information of arch dam from the vibration responses whose amplitudes are too small under ambient vibration, and the deviation caused by traditional identification methods will directly affect the estimation accuracy for structural modal parameters. Therefore, a novel methodology for modal parameter identification of arch dam based on multi-level information fusion is proposed in this paper. The proposed method is based on multi-sensor data-level fusion to identify the structural natural frequency and damping ratio, which greatly preserves and extracts structural modal properties in the vibration responses. Meanwhile, structural mode shapes are identified based on dynamic feature-level fusion, which significantly improves the identification accuracy. The effectiveness and feasibility of the proposed method are verified by the modal results of digital signals and simulated signals in the 7-DOF system. Prototype engineering case shows that the closely spaced and high-frequency modes can be decomposed and identified by the proposed method, and this method has a higher identification accuracy, which can provide a new idea for modal parameter identification of arch dam.

Hierarchical shared transfer learning for biomedical named entity recognition

BackgroundBiomedical named entity recognition (BioNER) is a basic and important medical information extraction task to extract medical entities with special meaning from medical texts. In recent years, deep learning has become the main research direction of BioNER due to its excellent data-driven context coding ability. However, in BioNER task, deep learning has the problem of poor generalization and instability.Resultswe propose the hierarchical shared transfer learning, which combines multi-task learning and fine-tuning, and realizes the multi-level information fusion between the underlying entity features and the upper data features. We select 14 datasets containing 4 types of entities for training and evaluate the model. The experimental results showed that the F1-scores of the five gold standard datasets BC5CDR-chemical, BC5CDR-disease, BC2GM, BC4CHEMD, NCBI-disease and LINNAEUS were increased by 0.57, 0.90, 0.42, 0.77, 0.98 and − 2.16 compared to the single-task XLNet-CRF model. BC5CDR-chemical, BC5CDR-disease and BC4CHEMD achieved state-of-the-art results.The reasons why LINNAEUS’s multi-task results are lower than single-task results are discussed at the dataset level.ConclusionCompared with using multi-task learning and fine-tuning alone, the model has more accurate recognition ability of medical entities, and has higher generalization and stability.

Two brains, one target: Design of a multi-level information fusion model based on dual-subject RSVP

Background. Rapid serial visual presentation (RSVP) based brain-computer interface (BCI) is widely used to categorize the target and non-target images. The available information limits the prediction accuracy of single-trial using single-subject electroencephalography (EEG) signals.New Method. Hyperscanning is a new manner to record two or more subjects’ signals simultaneously. So we designed a multi-level information fusion model for target image detection based on dual-subject RSVP, namely HyperscanNet. The two modules of this model fuse the data and features of the two subjects at the data and feature layers. A chunked long and short-term memory artificial neural network (LSTM) was used in the time dimension to extract features at different periods separately, completing fine-grained underlying feature extraction. While the feature layer is fused, some plain operations are used to complete the fusion of the data layer to ensure that important information is not missed.Results. Experimental results show that the F1-score (the harmonic mean of precision and recall) of this method with best group of channels and segment length is 82.76%.Comparison with existing methods. This method improves the F1-score by at least 5% compared to single-subject target detection.Conclusions. Target detection can be accomplished by the two subjects’ collaboration to achieve a higher and more stable F1-score than a single subject.

Research on Driver Status Recognition System of Intelligent Vehicle Terminal Based on Deep Learning

Automobile safety driving technology is a hot topic in today’s society, which is very significant to the social transportation system. Vehicle driving behavior monitoring is the foundation and core of safe driving techniques. The research on existing vehicle safety technology can not only improve the understanding of current safe driving research progress, but also provide reference for future researchers. This paper proposes a state recognition system based on a three-dimensional convolutional neural network, which can identify several improper states frequently encountered by drivers during driving, including drinking, making phone calls, and smoking, and can also issue alarm interventions. The system takes the collected continuous video frame information as the input of the three-dimensional convolutional network, carries out multi-level feature extraction and spatio-temporal information fusion, and identifies the driver state according to the extracted spatio-temporal features. The state is judged by the facial feature points of the video stream, and the design of the video surveillance driver state recognition system is completed. Then, the driver status recognition is improved and optimized, and finally, the actual deployment of the driver status recognition system on the mobile terminal is completed. A large number of experimental results show that the driver status recognition system proposed in this paper has achieved upper identification accuracy.

A novel fault diagnosis method based on multi-level information fusion and hierarchical adaptive convolutional neural networks for centrifugal blowers

Multi-sensor data fusion can provide abundant and complementary fault information. To improve the accuracy and robustness of diagnosis, this paper proposes a novel fault diagnosis method for centrifugal blowers based on multi-level information fusion and hierarchical adaptive convolutional neural network (HACNN). Multi-level information fusion integrates temporal information, feature extraction, feature selection into data fusion. This fusion strategy can acquire comprehensive and representative fault information from multi-sensor signals. The constructed HACNN greatly enhances the feature learning ability of the network and avoids unnecessary computational consumption by adaptive expansion. The effectiveness of the proposed method is evaluated by using datasets from a centrifugal blower test rig. The experimental results show that the testing accuracy and F1-score of the proposed method reach to 98.18%, which is obviously higher than that of CNN, DNN, DBN, BPNN and SVM in corresponding fusion method. It proves that the proposed method has superior diagnosis performance.