Information Fusion Strategy Research Articles

Temperature interpretation method for temperature indicating paint based on spectrogram

Accurate interpretation of temperature indicating paint (TIP) is of great importance in aviation and other industrial applications. This study presents a novel temperature interpretation approach based on the 2D spectrogram of TIP. A deep learning model called ESTNet(efficient spectrogram-temperature network) has been developed. The Gramian Angular Field (GAF) and the Continuous Wavelet Transform (CWT) are used to transform 1D reflectance spectrum into 2D spectrogram, GASF-Graph and scale2-Graph. The residual blocks and two composite coefficients are used to design RSTNet with an accuracy of 94% on KN3A samples when the input is GASF-Graph or scale2-Graph. Furthermore, RSTNet is optimized as ESTNet by the spectrogram information fusion strategy and channel attention mechanism. Using ESTNet, the interpretation accuracy of KN3A, KN6, and KN8 samples is 96%, 94%, and 96%, respectively, and RMSE is 1.5 ℃, 2.1 ℃, 1.8 ℃, respectively. This research provides valuable insight and reference for other spectroscopic applications.

The Analysis of the Expression Form of Ideological and Political Elements in Philosophy Courses Based on Multimedia Information Fusion Strategy

Abstract The application of multimedia technology to teaching in colleges and universities is an inevitable trend in the development of modern education. This technology has changed the traditional teaching mode and provided technical support for the development of modern education. Multimedia technology integrates sound, image, text, animation, and video, and has the remarkable characteristics of human-computer interaction and instant feedback. The application of multimedia technology in philosophy and ideological and political courses in colleges and universities can fully mobilize and stimulate students’ initiative and enthusiasm for learning. This paper aims at the teaching of ideological and political courses in the field of philosophy in colleges and universities, and teachers focus on students to carry out research. Given the problems existing in the application of multimedia technology at present, this paper discusses the effective methods of applying multimedia technology in ideological and political courses in the field of philosophy in colleges and universities. The effectiveness of ideological and political education in the field of philosophy in colleges and universities is a question worthy of continuous discussion. Focusing on the application of multimedia technology, this paper discusses the methods for teachers to use multimedia technology to improve the effect of ideological and political education in the field of philosophy. The research in this paper has positive practical significance and innovative value. Multimedia technology can better express ideological and political elements, which is helpful for the application of ideological and political elements in philosophy classroom teaching, and can bring a better learning experience to the student group.

Moving-Landmark Assisted Distributed Learning Based Decentralized Cooperative Localization (DL-DCL) with Fault Tolerance

This paper considers the problem of cooperative localization of multiple robots under uncertainty, communicating over a partially connected, dynamic communication network and assisted by an agile landmark. Each robot owns an IMU and a relative pose sensing suite, which can get faulty due to system or environmental uncertainty, and therefore exhibit large bias in their estimation output. For the robots to localize accurately under sensor failure and system or environmental uncertainty, a novel Distributed Learning based Decentralized Cooperative Localization (DL-DCL) algorithm is proposed that involves real-time learning of an information fusion strategy by each robot for combining pose estimates from its own sensors as well as from those of its neighboring robots, and utilizing the moving landmark's pose information as a feedback to the learning process. Convergence analysis shows that the learning process converges exponentially under certain reasonable assumptions. Simulations involving sensor failures inducing around 40-60 times increase in the nominal bias show DL-DCL's estimation performance to be approximately 40% better than the well-known covariance-based estimate fusion methods. For the evaluation of DL-DCL's implementability and fault-tolerance capability in practice, a high-fidelity simulation is carried out in Gazebo with ROS2.

Traversability analysis for autonomous driving in complex environment: A LiDAR‐based terrain modeling approach

AbstractFor autonomous driving, traversability analysis is one of the most basic and essential tasks. In this paper, we propose a novel LiDAR‐based terrain modeling approach, which could output stable, complete, and accurate terrain models and traversability analysis results. As terrain is an inherent property of the environment that does not change with different view angles, our approach adopts a multiframe information fusion strategy for terrain modeling. Specifically, a normal distributions transform mapping approach is adopted to accurately model the terrain by fusing information from consecutive LiDAR frames. Then the spatial‐temporal Bayesian generalized kernel inference and bilateral filtering are utilized to promote the stability and completeness of the results while simultaneously retaining the sharp terrain edges. Based on the terrain modeling results, the traversability of each region is obtained by performing geometric connectivity analysis between neighboring terrain regions. Experimental results show that the proposed method could run in real‐time and outperforms state‐of‐the‐art approaches.

A Multisource Domain Adaptation Network for Process Fault Diagnosis Under Different Working Conditions

Transfer learning-based process fault diagnosis has received intensive attention from researchers. However, a practical scenario of process fault diagnosis (i.e., multisource domain adaptation) has not been well solved under various working conditions. It is challenging since distribution difference coexists between different source domains and across source and target domains. In this study, a novel transfer learning model, feature-level and class-level based multi-source domain adaptation (FC-MSDA) is proposed for process fault diagnosis under varying working conditions. A common feature extractor is proposed to learn both global and local features from process signals. A feature selection module is developed to reduce negative transfer caused by irrelevant information in multiple source domains. Domain specific feature generator is developed for each source-target domain pair to learn domain-specific features. Moreover, class-level distribution alignment loss is proposed for each domain pair to settle the negative transfer caused by inconsistent label space between domains from different working conditions of process. An information fusion strategy is performed to ensemble multiple predictions. The experimental results on three industrial cases demonstrate the effectiveness of FC-MSDA in process fault diagnosis (i.e., FC-MSDA obtains the average accuracy of 99.17% on five transfer tasks in three phase process).

Multimodal medical image fusion in NSST domain with structural and spectral features enhancement

For the past 25 years, medical imaging has been extensively used for clinical diagnosis. The main difficulties in medicine are accurate disease recognition and improved therapy. Using a single imaging modality to diagnose disease is challenging for clinical personnel. In this paper, a novel structural and spectral feature enhancement method in NSST Domain for multimodal medical image fusion (MMIF) is proposed. Initially, the proposed method uses the Intensity, Hue, Saturation (IHS) method to generate two pairs of images. The input images are then decomposed using the Non-Subsampled Shearlet Transform (NSST) method to obtain low frequency and high frequency sub-bands. Next, a proposed Structural Information (SI) fusion strategy is employed to Low Frequency Sub-bands (LFS's). It will enhance the structural (texture, background) information. Then, Principal Component Analysis (PCA) is employed as a fusion rule to High Frequency Sub-bands (HFS's) to obtain the pixel level information. Finally, the fused final image is obtained by employing inverse NSST and IHS. The proposed algorithm was validated using different modalities containing 120 image pairs. The qualitative and quantitative results demonstrated that the algorithm proposed in this research work outperformed numerous state-of-the-art MMIF approaches.

Continuous motion finger joint angle estimation utilizing hybrid sEMG-FMG modality driven transformer-based deep learning model

Continuous motion joint angle estimation plays an important role in human–machine interaction (HMI). However, it is still a challenge to estimate continuous motion finger joint angles (CMFJA) accurately. To improve CMFJA estimation accuracy, we used hybrid surface electromyography-force myography (sEMG-FMG) modality as the decoding scheme since combining two sensing modalities could potentially compensate and correct for single sensing modality, and proposed a biosignals driven convolution neural networks (CNN) and Transformer model (BioCNN-T) to estimate CMFJA motivated by the potential of the Transformer architecture, which extracts local features through CNN, and captures dependencies between global features through Transformer encoder. The metacarpophalangeal joint angles of 6 hand movements commonly used in daily life were selected as estimation objects. The experimental results show that CMFJA estimation by hybrid sEMG-FMG modality can achieve higher accuracy than single modality. And we found that the addition of convolution block before Transformer encoder plays a positive role in improving estimation accuracy and pixel-level fusion is the most suitable among the information fusion strategies appearing in the paper. What’s more, compare with the CNN and long short-term memory neural networks (CNN-LSTM), the BioCNN-T has higher accuracy and less computing cost. (Average normalized root mean square error, 0.0456 ± 0.0030 versus 0.0539 ± 0.0035; Inference time, 9.12 × 10-6 s versus 3.20 × 10-4 s). To our knowledge, our work is a pioneer in the use of hybrid sEMG-FMG modality and Transformer architecture to estimate CMFJA. The Implications of it have promising potential in flexible and fine-grained HMI.

Spatio-temporal PV power forecasting considering the time-shift correction and the information fusion strategy of multi-stations

Accurate prediction of PV power is essential to ensuring the safe and economic operation of power systems with high PV penetration. The current PV power prediction scheme considering the spatio-temporal correlation characteristics is relatively simple in data processing, resulting in low prediction accuracy; at the same time, the missing data also poses a great problem to the prediction. Therefore, in order to improve the prediction accuracy and solve the problem of missing data, this paper proposes a PV power spatio-temporal prediction model considering time-shift correction and a multi-station information fusion strategy Firstly, relevant power station clusters are constructed using hierarchical clustering, and a similar daily data filtering model considering the variation characteristics of daily power characteristic curves is proposed to filter the data; Secondly, multiple BP neural network models are constructed and multiple reference power stations with high relevance are predicted using irradiance information; Thirdly, the prediction results of multiple reference power stations are input to the data processing module for time-shift analysis and spatial correlation information fusion correction, which solves the missing data problem of the target power station to be predicted. Finally, it is input to One-dimensional Convolutional Neural Network(1DCNN) to achieve the power prediction of the target power station with missing data. The simulation analysis shows that the root mean square error (RMSE) of a sunny day forecast is 3.31%; the RMSE of a non-sunny day forecast is 9.65%, which proves the accuracy of this two-layer neural network is higher compared to other model structures, so the proposed scheme has certain reliability and accuracy in the prediction of PV power with missing data.

Characteristic evaluation via multi-sensor information fusion strategy for spherical underwater robots

Currently, most of the existing fusion methods ignore the rich multi-source information of different types of sensor nodes in the underwater unknown environment, which makes it challenging for Autonomous Underwater Vehicles (AUVs) to accurately perceive the external environment and make actionable decisions. Considering the key issues such as attitude estimation, positioning and obstacle avoidance involved in performing AUV tasks, this paper proposed a Multi-Source Information Fusion (MSIF) model for Spherical Underwater Robots (SURs) we developed based on various low-cost sensors. Multi-source information from an Inertial Measurement Unit (IMU), Pressure Sensor Array (PSA), Obstacle Avoidance Sensor Array (OASA), Depth Sensor (DS), Looking-Down Camera (LDC) and Acoustic Communication System (ACS) were fused to enable SUR to obtain high-precision estimated data for attitude estimation, positioning and obstacle avoidance, etc. More precisely, according to the correlation between the sensors, the optimized model was constructed to compensate for angle errors, velocity errors, orientation errors, etc. Subsequently, a machine learning method using Back Propagation Neural Network (BPNN) was proposed to improve the accuracy and effectiveness of the MSIF model through feature selection, data training, and feature estimation, etc. Finally, a series of experiments were performed under different scenarios, such as motion and obstacle avoidance experiments. The theoretical derivation and comprehensive evaluations demonstrated the effectiveness and feasibility of the proposed model, which provided a new reference value for solving issues such as attitude estimation, positioning and obstacle avoidance of AUVs.

Semantic-Structure-Aware Multi-Level Information Fusion for Robust Global Orientation Optimization of Autonomous Mobile Robots.

Multi-camera-based simultaneous localization and mapping (SLAM) has been widely applied in various mobile robots under uncertain or unknown environments to accomplish tasks autonomously. However, the conventional purely data-driven feature extraction methods cannot utilize the rich semantic information in the environment, which leads to the performance of the SLAM system being susceptible to various interferences. In this work, we present a semantic-aware multi-level information fusion scheme for robust global orientation estimation. Specifically, a visual semantic perception system based on the synthesized surround view image is proposed for the multi-eye surround vision system widely used in mobile robots, which is used to obtain the visual semantic information required for SLAM tasks. The original multi-eye image was first transformed to the synthesized surround view image, and the passable space was extracted with the help of the semantic segmentation network model as a mask for feature extraction; moreover, the hybrid edge information was extracted to effectively eliminate the distorted edges by further using the distortion characteristics of the reverse perspective projection process. Then, the hybrid semantic information was used for robust global orientation estimation; thus, better localization performance was obtained. The experiments on an intelligent vehicle, which was used for automated valet parking both in indoor and outdoor scenes, showed that the proposed hybrid multi-level information fusion method achieved at least a 10-percent improvement in comparison with other edge segmentation methods, the average orientation estimation error being between 1 and 2 degrees, much smaller than other methods, and the trajectory drift value of the proposed method was much smaller than that of other methods.

Multi-objective particle swarm optimization with reverse multi-leaders.

Despite being easy to implement and having fast convergence speed, balancing the convergence and diversity of multi-objective particle swarm optimization (MOPSO) needs to be further improved. A multi-objective particle swarm optimization with reverse multi-leaders (RMMOPSO) is proposed as a solution to the aforementioned issue. First, the convergence strategy of global ranking and the diversity strategy of mean angular distance are proposed, which are used to update the convergence archive and the diversity archive, respectively, to improve the convergence and diversity of solutions in the archives. Second, a reverse selection method is proposed to select two global leaders for the particles in the population. This is conducive to selecting appropriate learning samples for each particle and leading the particles to quickly fly to the true Pareto front. Third, an information fusion strategy is proposed to update the personal best, to improve convergence of the algorithm. At the same time, in order to achieve a better balance between convergence and diversity, a new particle velocity updating method is proposed. With this, two global leaders cooperate to guide the flight of particles in the population, which is conducive to promoting the exchange of social information. Finally, RMMOPSO is simulated with several state-of-the-art MOPSOs and multi-objective evolutionary algorithms (MOEAs) on 22 benchmark problems. The experimental results show that RMMOPSO has better comprehensive performance.

Multisource Information Fusion Network for Optical Remote Sensing Image Super-Resolution

The super-resolution algorithms based on deep learning can effectively increase optical remote sensing image (ORSI) details for further analysis tasks. Deep unfolding methods have been studied in recent years to bridge the gap between optimization-based and learning-based methods, which can take good advantage of the prior knowledge. However, these unfolding methods usually ignore the utilization of intermediate network features between different iteration stages, thereby limiting the performance of super-resolution results. We propose a multi-source information fusion network (MSFNet) for ORSI super-resolution to address this problem. We mainly consider three strategies to enhance the image super-resolution performance, including feature extraction strategy, information fusion strategy, and the structure of the unfolding network. Firstly, image information of various scales is helpful for mining potential features of images for image super-resolution. Therefore, we introduce multi-scale implicit constraints to the objective function. Secondly, we unfold the optimization process into a neural network by alternating direction method of multipliers (ADMM). The network framework follows the ADMM method's iteration process. This unfolding strategy can effectively utilize the prior information for image reconstruction. Thirdly, we propose a row-column decoupling Transformer module based on this unfolding framework for feature fusion. Specifically, the row Transformer block completes the feature fusion of various scales, and the column Transformer block completes the feature fusion of various channels. The fused features are transmitted to the next iteration stage for feature enhancement. And this fusion strategy enables the network to focus on the global information of the features. We perform experiments on three remote sensing image datasets to fully demonstrate the algorithm's effectiveness. Experiment results show that the proposed algorithm can achieve better image reconstruction performance.

Dynamic operation optimization based on improved dynamic multi-objective dragonfly algorithm in continuous annealing process

The continuous annealing production process is the strip iron heat treatment process. Considering the numerous disturbance parameters and frequent environmental changes in the production process, dynamic multi-objective optimization is studied. (1)To anticipate whether the environment will change, this paper proposed dynamic detection based on Long-Short Term Memory (LSTM) prediction mechanism. A single-step prediction method is adopted to predict the strip hardness at the next moment which is compared with the value at the previous moment. (2)To track the dynamic Pareto frontier, the Information Fusion (IF) strategy in Dynamic Multi-Objective Dragonfly Algorithm (DMODA) is proposed. The population is updated by integrating the environmental offset with the evolution information. Computational experiments show the proposed strategy is effective when dealing with dynamic problems and it has been well applied in the continuous annealing process. The strip steel quality and production capacity can remain stable during the dynamic production process.

A Cooperative Relative Localization System for Distributed Multi-Agent Networks

While absolute position information is inaccessible for multi-agent networks due to the lack of global reference in GNSS-challenged or infrastructure-free scenarios, relative pose information among the agents can be obtained via agent cooperation. In this paper, we develop a cooperative relative localization system for distributed multi-agent networks. In particular, we propose a node activation strategy to select the backbone agents that minimize the information loss caused by malformed topology, and then the listener agents localize themselves based on the backbone topology. The scheme also incorporates a back calibration step that utilizes the position estimates of the listener agents to further improve the localization accuracy of the backbone topology. Moreover, a multi-sensory information fusion scheme is developed for dynamic scenes, where a coordinate transformation method is proposed to reduce the accumulated error caused by the coordinate misalignment. Finally, the system is implemented on a low-cost hardware platform with the ultra-wideband radios, and extensive simulations and experiments demonstrate that the proposed system can achieve decimeter-level relative localization accuracy.

Research on Road Condition Sensing Technology based on Vision and Radar Information

In the process of intelligent vehicle driving, due to the complexity of the environment, a single sensor or multiple homogeneous sensors cannot completely perceive the traffic environment around the intelligent vehicle. Therefore, it is necessary to study the information fusion scheme of different sensors and make use of the advantages of each sensor to make up for the deficiency of a single sensor, so as to realize the function of cooperation and mutual compensation between multiple sensors. In this paper, millimeter-wave radar and camera are selected as sensors for an intelligent vehicle to sense the environment ahead. Firstly, in the case of time and space synchronization of the two types of signals, fusion based on the cross-ratio of detection frames is used to find the region of common interest of radar and camera for fusion. Then the fusion based on data association is adopted. The experimental results show that the method based on radar and camera data can effectively realize accurate target recognition and location, and can meet the requirements of real-time accuracy in the application of intelligent vehicles.

Multi-scale information fusion network with label smoothing strategy for corneal ulcer classification in slit lamp images.

Corneal ulcer is the most common symptom of corneal disease, which is one of the main causes of corneal blindness. The accurate classification of corneal ulcer has important clinical importance for the diagnosis and treatment of the disease. To achieve this, we propose a deep learning method based on multi-scale information fusion and label smoothing strategy. Firstly, the proposed method utilizes the densely connected network (DenseNet121) as backbone for feature extraction. Secondly, to fully integrate the shallow local information and the deep global information and improve the classification accuracy, we develop a multi-scale information fusion network (MIF-Net), which uses multi-scale information for joint learning. Finally, to reduce the influence of the inter-class similarity and intra-class diversity on the feature representation, the learning strategy of label smoothing is introduced. Compared with other state-of-the-art classification networks, the proposed MIF-Net with label smoothing achieves high classification performance, which reaches 87.07 and 83.84% for weighted-average recall (W_R) on the general ulcer pattern and specific ulcer pattern, respectively. The proposed method holds promise for corneal ulcer classification in fluorescein staining slit lamp images, which can assist ophthalmologists in the objective and accurate diagnosis of corneal ulcer.

Research on the Application of Multi-Source Information Fusion in Multiple Gait Pattern Transition Recognition.

Multi-source information fusion technology is a kind of information processing technology which comprehensively processes and utilizes multi-source uncertain information. It is an effective scheme to solve complex pattern recognition and improve classification performance. This study aims to improve the accuracy and robustness of exoskeleton gait pattern transition recognition in complex environments. Based on the theory of multi-source information fusion, this paper explored a multi-source information fusion model for exoskeleton gait pattern transition recognition in terms of two aspects of multi-source information fusion strategy and multi-classifier fusion. For eight common gait pattern transitions (between level and stair walking and between level and ramp walking), we proposed a hybrid fusion strategy of multi-source information at the feature level and decision level. We first selected an optimal feature subset through correlation feature extraction and feature selection algorithm, followed by the feature fusion through the classifier. We then studied the construction of a multi-classifier fusion model with a focus on the selection of base classifier and multi-classifier fusion algorithm. By analyzing the classification performance and robustness of the multi-classifier fusion model integrating multiple classifier combinations with a number of multi-classifier fusion algorithms, we finally constructed a multi-classifier fusion model based on D-S evidence theory and the combination of three SVM classifiers with different kernel functions (linear, RBF, polynomial). Such multi-source information fusion model improved the anti-interference and fault tolerance of the model through the hybrid fusion strategy of feature level and decision level and had higher accuracy and robustness in the gait pattern transition recognition, whose average recognition accuracy for eight gait pattern transitions reached 99.70%, which increased by 0.15% compared with the highest average recognition accuracy of the single classifier. Moreover, the average recognition accuracy in the absence of different feature data reached 97.47% with good robustness.

What happens next? Combining enhanced multilevel script learning and dual fusion strategies for script event prediction

Script event prediction (SEP), aiming at predicting next event from context event sequences (i.e., scripts), has played an important role in many real-world applications such as government decision-making. While most of the existing research only depend on the top-level event prediction, they ignore the influence of other bottom levels or other relationship modeling manners. In this paper, we focus on the problem of SEP via multilevel script learning where the goal of is to explore a multistage, multiprediction and multilevel information fusion model for SEP. This is challenging in (1) simultaneously modeling of the multilevel event relationship semantic information and (2) effectively designing multilevel information fusion strategies. In this paper, we propose a new script event prediction model based on Enhanced Multilevel script learning and Dual Fusion strategies, named EMDF-Net. Specifically, EMDF-Net designs the multilevel (event/chain/segment level) script learning to model both temporal and casual information as well as the rich structural relevance via neural stacking of self-attention mechanism and graph neural networks. Then it proposes dual fusion strategies to fully integrate different-level information by nonlinear feature composition and weighted score fusion. Finally, a deep supervision strategy is utilized to end-to-end train the whole model and provide a good initialization for information fusion. Experimental results on the popular NYT corpus demonstrate the effectiveness and superiority of EMDF-Net.

Distributed State Estimation for Continuous-Time Linear Systems With Correlated Measurement Noise

In this article, the problem of distributed state estimation for a continuous-time linear system with a sensor network is investigated, where each sensor can only communicate with its neighbors and contains time-correlated measurement noise. To solve this problem, a novel augmented leader-following information fusion strategy is first proposed to collect measurements and system matrices. Then, a class of distributed state estimators is developed with bounded estimation error covariances. Further, a closed-form relation between the designed distributed estimator and the centralized estimator is established. It is found that the estimation performance of the former converges to that of the latter when the consensus gain tends to infinity. The proposed estimator is further extended to the fully distributed case by introducing an adaptive law for the consensus gain without using any global information. Moreover, it is shown that the designed estimator is applicable for systems with deterministic noise. Finally, several comparative numerical simulations are provided to demonstrate the effectiveness and superiority of the theoretical results.

Person re-identification in indoor videos by information fusion using Graph Convolutional Networks

Information contained in visual appearance and gait features can play an important role in designing computer vision assisted person re-identification (ReID) systems. Fusion of appearance and gait features has not yet been tested in this domain of research despite its great potential to solve some of the intriguing challenges faced by the research community due to viewpoint variations, illumination change, varying recording setups, etc. This paper proposes a new deep learning framework for person re-identification in videos. The framework referred to as Fused Graph Network (FGN-ReID) uses an information fusion strategy to deal with the aforementioned variations. We have used visual appearance and gait features independently and fused these features for re-identification of persons in close shot indoor video recordings. A feature similarity-based score-level fusion strategy has been adopted to fuse spatio-temporal and gait features using Graph Convolutional Networks (GCN). The ReID problem has been mapped to an inherent graph searching problem. In our proposed framework, appearance and gait features represent the nodes. And the relations between spatio-temporal segments denote the edges. Experiments using CASIA-B dataset reveal that the proposed method is more effective as compared to existing methods. We have improved the rank-1 accuracy by a margin of 8%–12% as compared to the baseline algorithms. The mean average precision (mAP) has improved significantly. The method has several applications including visual surveillance, biometric authentication, etc.