Network Information Fusion Research Articles

Optimal Linear Filter Based on Feedback Structure for Sensing Network with Correlated Noises and Data Packet Dropout.

This paper is concerned with the estimation of correlated noise and packet dropout for information fusion in distributed sensing networks. By studying the problem of the correlation of correlated noise in sensor network information fusion, a matrix weight fusion method with a feedback structure is proposed to deal with the interrelationship between multi-sensor measurement noise and estimation noise, and the method can achieve optimal estimation in the sense of linear minimum variance. Based on this, a method is proposed using a predictor with a feedback structure to compensate for the current state quantity to deal with packet dropout that occurs during multi-sensor information fusion, which can reduce the covariance of the fusion results. Simulation results show that the algorithm can solve the problem of information fusion noise correlation and packet dropout in sensor networks, and effectively reduce the fusion covariance with feedback.

A thermal load identification method based on physics-guided neural network for honeycomb sandwich structures

The identification of thermal load/thermal shock of aircraft during service is beneficial for collecting information of the service environment and avoiding risks. In the paper, a method based on multivariate information fusion and physics-guided neural network is developed for the inverse problem of thermal load identification of honeycomb sandwich structures. Two thermal feature parameters: temperature gradient and temperature variation rate are used to build the dataset. A 16-layers physics-guided neural network is presented to achieve the predicted results consistent with physical knowledge. In the work, laser irradiation is used as the thermal load, and two laser parameters are to be identified, i.e. spot diameter, power. Simulations and experiments are conducted to verify the effectiveness of the proposed method. The effects of physics-guided loss function and multivariate information fusion are discussed, and it is found that the results based on the proposed method are much better than the results based on the method without physical model. Besides, results based on multivariate information fusion are better than results based on single temperature response. Then, the effects of network models and hyper parameters on the proposed method are also discussed.

Single-cell RNA-seq data clustering by deep information fusion.

Determining cell types by single-cell transcriptomics data is fundamental for downstream analysis. However, cell clustering and data imputation still face the computation challenges, due to the high dropout rate, sparsity and dimensionality of single-cell data. Although some deep learning based solutions have been proposed to handle these challenges, they still can not leverage gene attribute information and cell topology in a sensible way to explore the consistent clustering. In this paper, we present scDeepFC, a deep information fusion-based single-cell data clustering method for cell clustering and data imputation. Specifically, scDeepFC uses a deep auto-encoder (DAE) network and a deep graph convolution network to embed high-dimensional gene attribute information and high-order cell-cell topological information into different low-dimensional representations, and then fuses them to generate a more comprehensive and accurate consensus representation via a deep information fusion network. In addition, scDeepFC integrates the zero-inflated negative binomial (ZINB) into DAE to model the dropout events. By jointly optimizing the ZINB loss and cell graph reconstruction loss, scDeepFC generates a salient embedding representation for clustering cells and imputing missing data. Extensive experiments on real single-cell datasets prove that scDeepFC outperforms other popular single-cell analysis methods. Both the gene attribute and cell topology information can improve the cell clustering.

A novel approach based on fault tree analysis and Bayesian network for multi-state reliability analysis of complex equipment systems

Due to the complex structure of multi-state complex systems and the lack of data, information, and knowledge, the uncertainty of the logical relationship between the failure states of systems and components and the uncertainty of related failure data become the key issues in the reliability analysis of multi-state complex systems. In this paper, combined with multi-state fault tree (MSFT), a multi-state reliability assessment framework for complex systems considering uncertainty based on multi-source information fusion and multi-state Bayesian network (MSBN) is proposed. The multi-source information fusion method combines historical data and experts’ opinions to solve the uncertainty problem of multi-state failure data in complex equipment systems effectively. Based on the multi-source information fusion method, the calculation method of multi-state prior probability and the construction method of conditional probability are given. By constructing the conditional probability table (CPT), the uncertain logic relationship between the multi-state nodes is effectively expressed, which effectively improves the efficiency of CPT acquisition for MSBN and reduces the workload of experts scoring. Finally, a mud circulating system is taken as an example to prove the proposed method, and the specific calculation process, evaluation results, and some discussions are given. The results show that the proposed method is an effective multi-state reliability analysis method for complex uncertain multi-state systems.

SANet: A novel segmented attention mechanism and multi-level information fusion network for 6D object pose estimation

Reliably and rapidly estimating the 6D position of an object is a critical challenge when using Internet of Things (IoT) technologies for monitoring. Nowadays, the prevalent 6D pose estimation architecture is based on a two-stage technique, which requires a significant amount of time for both training and deploying the algorithm in actual applications. Additionally, the majority of approaches include intricate high-low level features in the network that greatly influence training but contribute less to testing. To enable more accurate 6D object pose estimation while shortening the deployment time, we used a single-stage end-to-end algorithm to design the network. In this paper, we propose SANet, which is composed of a segmented attention module and a multi-level information fusion module. Specifically, by extracting high-level semantic information from images before fusing them to the decoder, and by removing redundant information using the multi-level information fusion module, the feature fusion complexity of the model is reduced by extracting high level features. In addition, the segmented attention module can suppress unreliable information to enhance network learning of channel and spatial information, enabling the network to more accurately understand the geometric aspects of the object. Extensive experiments on LM and LMO datasets demonstrate that our method outperforms state-of-the-art baselines, ranking 1st in both speed and accuracy.

MMST: A Multi-Modal Ground-Based Cloud Image Classification Method.

In recent years, convolutional neural networks have been in the leading position for ground-based cloud image classification tasks. However, this approach introduces too much inductive bias, fails to perform global modeling, and gradually tends to saturate the performance effect of convolutional neural network models as the amount of data increases. In this paper, we propose a novel method for ground-based cloud image recognition based on the multi-modal Swin Transformer (MMST), which discards the idea of using convolution to extract visual features and mainly consists of an attention mechanism module and linear layers. The Swin Transformer, the visual backbone network of MMST, enables the model to achieve better performance in downstream tasks through pre-trained weights obtained from the large-scale dataset ImageNet and can significantly shorten the transfer learning time. At the same time, the multi-modal information fusion network uses multiple linear layers and a residual structure to thoroughly learn multi-modal features, further improving the model's performance. MMST is evaluated on the multi-modal ground-based cloud public data set MGCD. Compared with the state-of-art methods, the classification accuracy rate reaches 91.30%, which verifies its validity in ground-based cloud image classification and proves that in ground-based cloud image recognition, models based on the Transformer architecture can also achieve better results.

Automatic Depression Detection in Social Networks Using Multiple User Characterizations

Depression is rapidly becoming one of the most common illnesses worldwide, currently affecting a significant number of people. These people may show different signs of depression depending on a number of characteristics (e.g., age, sex, personality, mood, etc.). Experts often use these signs to diagnose and monitor depression. However, due to the difficulty of obtaining this information through traditional methods, the use of social networks to characterize users has proven to be a valuable resource. In this paper, we study the potential of various user characterizations for the task of automatic depression detection. We consider two social networks and a variety of models for fusing the characterizations. In particular, we propose the use of a range of networks that learn to weight the contribution of each characterization from the data. We show that, using this model, the depression detection performance outperforms the state-of-the-art results on the two data sets considered. In addition, we present interesting findings on the correlation of the characterizations considered in the information fusion network.

ReCoMIF: Reading comprehension based multi-source information fusion network for Chinese spoken language understanding

Spoken language understanding (SLU) plays a crucial role in the performance of dialogue systems. It usually includes slot filling and intent detection (SFID) tasks aiming at semantic parsing of utterances. At present, researchers focus mainly on English SLU tasks, while such investigations on Chinese utterances are not sufficient. In this paper, we first propose a reading comprehension based multi-source information fusion network, called ReCoMIF for Chinese SFID tasks by transforming the SLU task into a multi-turn question answering procedure comprising multiple choice for intent detection and span extraction for slot filling. Moreover, we present a TCM_CLS module with a concise architecture composed of TextCNN, MaxPooling, and feed forward network plus the [CLS] representation. Such three TCM_CLS modules are stacked in the proposed ReCoMIF network that can serve as sufficient integration of multi-source information originating from contexts, reading comprehension based queries, and hidden representations concerning intent and slot semantics. Finally, experimental results and ablation studies on three Chinese SLU datasets show that our proposed model can effectively fuse intent and slot information achieved by\\ state-of-the-art performance compared with other baseline methods.

Research on Texture Classification Based on Multi-Scale Information Fusion

Texture feature is an important visual cue for an image, which is the unified description of human visual attributes and sensory attributes. The inherent problem of texture image is that the difference of intra-class images is large and the disparity of inter-class images is small. This problem increases the difficulty of texture image recognition. Therefore, improving the relevance embedding of intra-class images can reduce the classification errors caused by this problem. To solve this problem, this paper proposes a multi-scale information fusion network algorithm, which adopts a cascade structure. It combines multi-scale feature information with the corresponding background information. The shallow background information guides the next stage of feature formation and enhances the similarity of intra-class images. The intra-class feature information obtained is more general. The algorithm has been tested on data sets describable texture database (DTD) and Flickr material dataset (FMD), which has achieved good results.

Pred-SF: A Precipitation Prediction Model Based on Deep Neural Networks.

How to predict precipitation accurately and efficiently is the key and difficult problem in the field of weather forecasting. At present, we can obtain accurate meteorological data through many high-precision weather sensors and use them to forecast precipitation. However, the common numerical weather forecasting methods and radar echo extrapolation methods have insurmountable defects. Based on some common characteristics of meteorological data, this paper proposes a Pred-SF model for precipitation prediction in target areas. The model focuses on the combination of multiple meteorological modal data to carry out self-cyclic prediction and a step-by-step prediction structure. The model divides the precipitation prediction into two steps. In the first step, the spatial encoding structure and PredRNN-V2 network are used to construct the autoregressive spatio-temporal prediction network for the multi-modal data, and the preliminary predicted value of the multi-modal data is generated frame by frame. In the second step, the spatial information fusion network is used to further extract and fuse the spatial characteristics of the preliminary predicted value and, finally, output the predicted precipitation value of the target region. In this paper, ERA5 multi-meteorological mode data and GPM precipitation measurement data are used for testing to predict the continuous precipitation of a specific area for 4 h. The experimental results show that Pred-SF has strong precipitation prediction ability. Some comparative experiments were also set up for comparison to demonstrate the advantages of the combined prediction method of multi-modal data and the stepwise prediction method of Pred-SF.

IEF-CSNET: Information Enhancement and Fusion Network for Compressed Sensing Reconstruction.

The rapidly growing requirement for data has put forward Compressed Sensing (CS) to realize low-ratio sampling and to reconstruct complete signals. With the intensive development of Deep Neural Network (DNN) methods, performance in image reconstruction from CS measurements is constantly increasing. Currently, many network structures pay less attention to the relevance of before- and after-stage results and fail to make full use of relevant information in the compressed domain to achieve interblock information fusion and a great receptive field. Additionally, due to multiple resamplings and several forced compressions of information flow, information loss and network structure redundancy inevitably result. Therefore, an Information Enhancement and Fusion Network for CS reconstruction (IEF-CSNET) is proposed in this work, and a Compressed Information Extension (CIE) module is designed to fuse the compressed information in the compressed domain and greatly expand the receptive field. The Error Comprehensive Consideration Enhancement (ECCE) module enhances the error image by incorporating the previous recovered error so that the interlink among the iterations can be utilized for better recovery. In addition, an Iterative Information Flow Enhancement (IIFE) module is further proposed to complete the progressive recovery with loss-less information transmission during the iteration. In summary, the proposed method achieves the best effect, exhibits high robustness at this stage, with the peak signal-to-noise ratio (PSNR) improved by 0.59 dB on average under all test sets and sampling rates, and presents a greatly improved speed compared with the best algorithm.

CIFNet: context information fusion network for cloud and cloud shadow detection in optical remote sensing imagery

Accurately and automatically detecting cloud and cloud shadow is one of the key steps in the analysis of optical remote sensing imagery. Currently, most cloud and cloud shadow detection methods are prone to false detection, and some clouds and cloud shadows may be missing from the detection results. Due to the lack of contextual information extraction and fusion capabilities, the accuracy of these cloud detection algorithms cannot be guaranteed. This article proposes a deep learning-based convolutional neural network context information fusion network (named CIFNet) to solve the problem of cloud and cloud shadow detection on Gaofen-1 wide field of view optical remote sensing imageries. First, we preprocess the dataset and make the corresponding labels. Next, we design a Resblock-cloud to capture global and local features and prevent the network from degrading. Then, we utilize a global context fusion module to fuse different levels of global context information by dense skip-connections. Finally, a multiscale context fusion module is designed to extract multiscale contextual relations between cloud and cloud shadow. Experimental results show that the proposed CIFNet method can obtain better cloud and cloud shadow boundaries in complex situations and therefore outperforms the competing methods.

An Analysis Model of Learners’ Online Learning Status Based on Deep Neural Network and Multi-Dimensional Information Fusion

The learning status of learners directly affects the quality of learning. Compared with offline teachers, it is difficult for online teachers to capture the learning status of students in the whole class, and it is even more difficult to continue to pay attention to students while teaching. Therefore, this paper proposes an online learning state analysis model based on a convolutional neural network and multi-dimensional information fusion. Specifically, a facial expression recognition model and an eye state recognition model are constructed to detect students’ emotions and fatigue, respectively. By integrating the detected data with the homework test score data after online learning, an analysis model of students’ online learning status is constructed. According to the PAD model, the learning state is expressed as three dimensions of students’ understanding, engagement and interest, and then analyzed from multiple perspectives. Finally, the proposed model is applied to actual teaching, and procedural analysis of 5 different types of online classroom learners is carried out, and the validity of the model is verified by comparing with the results of the manual analysis.

Scattering Information Fusion Network for Oriented Ship Detection in SAR Images

Scattering Information Fusion Network for Oriented Ship Detection in SAR Images

Fast and Robust Online Handwritten Chinese Character Recognition With Deep Spatial and Contextual Information Fusion Network

Deep convolutional neural networks have achieved fairly high accuracy for single online handwritten Chinese character recognition (SOLHCCR). However, in real application scenarios, users always write multiple characters to form a complete sentence, and previous contextual information holds significant potential for improving the accuracy, robustness and efficiency of recognition. In this work, we first propose a simple and straightforward model named the vanilla compositional network (VCN) by coupling convolutional neural network with a sequence modeling architecture (i.e., a recurrent neural network or Transformer), which exploits the handwritten character's previous contextual information. Although VCN performs much better than the previous state-of-the-art SOLHCCR models, it is a two-stage architecture in nature. It suffers from high fragility when confronting with poorly written characters such as sloppy writing, and missing or broken strokes, due to relying heavily on contextual information. To improve the robustness of the OLHCCR model, we further propose a novel deep spatial & contextual information fusion network (DSCIFN). It utilizes an autoregresssive framework pre-trained on a large-scale sentence corpora as the backbone component, and highly integrates the spatial features of handwritten characters and their previous contextual information in a multi-layer fusion module. To verify the effectiveness of models, we reorganize a new form of online Chinese handwritten character with its previous context dataset, named OHCCC. Extensive experimental results demonstrate that DSCIFN achieves state-of-the-art performance and has increased strong robustness compared to VCN and previous SOLHCCR models. The in-depth empirical analysis and case study indicate that DSCIFN can significantly improve the efficiency of handwriting input because it does not need complete strokes to recognize a handwritten Chinese character precisely.

Multisource Information Fusion Network for Optical Remote Sensing Image Super-Resolution

The super-resolution algorithms based on deep learning can effectively increase optical remote sensing image (ORSI) details for further analysis tasks. Deep unfolding methods have been studied in recent years to bridge the gap between optimization-based and learning-based methods, which can take good advantage of the prior knowledge. However, these unfolding methods usually ignore the utilization of intermediate network features between different iteration stages, thereby limiting the performance of super-resolution results. We propose a multi-source information fusion network (MSFNet) for ORSI super-resolution to address this problem. We mainly consider three strategies to enhance the image super-resolution performance, including feature extraction strategy, information fusion strategy, and the structure of the unfolding network. Firstly, image information of various scales is helpful for mining potential features of images for image super-resolution. Therefore, we introduce multi-scale implicit constraints to the objective function. Secondly, we unfold the optimization process into a neural network by alternating direction method of multipliers (ADMM). The network framework follows the ADMM method's iteration process. This unfolding strategy can effectively utilize the prior information for image reconstruction. Thirdly, we propose a row-column decoupling Transformer module based on this unfolding framework for feature fusion. Specifically, the row Transformer block completes the feature fusion of various scales, and the column Transformer block completes the feature fusion of various channels. The fused features are transmitted to the next iteration stage for feature enhancement. And this fusion strategy enables the network to focus on the global information of the features. We perform experiments on three remote sensing image datasets to fully demonstrate the algorithm's effectiveness. Experiment results show that the proposed algorithm can achieve better image reconstruction performance.

Substation equipment temperature prediction based on multivariate information fusion and deep learning network.

Substation equipment temperature is difficult to achieve accurate prediction because of its typical seasonality, periodicity and instability, complex working environment and less available characteristic information. To overcome these difficulties, a substation equipment temperature prediction method is proposed based on multivariate information fusion, convolutional neural network (CNN) and gated recurrent unite (GRU) in this article. Firstly, according to the correlation analysis including linear correlation mapping, autocorrelation function and partial autocorrelation function for substation equipment temperature data, the feature vectors from ambient, time and space are determined, that is the multivariate information fusion feature vector (denoted as MIFFV); secondly, the dimension of MIFFV is reduced by principal component analysis (PCA), extract some of the most important features and form the reduced feature vector (denoted as RFV); then, CNN is used for deep learning to extract the relationship between RFV and the high-dimensional space feature, and construct the high-dimensional feature vector of multivariate time series (denoted as HDFV); finally, the high-dimensional feature vector is used to train GRU deep learning network and predict the equipment temperature. A substation equipment in Taizhou City, Zhejiang Province is conducted by the method proposed in this article. Through the comparative experiment from the two aspects of features and methods, under the two prediction performance evaluation indexes of mean absolute percentage error (MAPE) and root mean square error (RSME), two main conclusions are drawn: (1) MIFFV from three aspects of ambient features, time features and space features have better prediction performance than the single feature vector and the combined feature vector of two aspects; (2) compared with other four related models under the same conditions, RFV is regarded as the input of the models, the proposed model has better prediction performance.

H2-Stereo: High-Speed, High-Resolution Stereoscopic Video System

High-speed, high-resolution stereoscopic (H2-Stereo) video allows us to perceive dynamic 3D content at fine granularity. The acquisition of H2-Stereo video, however, remains challenging with commodity cameras. Existing spatial super-resolution or temporal frame interpolation methods provide compromised solutions that lack temporal or spatial details, respectively. To alleviate this problem, we propose a dual camera system, in which one camera captures high-spatial-resolution low-frame-rate (HSR-LFR) videos with rich spatial details, and the other captures low-spatial-resolution high-frame-rate (LSR-HFR) videos with smooth temporal details. We then devise a Learned Information Fusion network (LIFnet) that exploits the cross-camera redundancies to enhance both camera views to high spatiotemporal resolution (HSTR) for reconstructing the H2-Stereo video effectively. We utilize a disparity network to transfer spatiotemporal information across views even in large disparity scenes, based on which, we propose disparity-guided flow-based warping for LSR-HFR view and complementary warping for HSR-LFR view. A multi-scale fusion method in feature domain is proposed to minimize occlusion-induced warping ghosts and holes in HSR-LFR view. The LIFnet is trained in an end-to-end manner using our collected high-quality Stereo Video dataset from YouTube. Extensive experiments demonstrate that our model outperforms existing state-of-the-art methods for both views on synthetic data and camera-captured real data with large disparity. Ablation studies explore various aspects, including spatiotemporal resolution, camera baseline, camera desynchronization, long/short exposures and applications, of our system to fully understand its capability for potential applications.

Multi-scale information fusion network with label smoothing strategy for corneal ulcer classification in slit lamp images.

Corneal ulcer is the most common symptom of corneal disease, which is one of the main causes of corneal blindness. The accurate classification of corneal ulcer has important clinical importance for the diagnosis and treatment of the disease. To achieve this, we propose a deep learning method based on multi-scale information fusion and label smoothing strategy. Firstly, the proposed method utilizes the densely connected network (DenseNet121) as backbone for feature extraction. Secondly, to fully integrate the shallow local information and the deep global information and improve the classification accuracy, we develop a multi-scale information fusion network (MIF-Net), which uses multi-scale information for joint learning. Finally, to reduce the influence of the inter-class similarity and intra-class diversity on the feature representation, the learning strategy of label smoothing is introduced. Compared with other state-of-the-art classification networks, the proposed MIF-Net with label smoothing achieves high classification performance, which reaches 87.07 and 83.84% for weighted-average recall (W_R) on the general ulcer pattern and specific ulcer pattern, respectively. The proposed method holds promise for corneal ulcer classification in fluorescein staining slit lamp images, which can assist ophthalmologists in the objective and accurate diagnosis of corneal ulcer.

Rolling bearing fault diagnosis based on information fusion and parallel lightweight convolutional network

With the development of technologies such as Internet of Things and big data, the realization of fusion and cross analysis of multi-sensor signals provides the possibility for comprehensive condition monitoring and intelligent fault diagnosis of rolling bearings. Considering the bottleneck that deep neural networks with complex parameters and a single convolution kernel style may cause high computational effort and information loss, an intelligent fault diagnosis method based on information fusion and parallel lightweight convolutional network is proposed. Firstly, a normalized pulse energy kurtosis weighted rule is constructed to enable the fusion of multi-channel vibration signals. What’s more, a novel lightweight convolutional neural network is designed to achieve feature extraction and classification. Finally, a novel nonlinear piecewise activation function is introduced to further improve nonlinear learning ability. The effectiveness and superiority are verified by bearing data sets with different speeds and loads. Compared with other models, our diagnostic framework has better effect and provides a new idea for intelligent fault diagnosis.