Proposed Fusion Scheme Research Articles

Heterogeneous Multi-Sensor Fusion With Random Finite Set Multi-Object Densities

This paper addresses the density based multi-sensor cooperative fusion using random finite set (RFS) type multi-object densities (MODs). Existing fusion methods use scalar weights to characterize the relative information confidence among the local MODs, and in this way the portion of contribution of each local MOD to the fused global MOD can be tuned via adjusting these weights. Our analysis shows that the fusion mechanism of using a scalar coefficient can be oversimplified for practical scenarios, as the information confidence of an MOD is complex and usually space-varying due to the imperfection of sensor ability and the various impacts from surveillance environment. Consequently, severe fusion performance degradation can be observed when these scalar weights fail to reflect the actual situation. We make two contributions towards addressing this problem. Firstly, we propose a novel heterogeneous fusion method to perform the information averaging among local RFS MODs. By factorizing each local MODs into a number of smaller size sub-MODs, it can transform the original complicated fusion problem into a much easier parallelizable multi-cluster fusion problem. Secondly, as the proposed fusion strategy is a general procedure without any particular model assumptions, we further derive the detailed heterogeneous fusion equations, with centralized network architecture, for both the probability hypothesis density (PHD) filter and the multi-Bernoulli (MB) filter. The Gaussian mixture implementations of the proposed fusion algorithms are also presented. Various numerical experiments are designed to demonstrate the efficacy of the proposed fusion methods.

Fast and robust identification of railway track stiffness from simple field measurement

We propose to combine a physics-based finite element (FE) track model and a data-driven Gaussian process regression (GPR) model to directly infer railpad and ballast stiffness from measured frequency response functions (FRF) by field hammer tests. Conventionally, only the rail resonance and full track resonance are used as the FRF features to identify track stiffness. In this paper, eleven features, including sleeper resonances, from a single FRF curve are selected as the predictors of the GPR. To deal with incomplete measurements and uncertainties in the FRF features, we train multiple candidate GPR models with different features, kernels and training sets. Predictions by the candidate models are fused using a weighted Product of Experts method that automatically filters out unreliable predictions. We compare the performance of the proposed method with a model updating method using the particle swam optimization (PSO) on two synthesis datasets in a wide range of scenarios. The results show that the enriched features and the proposed fusion strategy can effectively reduce prediction errors. In the worst-case scenario with only three features and 5% injected noise, the average prediction errors for the railpad and ballast stiffness are approximately 12% and 6%, outperforming the PSO by about 6% and 3%, respectively. Moreover, the method enables fast predictions for large datasets. The predictions for 400 samples takes only approximately 10 s compared with 40 min using the PSO. Finally, a field application example shows that the proposed method is capable of extracting the stiffness values using a simple setup, i.e., with only one accelerometer and one impact location.

NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models

In this paper we propose a novel method for infrared and visible image fusion where we develop nest connection-based network and spatial/channel attention models. The nest connection-based network can preserve significant amounts of information from input data in a multi-scale perspective. The approach comprises three key elements: encoder, fusion strategy and decoder respectively. In our proposed fusion strategy, spatial attention models and channel attention models are developed that describe the importance of each spatial position and of each channel with deep features. Firstly, the source images are fed into the encoder to extract multi-scale deep features. The novel fusion strategy is then developed to fuse these features for each scale. Finally, the fused image is reconstructed by the nest connection-based decoder. Experiments are performed on publicly available datasets. These exhibit that our proposed approach has better fusion performance than other state-of-the-art methods. This claim is justified through both subjective and objective evaluation. The code of our fusion method is available at https://github.com/hli1221/imagefusion-nestfuse

Non-contact Robust Respiration Detection By Using Radar-Depth Camera Sensor Fusion.

In this paper, a non-contact respiration detection scheme based on Doppler radar-depth camera sensor fusion has been proposed. A continuous-wave (CW) Doppler radar sensor and a depth camera are used to measure the respiratory motion separately. Then the Bayesian sensor fusion algorithm is used to estimate the cycle-to-cycle breathing rate. The experiments prove that the proposed fusion scheme can provide an accurate breathing rate estimation than using a single sensor. In particular, the proposed scheme can give a reasonable estimation even under the influence of body movement.

Robust spiking cortical model and total-variational decomposition for multimodal medical image fusion

In field of medical image, a large number of related works demonstrated that the decomposition theories (such as multi-scale transform (MST) techniques) have better performance in preserving details and structures information. However, the most of hierarchical structure-based methods fail to appropriately separate energy and detailed information, they even produce some unsatisfied effects such as low-contrast resolution and anamorphose. To address the above issues, we present a modified image decomposition algorithm to get a good trade-off between speed and performance. Specifically, it utilizes the total-variational transform into moving least squares method (TV-MLS), which can make the result more robust to noise and fully reserve the dominant structure. Firstly, we perform total-variational decomposition on the source images, and yield a series of detail layers and base layers. Next, the layers are fused using the robust adaptive dual-channel spiking cortical model (RA-DCSCM), CNNs and the fine-designed fusion rule. Eventually, we provide extensive qualitative analysis for the proposed fusion scheme in both subjective visual inspection and quality metrics to verify that our method is more competitive against other existing methods.

Image and WLAN Bimodal Integration for Indoor User Localization

Recently, we experience the increasing prevalence of wearable cameras, some of which feature Wireless Local Area Network (WLAN) connectivity, and the abundance of mobile devices equipped with on-board camera and WLAN modules. Motivated by this fact, this work presents an indoor localization system that leverages both imagery and WLAN data for enabling and supporting a wide variety of envisaged location-aware applications ranging from ambient and assisted living to indoor mobile gaming and retail analytics. The proposed solution integrates two complementary localization approaches, i.e., one based on WLAN and another one based on image location-dependent data, using a fusion engine. Two fusion strategies are developed and investigated to meet different requirements in terms of accuracy, run time, and power consumption. The one is a light-weight threshold-based approach that combines the location outputs of two localization algorithms, namely a WLAN-based algorithm that processes signal strength readings from the surrounding wireless infrastructure using an extended Naive Bayes approach and an image-based algorithm that follows a novel approach based on hierarchical vocabulary tree of SURF (Speeded Up Robust Features) descriptors. The second fusion strategy employs a particle filter algorithm that operates directly on the WLAN and image readings and also includes prior position estimation information in the localization process. Extensive experimental results using real-life data from an indoor office environment indicate that the proposed fusion strategies perform well and are competitive against standalone WLAN and imaged-based algorithms, as well as alternative fusion localization solutions.

Weighted quasi‐arithmetic mean based score level fusion for multi‐biometric systems

Biometrics is now being principally employed in many daily applications ranging from the border crossing to mobile user authentication. In the high-security scenarios, biometrics require stringent accuracy and performance criteria. Towards this aim, multi-biometric systems that fuse the evidences from multiple sources of biometric have exhibited to diminish the error rates and alleviate inherent frailties of the individual biometric systems. In this article, a novel scheme for score-level fusion based on weighted quasi-arithmetic mean (WQAM) has been proposed. Specifically, WQAMs are estimated via different trigonometric functions. The proposed fusion scheme encompasses properties of both weighted mean and quasi-arithmetic mean. Moreover, it does not require any leaning process. Experimental results on three publicly available data sets (i.e. NIST-BSSR1 Multimodal, NIST-BSSR1 Fingerprint and NIST-BSSR1 Face) for multi-modal, multi-unit and multi-algorithm systems show that presented WQAM fusion algorithm outperforms the previously proposed score fusion rules based on transformation (e.g. t-norms), classification (e.g. support vector machines) and density estimation (e.g. likelihood ratio) methods.

Vision tracking based on adaptive interactive fusion

Under the influence of COVID-19, detection and identification of moving targets are very important for personnel management. A lot of research work has improved the accuracy and robustness of the moving target tracking method, but the recognition accuracy of the traditional target tracking method in complex scenes (lighting changes, background interference, posture changes and other factors) is not satisfactory. In this paper, in view of the limitations of single feature representation of target objects, the method of fusion of HSV color features and edge direction features is used to identify and detect moving targets. In each frame of the tracking process, the weight of each feature is adjusted adaptively according to the proposed fusion strategy, and the position of the target is located by using the method of double template matching. Experiments show that the proposed tracking algorithm based on multi feature fusion can meet the requirements of moving target recognition in complex scenes. The method proposed in this paper has a certain reference value for personnel management under the influence of COVID-19.

A New Fusion Estimation Method for Multi-Rate Multi-Sensor Systems With Missing Measurements

A new fusion strategy is introduced in this article to estimate state for multi-rate multi-sensor systems with missing measurements. N sensors, which possess various sampling rates, render the measurements. Missing measurements with a certain probability pattern are also investigated. For these types of systems, Multi-rate Kalman filters are designed to estimate a target position at various sampling rates. Next, Ordered Weighted Averaging (OWA) operator is utilized to integrate multi-rate Kalman filters and improve the estimation quality. A new fusion strategy based on a real covariance matrix is introduced for updating the weighting factors, and proof of convergence is granted. Simulation studies on a tracking system verify the superior performance of the proposed fusion strategy in comparison with the Kalman filter, the multi-rate Kalman filters, and also the previous fusion methodology.

Functional characterization of a plant-produced infectious bursal disease virus antigen fused to the constant region of avian IgY immunoglobulins.

Infectious bursal disease virus (IBDV) is the cause of an economically important highly contagious disease of poultry, and vaccines are regarded as the most beneficial interventions for its prevention. In this study, plants were used to produce a recombinant chimeric IBDV antigen for the formulation of an innovative subunit vaccine. The fusion protein (PD-FcY) was designed to combine the immunodominant projection domain (PD) of the viral structural protein VP2 with the constant region of avian IgY (FcY), which was selected to enhance antigen uptake by avian immune cells. The gene construct encoding the fusion protein was transiently expressed in Nicotiana benthamiana plants and an extraction/purification protocol was set up, allowing to reduce the contamination by undesired plant compounds/proteins. Mass spectrometry analysis of the purified protein revealed that the glycosylation pattern of the FcY portion was similar to that observed in native IgY, while in vitro assays demonstrated the ability of PD-FcY to bind to the avian immunoglobulin receptor CHIR-AB1. Preliminary immunization studies proved that PD-FcY was able to induce the production of protective anti-IBDV-VP2 antibodies in chickens. In conclusion, the proposed fusion strategy holds promises for the development of innovative low-cost subunit vaccines for the prevention of avian viral diseases.

Decision‐fusion‐based reliable CSS scheme in CR networks

Cognitive radio (CR) is a rapidly growing technology that can be employed to effectively utilise the radio spectrum. The detection accuracy of the CR user is compromised when a network is under degrading conditions like fading and shadowing effects. Cooperative spectrum sensing (CSS) has been extensively employed to overcome these issues. In CSS, all secondary users communicate with the fusion centre (FC) to share the PU information. Hard fusion schemes are applied to make a correct decision about PU presence at FC; however, such approaches lack reliable decision-making. Hence, there is a need to construct more accurate and reliable fusion schemes. In this study, a neural network (NN)-based decision fusion scheme at FC is used to construct a reliable decision. In the proposed scheme, NN is used as a machine learning tool that is trained using input data and output data to generate the desired fusion model. It has been evaluated through simulation results that the proposed fusion scheme sensing accuracy is much better as compared to conventional fusion schemes and other state-of-the-art schemes proposed in the literature. Further, the proposed fusion scheme is tested using the clustering approach to make it more reliable and accurate.

An optimal weighted averaging fusion strategy for remotely sensed images

Image fusion plays a vital role in providing better visualization of remotely sensed image data. Most earth observation satellites have sensors that provide both high spatial resolution panchromatic (PAN) images and low resolution multispectral (MS) images. In this paper, we propose a new fusion algorithm that optimally combines spectral information from MS image and spatial information from the PAN image of the same scene to create a single comprehensive fused image. As the performance of the fusion scheme relies on the choice of fusion rule, the proposed algorithm is based on a weighted averaging fusion rule that uses optimal weights obtained from brain storm optimization (BSO) algorithm for the fusion of high frequency and low frequency coefficients obtained by applying Curvelet transform to the source images. The objective function in BSO is formulated with twin objectives of maximizing the entropy and minimizing the root mean square error. The fusion results are compared with the existing fusion techniques, such as Brovey, principal component analysis, discrete wavelet transform, non sub-sampled contourlet transform, and intensity hue saturation. From the experimental results and analysis, the proposed fusion algorithm gives a better fusion performance in terms of subjective and objective measures than the traditional algorithms. As a benefit, the proposed fusion scheme preserves spectral information of the MS image with increased spatial resolution and edge information.

GF-4 Satellite and Automatic Identification System Data Fusion for Ship Tracking

GF-4 satellite, as the first geostationary orbit optical remote sensing satellite with medium resolution, has the abilities to point and stare at a particular sea area for maritime surveillance. The integration of GF-4 image sequence and automatic identification system (AIS) reports has the appealing potential to provide a better maritime situational awareness by tracking ships, especially noncooperative targets. The aim of this letter is to provide a track-level fusion architecture for GF-4 and AIS heterogeneous data based on the characteristics of different data sources to form a consolidated and comprehensive maritime picture. The proposed fusion strategy is tested in the specific area of the East China Sea, and experimental results verify the effectiveness of our method.

Land-cover classification from multiple classifiers using decision fusion based on the probabilistic graphical model

ABSTRACTA novel method of using different classification algorithms in an integrated manner by adaptively weighted decision level fusion was proposed. The proposed fusion scheme involves two steps. First, we processed the data using each classifier separately and provided probability estimations for each pixel of the considered classes. Then, the results are aggregated on the basis of the decision rule of probabilistic graphical model according to the capabilities of classifiers and ancillary information. The method was tested and validated through the Landsat 8 operational land imager data using two different classifiers, namely, maximum likelihood classifier and support vector machine. The proposed method provided higher accuracy improvement than the separate use of different classifiers and that complex landscapes, such as mountainous regions, have higher accuracy improvement than the relatively homogenous ones. Moreover, the method can handle more than two types of classifiers and effectively introduce additional ancillary information for adaptive weight selection. These findings can help promote our proposed method as an emerging approach for land-cover classification through remote sensing technology.

A Multi-Biometric System Based on Feature and Score Level Fusions

In general, the information of multiple biometric modalities is fused at a single level, for example, score level or feature level. The recognition accuracy of a multimodal biometric system may not be improved by carrying fusion at a single level, since one matcher may provide a performance lower than that provided by other matchers. In view of this, we propose a new fusion scheme, referred to as the matcher performance-based (MPb) fusion scheme, in which the fusion is carried out at two levels, feature level, and score level, to improve the overall recognition accuracy. First, we consider the performance of the individual matchers in order to find out which of the modalities should be used for fusion at the feature level. Then, the selected modalities are fused at this level by utilizing their encoded features. Next, we fuse the score obtained from the feature-level fusion with that of the modality for which the performance is the highest. In order to carry out this fusion, a new normalization technique referred to as the overlap extrema-variation-based anchored min-max (OEVBAMM) normalization technique, is also proposed. By considering three modalities, namely, fingerprint, palmprint, and earprint, the performance of the proposed fusion scheme as well as that of the single level fusion scheme, both with various normalization and weighting techniques are evaluated in terms of a number of metrics. It is shown that the multi-biometric system based on the proposed fusion scheme provides the best performance when it employs the new normalization technique and the confidence-based weighting (CBW) method.

Exploiting Quality-Guided Adaptive Optimization for Fusing Multimodal Medical Images

Multimodal medical image fusion (MMIF) plays critical roles in image-guided clinical diagnostics and treatment. Pulse coupled neural network (PCNN) has been applied in image fusion for several years. In the schemes of image fusion based on PCNN, the authors have adjusted variables manually, so that it is difficult to get satisfying effects which limit in dealing with medical images with different modalities. This paper presents a quality-guided adaptive optimization method for MMIF, which is based on PCNN optimized by multi-swarm fruit fly optimization algorithm (MFOA). To reduce the implementation cost and improve the performance of the MFOA, quality assessment for multimodal medical image fusion was chosen to be the hybrid fitness function. Guided by such quality measurement, the adaptive PCNN using the MFOA (PCNN-MFOA) is proposed, which could automatically fit the optimal variables to the source images and enhance the fusion effect. The experimental results visually and quantitatively show that the proposed fusion strategy is more effective than the state-of-the-art methods and it is more effective in processing medical images with different modalities.

An End-to-End Human Segmentation by Region Proposed Fully Convolutional Network

Person segmentation in images has various applications, for example, smart home, human–computer interaction, and scene perception for self-driving cars, which are a key feature of the Internet of Things. Due to limitations in performance, such as accuracy and runtime, most traditional methods do not fulfill the practical requirements. Deep learning-based modern segmentation systems become prevalent. Fully convolutional network (FCN), as a classic image semantic segmentation method, directly optimizes the semantic map from the original image in a pixel-wise manner without using pixel-correlations or global object information. In this paper, we propose an efficient end-to-end person segmentation network structure fusing the person detection network with the FCN. The person detection network estimates the region of interest of persons and enforces the segmentation network to focus on the optimization of person segmentation. The loss function of the proposed network considers both the segmentation error and the detection bias error. In addition, the lightweight design of the detection network that optimizes only person bounding-box coordinates enables real-time person detection. The experimental comparison and analysis of several different networks on several datasets show the effectiveness of the proposed fusion strategy. The approach shows a promising practical application potential by fast running time and high segmentation accuracy.

Fusion schemes for image-to-video person re-identification

ABSTRACTThis paper focuses on improving the performance of image-to-video person re-identification through feature fusion. In this study, image-to-video person re-identification is formulated as a classification-based information retrieval in which a pedestrian appearance model is learned in the training phase, and the identity of an interested person is determined based on the probability his/her probe image belongs to the model. Four state-of-the-art features belonging to two categories: hand-designed features and learned features are investigated for person image representation. They are Kernel Descriptor, Gaussian of Gaussian, features extracted from two famous convolutional neural networks (GoogleNet and ResNet). Furthermore, three fusion schemes that are early fusion, product-rule and query-adaptive late fusion are proposed. To evaluate the performance of the chosen features for person appearance representation as well as their combination in three proposed fusion schemes, 114 experiments on two public benchmark datasets (CAVIAR4REID and RAiD) have been conducted. The experiments confirm the robustness and effectiveness of the proposed fusion schemes. The proposed schemes obtain improvement of +7.16%, +5.42%, and +6.30% at rank-1 over those of single feature in case A, case B of CAVIAR4REID, and RAiD, respectively.

Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition

Multimodal emotion recognition has attracted great interest recently and numerous methodologies have been successfully investigated. However, the task requires the effective fusion multimodal representations in audio and video domains, and existing approaches still perform poorly on such a challenging task. This paper proposes a novel framework for recognizing emotion from multiple sources including facial expression, pose, body movements, and voice. In this framework, we first introduce new deep spatio-temporal features by cascading 3-dimensional convolution neural networks (C3Ds) and deep belief networks (DBNs) to effectively model spatial and temporal information presented in video and audio for emotion recognition. We subsequently propose a new feature-level fusion approach based on a bilinear pooling theory to combine the visual and audio feature vectors. The proposed fusion strategy allows all elements of the component vectors to interact with each other in an effective way, resulting in expressively capturing the complex and intrinsic associations between the component modalities. Extensive experiments conducted on the eNTERFACE and FABO multimodal emotion databases demonstrate that our proposed system leads to improved multimodal emotion recognition performance and significantly outperforms recent state-of-the-art approaches.

Modification of Extracellular Vesicles by Fusion with Liposomes for the Design of Personalized Biogenic Drug Delivery Systems.

Extracellular vesicles (EVs) are recognized as nature's own carriers to transport macromolecules throughout the body. Hijacking this endogenous communication system represents an attractive strategy for advanced drug delivery. However, efficient and reproducible loading of EVs with therapeutic or imaging agents still represents a bottleneck for their use as a drug delivery system. Here, we developed a method for modifying cell-derived EVs through their fusion with liposomes containing both membrane and soluble cargoes. The fusion of EVs with functionalized liposomes was triggered by polyethylene glycol (PEG) to create smart biosynthetic hybrid vectors. This versatile method proved to be efficient to enrich EVs with exogenous lipophilic or hydrophilic compounds, while preserving their intrinsic content and biological properties. Hybrid EVs improved cellular delivery efficiency of a chemotherapeutic compound by a factor of 3-4, as compared to the free drug or the drug-loaded liposome precursor. On one side, this method allows the biocamouflage of liposomes by enriching their lipid bilayer and inner compartment with biogenic molecules. On the other side, the proposed fusion strategy enables efficient EV loading, and the pharmaceutical development of EVs with adaptable activity and drug delivery property.