Multi-view Information Fusion Research Articles

Evidential dissonance measure in robust multi-view classification to resist adversarial attack

Multi-view learning is effective in improving data classification accuracy through integrating information from multiple sources. To guarantee the reliability of multi-view classification, the trusted multi-view classification methods have been explored. However, extant trusted multi-view classification methods are still vulnerable to the low-quality views containing adversarial samples. This vulnerability arises from a challenge in accurately assessing the quality of data views that are subjected to adversarial attacks. To address this issue, we propose a robust multi-view classification method with dissonance measure of adversarial samples. Specifically, the proposed method utilizes the evidential dissonance measure in subjective logic to assess the quality of data views when encountering adversarial attacks. Based on the dissonance measure, we further propose a strategy of dissonance-aware belief integration for multi-view information fusion and construct the inter-view evidential gradient penalty in the learning objective of multi-view classification to improve the model robustness to resist adversarial samples. Experiments on diverse multi-view datasets confirm the reliability and robustness of the proposed multi-view classification method.

Automatic liver segmentation from CT volumes based on multi-view information fusion and condition random fields

Liver segmentation from CT images is a fundamental and essential step for many clinical applications, such as disease diagnosis, surgery navigator, and radiation therapy. Due to the limited computing resources, automatic and accurate liver segmentation from 3D CT volume is still a challenge task. In this paper, we propose an automatic liver segmentation method by fusing multi-view information. First, a 2D network combined with Dual Self-Attention (DSA) is proposed and applied on multi-sectional 2D slices reconstructed from axial, coronal, and sagittal directions. Then, to generate 3D segmentation results, a lightweight 3D network is designed to fuse the segmentation results from different viewing. Finally, a full connection condition random field is used to refine the 3D results. The proposed method is evaluated on four public datasets, including 3DIRCADb, CHAOS, Sliver07, and LiTS. The experimental results show that the proposed method can accurately segment the liver region for the datasets with large or small cases, as well as healthy or lesion livers. The dice values obtained by the proposed method on four datasets are 0.952, 0.969, 0.958, and 0.964, respectively, which are better than those obtained by many existing ones.

RDPGNet: A road extraction network with dual-view information perception based on GCN

Automatic road extraction from satellite images is a popular research hotspot topic in the field of remote sensing. However, the complexity of road features and structures hinders their precise extraction. Road features have different saliency due to the occlusion of other objects and diverse road types. Road structures are scattered across various locations and affected by geographical conditions, population distribution, and transportation demands. These issues render current methods based on convolutional neural networks prone to fragmented and incomplete road extraction results. To address these issues, a Road Extraction Network with Dual-View Information Perception Based on GCN (RDPGNet) is proposed, where the GCN-based dual-view perceptron (GDVP) uses the superior information interaction capability of the GCN to explore road information under a dual view. In the GDVP, we first design the road feature saliency graph reasoning module (RFSG) and road structure homogeneous space graph reasoning module (RSHS) based on the features and structures. The RFSG obtains the similarity of road feature information in different regions as a similarity matrix to participate in the graph reasoning process, ensuring that equal treatment is given to regions with varying saliency levels. The RSHS projects the road structure onto a corresponding homogeneous space while aggregating and interacting with information through graph convolution, thereby enhancing the network’s perception of roads in diverse locations. Second, because multi-view information usually shares a potential common representation, a multi-view information fusion and alignment strategy (MVFA) is designed to comprehensively model road information. Experimental results obtained from two public datasets indicate that RDPGNet outperforms other state-of-the-art networks.

Align While Fusion: A Generalized Nonaligned Multiview Multilabel Classification Method.

In the task of multiview multilabel (MVML) classification, each object is described by several heterogeneous view features and annotated with multiple relevant labels. Existing MVML methods usually assume that these heterogeneous features are strictly view-aligned, and they directly conduct cross-view information fusion to train a multilabel prediction model. However, in real-world scenarios, such strict view-aligned requirement can be hardly satisfied due to the recurrent spatiotemporal asynchronism when collecting MVML data, which would cause inaccurate multiview fusion results and degrade the classification performance. To address this issue, we propose a generalized nonaligned MVML (GNAM) classification method, which achieves multiview information fusion while aligning cross-view features and accordingly learns a desired multilabel classifier. Specifically, we first introduce a multiorder matching alignment strategy to achieve cross-view feature alignments, where both first-order feature correspondence and second-order structure correspondence are jointly integrated to guarantee the compactness of the view-alignment results. Afterward, a commonality-and individuality-based multiview fusion structure is formulated on the aligned-view features to excavate the consistencies and complementarities across different views, which leads all relevant multiview semantic labels, especially rare labels, to be characterized more comprehensively. Finally, we embed adaptive global label correlations to multilabel classification model to further enhance its semantic expression integrity and develop an alternative algorithm to optimize the whole model. Extensive experimental results have verified that GNAM is significantly superior to other state-of-the-art methods.

Atypical architectural distortion detection in digital breast tomosynthesis: a multi-view computer-aided detection model with ipsilateral learning

Objective. Breast architectural distortion (AD), a common imaging symptom of breast cancer, is associated with a particularly high rate of missed clinical detection. In clinical practice, atypical ADs that lack an obvious radiating appearance constitute most cases, and detection models based on single-view images often exhibit poor performance in detecting such ADs. Existing multi-view deep learning methods have overlooked the correspondence between anatomical structures across different views. Approach. To develop a computer-aided detection (CADe) model for AD detection that effectively utilizes the craniocaudal (CC) and mediolateral oblique (MLO) views of digital breast tomosynthesis (DBT) images, we proposed an anatomic-structure-based multi-view information fusion approach by leveraging the related anatomical structure information between these ipsilateral views. To obtain a representation that can effectively capture the similarity between ADs in images from ipsilateral views, our approach utilizes a Siamese network architecture to extract and compare information from both views. Additionally, we employed a triplet module that utilizes the anatomical structural relationship between the ipsilateral views as supervision information. Main results. Our method achieved a mean true positive fraction (MTPF) of 0.05–2.0, false positives (FPs) per volume of 64.40%, and a number of FPs at 80% sensitivity (FPs@0.8) of 3.5754; this indicates a 6% improvement in MPTF and 16% reduction in FPs@0.8 compared to the state-of-the-art baseline model. Significance. From our experimental results, it can be observed that the anatomic-structure-based fusion of ipsilateral view information contributes significantly to the improvement of CADe model performance for atypical AD detection based on DBT. The proposed approach has the potential to lead to earlier diagnosis and better patient outcomes.

Multi-view information fusion using multi-view variational autoencoder to predict proximal femoral fracture load.

Hip fracture occurs when an applied force exceeds the force that the proximal femur can support (the fracture load or "strength") and can have devastating consequences with poor functional outcomes. Proximal femoral strengths for specific loading conditions can be computed by subject-specific finite element analysis (FEA) using quantitative computerized tomography (QCT) images. However, the radiation and availability of QCT limit its clinical usability. Alternative low-dose and widely available measurements, such as dual energy X-ray absorptiometry (DXA) and genetic factors, would be preferable for bone strength assessment. The aim of this paper is to design a deep learning-based model to predict proximal femoral strength using multi-view information fusion. We developed new models using multi-view variational autoencoder (MVAE) for feature representation learning and a product of expert (PoE) model for multi-view information fusion. We applied the proposed models to an in-house Louisiana Osteoporosis Study (LOS) cohort with 931 male subjects, including 345 African Americans and 586 Caucasians. We performed genome-wide association studies (GWAS) to select 256 genetic variants with the lowest p-values for each proximal femoral strength and integrated whole genome sequence (WGS) features and DXA-derived imaging features to predict proximal femoral strength. The best prediction model for fall fracture load was acquired by integrating WGS features and DXA-derived imaging features. The designed models achieved the mean absolute percentage error of 18.04%, 6.84% and 7.95% for predicting proximal femoral fracture loads using linear models of fall loading, nonlinear models of fall loading, and nonlinear models of stance loading, respectively. The proposed models are capable of predicting proximal femoral strength using WGS features and DXA-derived imaging features. Though this tool is not a substitute for predicting FEA using QCT images, it would make improved assessment of hip fracture risk more widely available while avoiding the increased radiation exposure from QCT.

Multi-view graph convolution network for the recognition of human action with spatial and temporal occlusion problems

Human action recognition holds great significance in the field of social security. However, the challenges posed by occlusion and changes in viewpoint create specific obstacles in achieving accurate recognition. In this study, we propose a multi-view graph convolution fusion method to effectively address this issue. Specifically, considering the limited availability of public human action datasets that include occlusion, we introduce an adaptive multi-view spatial–temporal occlusion generation method. It allows us to generate occlusion skeleton data from multiple viewpoints, ensuring a close resemblance to real-life scenarios with minimal modifications to existing public datasets. Additionally, we present a plug-and-play multi-view information fusion module, briefed as MGL, which aims to solve the occlusion problem. The MGL combines the capabilities of Graph Convolutional network (GCN) and Long Short-Term Memory network (LSTM), in which GCN is employed to reconstruct human skeleton information using multi-view spatial occlusion data, and LSTM captures the long-term dependencies within the temporal sequences of the multi-view data. Moreover, an attention mask mechanism is introduced to highlight key joint features. Experiment results illustrate the excellent performance of our method on the NTU RGB+D 60 and NTU RGB+D 120 datasets.

UNTIE: Clustering analysis with disentanglement in multi-view information fusion

Multi-view clustering focuses on exploring cluster structures among multiple views and is an effective approach to achieve multi-view information fusion without requiring label supervision. However, multiple views’ useful and meaningless information usually is entangled and the latter might cause negative influences in the fusion process. In this paper, we research the interpretability for different information in multiple views and propose a novel method (termed UNTIE) to address their entanglement. Specifically, in UNTIE, discrete view-common variable is introduced to explore all views’ common information, and continuous view-private variables are introduced to learn individual views’ private information. In this way, the learned representations are disentangled and interpretable where each variable represents a single factor among multi-view data. Then, the useful information in disentangled view-common and view-private representations are leveraged to conduct comprehensive multi-view clustering, making UNTIE can explore common and complementary information from multiple views while obtaining the robustness to meaningless information. Finally, UNTIE has the ability to controllably generate samples and extensive experiments demonstrate its superior representation ability and clustering performance.

Dual Fusion-Propagation Graph Neural Network for Multi-View Clustering

Deep multi-view representation learning focuses on training a unified low-dimensional representation for data with multiple sources or modalities. With the rapidly growing attention of graph neural networks, more and more researchers have introduced various graph models into multi-view learning. Although considerable achievements have been made, most existing methods usually propagate information in a single view and fuse multiple information only from the perspective of attributes or relationships. To solve the aforementioned problems, we propose an efficient model termed Dual Fusion-Propagation Graph Neural Network (DFP-GNN) and apply it to deep multi-view clustering tasks. The proposed method is designed with three submodules and has the following merits: a) The proposed view-specific and cross-view propagation modules can capture the consistency and complementarity information among multiple views; b) The designed fusion module performs multi-view information fusion with the attributes of nodes and the relationships among them simultaneously. Experiments on popular databases show that DFP-GNN achieves significant results compared with several state-of-the-art algorithms.

Multi-View Information Fusion Fault Diagnosis Method Based on Attention Mechanism and Convolutional Neural Network

Multi-view information fusion can provide more accurate, complete and reliable data descriptions for monitoring objects, effectively improve the limitations and unreliability of single-view data. Existing multi-view information fusion based on deep learning mostly focuses on the feature level and decision level, with large information loss, and does not distinguish the view weight in the fusion process. To this end, a multi-view data level information fusion model CAM_MCFCNN with view weight was proposed based on a channel attention mechanism and convolutional neural network. The model used the channel characteristics to implement multi-view information fusion at the data level stage, which made the fusion position and mode more natural and reduced the loss of information. A multi-channel fusion convolutional neural network was used for feature learning. In addition, the channel attention mechanism was used to learn the view weight, so that the algorithm could pay more attention to the views that contribute more to the fault identification task during the training process, and more reasonably integrate the information of different views. The proposed method was verified by the data of the planetary gearbox experimental platform. The multi-view data and single-view data were used as the input of the CAM_MCFCNN model and single-channel CNN model respectively for comparison. The average accuracy of CAM_MCFCNN on three constant-speed datasets reached 99.95%, 99.87% and 99.92%, which was an improvement of 0.95%, 2.25%, and 0.04%, compared with the single view with the highest diagnostic accuracy, respectively. When facing limited samples, CAM_MCFCNN had similar performance. Finally, compared with different multi-view information fusion algorithms, CAM_MCFCNN showed better stability and higher accuracy. The experimental results showed that the proposed method had better performance, higher diagnostic accuracy and was more reliable, compared with other methods.

Predicting miRNA-disease associations based on multi-view information fusion.

MicroRNAs (miRNAs) play an important role in various biological processes and their abnormal expression could lead to the occurrence of diseases. Exploring the potential relationships between miRNAs and diseases can contribute to the diagnosis and treatment of complex diseases. The increasing databases storing miRNA and disease information provide opportunities to develop computational methods for discovering unobserved disease-related miRNAs, but there are still some challenges in how to effectively learn and fuse information from multi-source data. In this study, we propose a multi-view information fusion based method for miRNA-disease association (MDA)prediction, named MVIFMDA. Firstly, multiple heterogeneous networks are constructed by combining the known MDAs and different similarities of miRNAs and diseases based on multi-source information. Secondly, the topology features of miRNAs and diseases are obtained by using the graph convolutional network to each heterogeneous network view, respectively. Moreover, we design the attention strategy at the topology representation level to adaptively fuse representations including different structural information. Meanwhile, we learn the attribute representations of miRNAs and diseases from their similarity attribute views with convolutional neural networks, respectively. Finally, the complicated associations between miRNAs and diseases are reconstructed by applying a bilinear decoder to the combined features, which combine topology and attribute representations. Experimental results on the public dataset demonstrate that our proposed model consistently outperforms baseline methods. The case studies further show the ability of the MVIFMDA model for inferring underlying associations between miRNAs and diseases.

MFHE: Multi-View Fusion-Based Heterogeneous Information Network Embedding

Depending on the type of information network, information network embedding is classified into homogeneous information network embedding and heterogeneous information network (HIN) embedding. Compared with the homogeneous network, HIN composition is more complex and contains richer semantics. At present, the research on homogeneous information network embedding is relatively mature. However, if the homogeneous information network model is directly applied to HIN, it will cause incomplete information extraction. It is necessary to build a specialized embedding model for HIN. Learning information network embedding based on the meta-path is an effective approach to extracting semantic information. Nevertheless, extracting HIN embedding only from a single view will cause information loss. To solve these problems, we propose a multi-view fusion-based HIN embedding model, called MFHE. MFHE includes four parts: node feature space transformation, subview information extraction, multi-view information fusion, and training. MFHE divides HIN into different subviews based on meta-paths, models the local information accurately in the subviews based on the multi-head attention mechanism, and then fuses subview information through a spatial matrix. In this paper, we consider the relationship between subviews; thus, the MFHE is applicable to complex HIN embedding. Experiments are conducted on ACM and DBLP datasets. Compared with baselines, the experimental results demonstrate that the effectiveness of MFHE and HIN embedding has been improved.

Condenser Vacuum Degree Prediction Model with Multi-View Information Fusion

Vacuum degree is a crucial factor for the operation of thermoelectric generating set. Existing approaches typically use machine learning algorithms to link the relationship between unit operating data and condenser vacuum degree by focusing on the temporal information within the data, while ignoring the frequency information implied in the historical condenser vacuum degree. To fully use of frequency information and further improve prediction accuracy, we propose a novel condenser vacuum degree prediction model with multi-view information fusion. In specific, the implicit frequency information in the historical vacuum degree sequence is explored via a combination of Variational Mode Decomposition (VMD) and Convolutional Neural Network (CNN). Furthermore, Transformer encoder is used to extract the temporal information from the unit operating data. Finally, the two views information are fused for condenser vacuum degree prediction. Extensive experiments conducted on the real data collected from a power plant demonstrate the superiority of the proposed method over several state-of-the-art methods.

Fusing Local and Global Information for One-Step Multi-View Subspace Clustering

Multi-view subspace clustering has drawn significant attention in the pattern recognition and machine learning research community. However, most of the existing multi-view subspace clustering methods are still limited in two aspects. (1) The subspace representation yielded by the self-expression reconstruction model ignores the local structure information of the data. (2) The construction of subspace representation and clustering are used as two individual procedures, which ignores their interactions. To address these problems, we propose a novel multi-view subspace clustering method fusing local and global information for one-step multi-view clustering. Our contribution lies in three aspects. First, we merge the graph learning into the self-expression model to explore the local structure information for constructing the specific subspace representations of different views. Second, we consider the multi-view information fusion by integrating these specific subspace representations into one common subspace representation. Third, we combine the subspace representation learning, multi-view information fusion, and clustering into a joint optimization model to realize the one-step clustering. We also develop an effective optimization algorithm to solve the proposed method. Comprehensive experimental results on nine popular multi-view data sets confirm the effectiveness and superiority of the proposed method by comparing it with many state-of-the-art multi-view clustering methods.

Multiview three-dimensional imaging using a Risley-prism-based spatially adaptive virtual camera field.

We present a three-dimensional (3D) imaging system that incorporates a stationary camera and Risley prisms. By planning prism rotation to generate a spatially adaptive virtual camera field, the system allows multiple virtual cameras in the field to capture any object from different perspectives for 3D reconstruction. An automatic virtual camera calibration method based on perspective projection and geometric optics is developed to enable virtual camera field construction and characterization. Moreover, a 3D computational reconstruction framework is proposed for multiview information fusion using the virtual camera field. This framework combines nonlinear distortion correction with epipolar geometry computation to efficiently perform image rectification and stereo matching, which can further facilitate 3D object reconstruction through multiview triangulation. The experiments on synthetic and real data validate the feasibility and flexibility of our 3D imaging technique.

Hyperspectral Image Classification Based on Semisupervised Self-Learning and Multiview Information Fusion

Hyperspectral images (HSIs) have been utilized in various fields due to abundant information, whose application is to a great extent limited by the number of labeled samples. Hence, how to exploit the diverse and complementary characteristics of multiple information from different views inherent in the HSIs is very critical for improving the classification performance under the condition of small-sized training set. This letter presents a novel method to tackle the small labeled sample size problem with semisupervised self-learning (S³L) and multiview information fusion. First, a sample augmentation scheme based on S³L and high-reliable neighborhood structure is designed for realizing training set enlargement. Thus, the pseudo-labeled samples with high-quality would be automatically picked out from the unlabeled data by exploiting complementary information from multiple views, i.e., semantic information, spectral bands, texture and geospatial information. Then, we retrain the classifier by the enlarged training set and generate an intermediate classification map. Finally, we simultaneously utilize the location and gradient information of the samples to adaptively realize the refined landcover classification. Experimental results on three widely-used data sets compared with several representative methods for small-sized HSI classification validate the effectiveness of the proposed method.

Sparse additive discriminant canonical correlation analysis for multiple features fusion

Canonical correlation analysis (CCA) is an unsupervised representation learning technique to correlate multi-view data by learning a set of projection matrices. Being complementary with CCA, many discriminant methods are proposed to extract discriminative features of multi-view data by introducing the supervised class information. However, the learned projection matrices in these methods are mathematically constrained to be equal rank to the class number, and thus cannot represent the original data comprehensively. In this paper, we propose a general multi-view information fusion technique, named sparse additive discriminative canonical correlation analysis (SaDCCA). On one hand, SaDCCA is equipped with a strong degree of discrimination by defining a new affinity matrix that reflects the high-order characteristics of intra-class and the separability of inter-class. On the other hand, SaDCCA can exploit the correlation among multi-view data by maintaining the spirit of CCA. The discrimination among classes and the correlation among views are integrated in an additive manner. To obtain the sparse solutions, we first establish the relationship between the objective function and the underdetermined linear system equations, and then obtain the ℓ1-norm solution by accelerated Bregman iteration with matrix form. SaDCCA has no rank constraint on the projection matrices and is capable to provide accurate recognition performance. Experiments conducted on some publicly available datasets demonstrate the effectiveness of the proposed approach.

An aggregate gradation detection method based on multi-view information fusion

In this study, the morphologies of the aggregate in multiple views were analysed during the falling of particles to calculate aggregate gradation. Four types of characterisation parameters were selected to extract the multi-view information of aggregate particles in five views. Based on the multi-view information, the aggregate particles were classified using principal component analysis and a probabilistic neural network. An aggregate equivalent volume characterisation method was formulated to calculate the aggregate mass, whereby the aggregate volume was converted into the aggregate mass by the least-squares method. The experimental results show that the proposed aggregate sieving method can effectively realise the gradation classification of aggregates. Considering the product of the maximum area and the minimum equivalent Feret ellipse minor axis as the equivalent volume, the calculated aggregate mass yielded a good correlation with the actual aggregate mass. Compared with single-view information, multi-view information can improve the accuracy and repeatability of gradation calculations. The use of multi-view information to calculate aggregate gradation can reduce manpower and improve detection efficiency, which is important for applications in the construction industry.

Learning disentangled user representation with multi-view information fusion on social networks

User representation learning is one prominent and critical task of user analysis on social networks, which derives conceptual user representations to improve the inference of user intentions and behaviors. Previous efforts have shown its substantial value in multifarious real-world applications, including product recommendation, textual content modeling, link prediction, and many more. However, existing studies either underutilize multi-view information, or neglect the stringent entanglement among underlying factors that govern user intentions, thus deriving deteriorated representations. To overcome these shortages, this paper proposes an adversarial fusion framework to fully exploit substantial multi-view information for user representation, consisting of a generator and a discriminator. The generator learns representations with a variational autoencoder, and is forced by the adversarial fusion framework to pay specific attention to substantial informative signs, thus integrating multi-view information. Furthermore, the variational autoencoder used in the generator is novelly designed to capture and disentangle the latent factors behind user intentions. By fully utilizing multi-view information and achieving disentanglement, our model learns robust and interpretable user representations. Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of our proposed model.

Consensus One-Step Multi-View Subspace Clustering

Multi-view clustering has attracted increasing attention in multimedia, machine learning and data mining communities. As one kind of the essential multi-view clustering algorithm, multi-view subspace clustering (MVSC) becomes more and more popular due to its strong ability to reveal the intrinsic low dimensional clustering structure hidden across views. Despite superior clustering performance in various applications, we observe that existing MVSC methods <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">directly fuse multi-view information in the similarity level by merging noisy affinity matrices</i> ; and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">isolate the processes of affinity learning, multi-view information fusion and clustering</i> . Both factors may cause insufficient utilization of multi-view information, leading to unsatisfying clustering performance. This paper proposes a novel consensus one-step multi-view subspace clustering (COMVSC) method to address these issues. Instead of directly fusing multiple affinity matrices, COMVSC optimally integrates discriminative partition-level information, which is helpful to eliminate noise among data. Moreover, the affinity matrices, consensus representation and final clustering labels matrix are learned simultaneously in a unified framework. By doing so, the three steps can negotiate with each other to best serve the clustering task, leading to improved performance. Accordingly, we propose an iterative algorithm to solve the resulting optimization problem. Extensive experiment results on benchmark datasets demonstrate the superiority of our method against other state-of-the-art approaches.