Multi-view Data Research Articles

Partially multi-view clustering via re-alignment

Multi-view clustering learns consistent information from multi-view data, aiming to achieve more significant clustering characteristics. However, data in real-world scenarios often exhibit temporal or spatial asynchrony, leading to views with unaligned instances. Existing methods primarily address this issue by learning transformation matrices to align unaligned instances, but this process of learning differentiable transformation matrices is cumbersome. To address the challenge of partially unaligned instances, we propose Partially Multi-view Clustering via Re-alignment (PMVCR). Our approach integrates representation learning and data alignment through a two-stage training and a re-alignment process. Specifically, our training process consists of three stages: (i) In the coarse-grained alignment stage, we construct negative instance pairs for unaligned instances and utilize contrastive learning to preliminarily learn the view representations of the instances. (ii) In the re-alignment stage, we match unaligned instances based on the similarity of their view representations, aligning them with the primary view. (iii) In the fine-grained alignment stage, we further enhance the discriminative power of the view representations and the model’s ability to differentiate between clusters. Compared to existing models, our method effectively leverages information between unaligned samples and enhances model generalization by constructing negative instance pairs. Clustering experiments on several popular multi-view datasets demonstrate the effectiveness and superiority of our method. Our code is publicly available at https://github.com/WenB777/PMVCR.git.

Recovery of coronal dimmings

Context. Coronal dimmings are regions of reduced emission in the lower corona observed in the wake of coronal mass ejections (CMEs), representing their footprints. Studying the lifetime evolution of coronal dimmings helps us to better understand the recovery and replenishment of the corona after large-scale eruptions. Aims. We study the recovery of dimmings on different spatial scales to enhance our understanding of the replenishment and dynamics of the corona after CMEs. Methods. In order to investigate the long-term evolution of coronal dimming and its recovery, we propose two approaches that focus on both the global and the local evolution of dimming regions: the fixed mask approach and the pixel boxes approach. We present four case studies (September 6, 2011; March 7, 2012; June 14, 2012; and March 8, 2019) in which a coronal dimming is associated with a flare/CME eruption. We analyzed each event with the same methodology, using extreme-ultraviolet filtergrams from the Solar Dynamics Observatory’s Atmospheric Imaging Assembly (SDO/AIA) and Solar TErrestrial RElations Observatory’s Extreme UltraViolet Imager (STEREO/EUVI) instruments. We identified the dimming region by image segmentation, then restricted the analysis to a specific portion of the dimming and tracked the time evolution of the dimming brightness and area. In addition, we study the behavior of small subregions inside the dimming area, of about 3 × 3 pixels, to compare the recovery in different regions of the dimming. Results. Three out of the four cases show a complete recovery 24 hours after the flare/CME eruption. The primary recovery mechanism identified in the observations is the expansion of coronal loops into the dimming region. The recovery of the brightness follows a two-step trend, with a steeper and quicker segment followed by a slower one. In addition, some parts of the dimming, which may be core dimmings, are still present at the end of the analysis time and do not recover within 3 days, whereas the peripheral regions (secondary dimmings) show a full recovery. Conclusions. The high temporal and spatial resolution of SDO/AIA observations combined with multi-view data of the STEREO/EUV instrument reveal high-situated coronal loops expanding after CME eruptions, which cover dimming regions and gradually increase their intensity. Our developed approaches enable the analysis of dimmings alongside these bright structures, revealing different timescales of recovery for core and secondary or twin dimming regions. Combined with magnetic field modeling, these methods lay the foundation for further systematic analysis of dimming recovery and enhance the knowledge gained from already-analyzed events.

Robust mixed-order graph learning for incomplete multi-view clustering

Incomplete multi-view clustering (IMVC) aims to address the clustering problem of multi-view data with partially missing samples and has received widespread attention in recent years. Most existing IMVC methods still have the following issues that require to be further addressed. They focus solely on the first-order correlation information among samples, neglecting the more intricate high-order connections. Additionally, these methods always overlook the noise or inaccuracies in the similarity matrix. To address above issues, a novel method named Robust Mixed-order Graph Learning (RMoGL) is proposed for IMVC. Specifically, to enhance the robustness to noise, the similarity matrices are separated into clean graphs and noise graphs. To capture complex high-order relationships among samples, the dynamic high-order similarity graphs are innovatively constructed from the recovered data. The clean graphs are endowed with mixed-order information and tend towards to obtain a consensus graph via a self-weighted manner. An efficient algorithm based on Alternating Direction Method of Multipliers (ADMM) is designed to solve the proposed RMoGL, and superior performance is demonstrated by compared with nine state-of-the-art methods across eight datasets. The source code of this work is available at https://github.com/guowei1314/RMoGL.

Feature decomposition and structural learning for multi-diverse and multi-view data clustering

Feature decomposition and structural learning for multi-diverse and multi-view data clustering

Graph neural networks for multi-view learning: a taxonomic review

With the explosive growth of user-generated content, multi-view learning has become a rapidly growing direction in pattern recognition and data analysis areas. Due to the significant application value of multi-view learning, there has been a continuous emergence of research based on machine learning methods and traditional deep learning paradigms. The core challenge in multi-view learning lies in harnessing both consistent and complementary information to forge a unified, comprehensive representation. However, many multi-view learning tasks are based on graph-structured data, making existing methods unable to effectively mine the information contained in the input multiple data sources. Recently, graph neural networks (GNN) techniques have been widely utilized to deal with non-Euclidean data, such as graphs or manifolds. Thus, it is essential to combine the advantages of the powerful learning capability of GNN models and multi-view data. In this paper, we aim to provide a comprehensive survey of recent research works on GNN-based multi-view learning. In detail, we first provide a taxonomy of GNN-based multi-view learning methods according to the input form of models: multi-relation, multi-attribute and mixed. Then, we introduce the applications of multi-view learning, including recommendation systems, computer vision and so on. Moreover, several public datasets and open-source codes are introduced for implementation. Finally, we analyze the challenges of applying GNN models on various multi-view learning tasks and state new future directions in this field.

MetsDB: a knowledgebase of cancer metastasis at bulk, single-celland spatial levels.

Cancer metastasis, the process by which tumour cells migrate and colonize distant organs from a primary site, is responsible for the majority of cancer-related deaths. Understanding the cellular and molecular mechanisms underlying this complex process is essential for developing effective metastasis prevention and therapy strategies. To this end, we systematically analysed 1786 bulk tissue samples from 13 cancer types, 988463 single cells from 17 cancer types, and 40252 spots from 45 spatial slides across 10 cancer types. The results of these analyses are compiled in the metsDB database, accessible at https://relab.xidian.edu.cn/metsDB/. This database provides insights into alterations in cell constitutions, cell relationships, biological pathways, molecular biomarkers, and drug responses during cancer metastasis at bulk, single-cell, and spatial levels. Users can perform cell or gene searches to obtain multi-view and multi-scale metastasis-related data. This comprehensive resource is invaluable for understanding the metastasis process and for designing molecular therapies.

Deep models for multi-view 3D object recognition: a review

This review paper focuses on the progress of deep learning-based methods for multi-view 3D object recognition. It covers the state-of-the-art techniques in this field, specifically those that utilize 3D multi-view data as input representation. The paper provides a comprehensive analysis of the pipeline for deep learning-based multi-view 3D object recognition, including the various techniques employed at each stage. It also presents the latest developments in CNN-based and transformer-based models for multi-view 3D object recognition. The review discusses existing models in detail, including the datasets, camera configurations, view selection strategies, pre-trained CNN architectures, fusion strategies, and recognition performance. Additionally, it examines various computer vision applications that use multi-view classification. Finally, it highlights future directions, factors impacting recognition performance, and trends for the development of multi-view 3D object recognition method.

Combining Dielectric and Hyperspectral Data for Apple Core Browning Detection

Apple core browning not only affects the nutritional quality of apples, but also poses a health risk to consumers. Therefore, there is an urgent need to develop a fast and reliable non-destructive detection method for apple core browning. To deal with the challenges of the long incubation period, strong infectivity, and difficulty in the prevention and control of apple core browning, a novel non-destructive detection method for apple core browning has been developed through combining hyperspectral imaging and dielectric techniques. To reduce the computational complexity of high-dimensional multi-view data, canonical correlation analysis is employed for feature dimensionality reduction. Then, the two low-dimensional vectors extracted from two different sensors are concatenated into one united feature vector; therefore, the information contained in the hyperspectral and dielectric data is fused to improve the detection accuracy of the non-destructive method. At last, five traditional classifiers, such as k-Nearest Neighbors, a support vector machine with radial basis function kernel and polynomial kernel, Decision Tree, and neural network, are trained on the fused feature vectors to discriminate apple core browning. The experimental results on our own constructed dataset have shown that the sensitivity, specificity, and precision of SVM with RBF kernel based on concatenated 70-dimensional feature vectors extracted via canonical correlation analysis reached 99.98%, 99.70%, and 99.70%, respectively, which achieved better results than other models. This study can provide theoretical assurance and technical support for further development of higher accuracy and lower-cost non-destructive detection devices for apple core browning.

Discriminative Deep Canonical Correlation Analysis for Multi-View Data.

Over the past few years, multimodal data analysis has emerged as an inevitable method for identifying sample categories. In the multi-view data classification problem, it is expected that the joint representation should include the supervised information of sample categories so that the similarity in the latent space implies the similarity in the corresponding concepts. Since each view has different statistical properties, the joint representation should be able to encapsulate the underlying nonlinear data distribution of the given observations. Another important aspect is the coherent knowledge of the multiple views. It is required that the learning objective of the multi-view model efficiently captures the nonlinear correlated structures across different modalities. In this context, this article introduces a novel architecture, termed discriminative deep canonical correlation analysis (D2CCA), for classifying given observations into multiple categories. The learning objective of the proposed architecture includes the merits of generative models to identify the underlying probability distribution of the given observations. In order to improve the discriminative ability of the proposed architecture, the supervised information is incorporated into the learning objective of the proposed model. It also enables the architecture to serve as both a feature extractor as well as a classifier. The theory of CCA is integrated with the objective function so that the joint representation of the multi-view data is learned from maximally correlated subspaces. The proposed framework is consolidated with corresponding convergence analysis. The efficacy of the proposed architecture is studied on different domains of applications, namely, object recognition, document classification, multilingual categorization, face recognition, and cancer subtype identification with reference to several state-of-the-art methods.

One-Step Multiview Fuzzy Clustering With Collaborative Learning Between Common and Specific Hidden Space Information.

Multiview data are widespread in real-world applications, and multiview clustering is a commonly used technique to effectively mine the data. Most of the existing algorithms perform multiview clustering by mining the commonly hidden space between views. Although this strategy is effective, there are two challenges that still need to be addressed to further improve the performance. First, how to design an efficient hidden space learning method so that the learned hidden spaces contain both shared and specific information of multiview data. Second, how to design an efficient mechanism to make the learned hidden space more suitable for the clustering task. In this study, a novel one-step multiview fuzzy clustering (OMFC-CS) method is proposed to address the two challenges by collaborative learning between the common and specific space information. To tackle the first challenge, we propose a mechanism to extract the common and specific information simultaneously based on matrix factorization. For the second challenge, we design a one-step learning framework to integrate the learning of common and specific spaces and the learning of fuzzy partitions. The integration is achieved in the framework by performing the two learning processes alternately and thereby yielding mutual benefit. Furthermore, the Shannon entropy strategy is introduced to obtain the optimal views weight assignment during clustering. The experimental results based on benchmark multiview datasets demonstrate that the proposed OMFC-CS outperforms many existing methods.

Hypergraph-Based Multi-View Action Recognition Using Event Cameras.

Action recognition from video data forms a cornerstone with wide-ranging applications. Single-view action recognition faces limitations due to its reliance on a single viewpoint. In contrast, multi-view approaches capture complementary information from various viewpoints for improved accuracy. Recently, event cameras have emerged as innovative bio-inspired sensors, leading to advancements in event-based action recognition. However, existing works predominantly focus on single-view scenarios, leaving a gap in multi-view event data exploitation, particularly in challenges like information deficit and semantic misalignment. To bridge this gap, we introduce HyperMV, a multi-view event-based action recognition framework. HyperMV converts discrete event data into frame-like representations and extracts view-related features using a shared convolutional network. By treating segments as vertices and constructing hyperedges using rule-based and KNN-based strategies, a multi-view hypergraph neural network that captures relationships across viewpoint and temporal features is established. The vertex attention hypergraph propagation is also introduced for enhanced feature fusion. To prompt research in this area, we present the largest multi-view event-based action dataset THUMV-EACT-50, comprising 50 actions from 6 viewpoints, which surpasses existing datasets by over tenfold. Experimental results show that HyperMV significantly outperforms baselines in both cross-subject and cross-view scenarios, and also exceeds the state-of-the-arts in frame-based multi-view action recognition.

An End-to-End Approach for Graph-Based Multi-View Data Clustering

An End-to-End Approach for Graph-Based Multi-View Data Clustering

Iterative Multiview Subspace Learning for Unpaired Multiview Clustering.

In real applications, several unpredictable or uncertain factors could result in unpaired multiview data, i.e., the observed samples between views cannot be matched. Since joint clustering among views is more effective than individual clustering in each view, we investigate unpaired multiview clustering (UMC), which is a valuable but insufficiently studied problem. Due to lack of matched samples between views, we could fail to build the connection between views. Therefore, we aim to learn the latent subspace shared by views. However, existing multiview subspace learning methods usually rely on the matched samples between views. To address this issue, we propose an iterative multiview subspace learning strategy [iterative unpaired multiview clustering (IUMC)], aiming to learn a complete and consistent subspace representation among views for UMC. Moreover, based on IUMC, we design two effective UMC methods: 1) Iterative unpaired multiview clustering via covariance matrix alignment (IUMC-CA) that further aligns the covariance matrix of subspace representations and then performs clustering on the subspace and 2) iterative unpaired multiview clustering via one-stage clustering assignments (IUMC-CY) that performs one-stage multiview clustering (MVC) by replacing the subspace representations with clustering assignments. Extensive experiments show the excellent performance of our methods for UMC, compared with the state-of-the-art methods. Also, the clustering performance of observed samples in each view can be considerably improved by those observed samples from the other views. In addition, our methods have good applicability in incomplete MVC.

Multiview Tensor Spectral Clustering via Co-Regularization.

Graph-based multi-view clustering encodes multi-view data into sample affinities to find consensus representation, effectively overcoming heterogeneity across different views. However, traditional affinity measures tend to collapse as the feature dimension expands, posing challenges in estimating a unified alignment that reveals both cross-view and inner relationships. To tackle this challenge, we propose to achieve multi-view uniform clustering via consensus representation co-regularization. First, the sample affinities are encoded by both popular dyadic affinity and recent high-order affinities to comprehensively characterize spatial distributions of the HDLSS data. Second, a fused consensus representation is learned through aligning the multi-view low-dimensional representation by co-regularization. The learning of the fused representation is modeled by a high-order eigenvalue problem within manifold space to preserve the intrinsic connections and complementary correlations of original data. A numerical scheme via manifold minimization is designed to solve the high-order eigenvalue problem efficaciously. Experiments on eight HDLSS datasets demonstrate the effectiveness of our proposed method in comparison with the recent thirteen benchmark methods.

Global calibration method for multi-view-based vibration measurement of large structures

A multi-view global calibration method is proposed to overcome the limitations of overlapping regions in multi-view structural vibration measurement. This method establishes a mapping relationship from image points to experimental model points at the outset of vibration measurement, enabling a one-time global calibration of multi-view vibration response data. Furthermore, an image point detection method based on shape-function matching is designed to improve global calibration accuracy. Experimental results demonstrate that the proposed method excels in point detection and global calibration. Finally, the feasibility of this method is validated through vibration experiments on large aircraft fuselage structures.

HIP: a method for high-dimensional multi-view data integration and prediction accounting for subgroup heterogeneity.

Epidemiologic and genetic studies in many complex diseases suggest subgroup disparities (e.g. by sex, race) in disease course and patient outcomes. We consider this from the standpoint of integrative analysis where we combine information from different views (e.g. genomics, proteomics, clinical data). Existing integrative analysis methods ignore the heterogeneity in subgroups, and stacking the views and accounting for subgroup heterogeneity does not model the association among the views. We propose Heterogeneity in Integration and Prediction (HIP), a statistical approach for joint association and prediction that leverages the strengths in each view to identify molecular signatures that are shared by and specific to a subgroup. We apply HIP to proteomics and gene expression data pertaining to chronic obstructive pulmonary disease (COPD) to identify proteins and genes shared by, and unique to, males and females, contributing to the variation in COPD, measured by airway wall thickness. Our COPD findings have identified proteins, genes, and pathways that are common across and specific to males and females, some implicated in COPD, while others could lead to new insights into sex differences in COPD mechanisms. HIP accounts for subgroup heterogeneity in multi-view data, ranks variables based on importance, is applicable to univariate or multivariate continuous outcomes, and incorporates covariate adjustment. With the efficient algorithms implemented using PyTorch, this method has many potential scientific applications and could enhance multiomics research in health disparities. HIP is available at https://github.com/lasandrall/HIP, a video tutorial at https://youtu.be/O6E2OLmeMDo and a Shiny Application at https://multi-viewlearn.shinyapps.io/HIP_ShinyApp/ for users with limited programming experience.

Multi-level multi-view network based on structural contrastive learning for scRNA-seq data clustering.

Clustering plays a crucial role in analyzing scRNA-seq data and has been widely used in studying cellular distribution over the past few years. However, the high dimensionality and complexity of scRNA-seq data pose significant challenges to achieving accurate clustering from a singular perspective. To address these challenges, we propose a novel approach, called multi-level multi-view network based on structural consistency contrastive learning (scMMN), for scRNA-seq data clustering. Firstly, the proposed method constructs shallow views through the $k$-nearest neighbor ($k$NN) and diffusion mapping (DM) algorithms, and then deep views are generated by utilizing the graph Laplacian filters. These deep multi-view data serve as the input for representation learning. To improve the clustering performance of scRNA-seq data, contrastive learning is introduced to enhance the discrimination ability of our network. Specifically, we construct a group contrastive loss for representation features and a structural consistency contrastive loss for structural relationships. Extensive experiments on eight real scRNA-seq datasets show that the proposed method outperforms other state-of-the-art methods in scRNA-seq data clustering tasks. Our source code has already been available at https://github.com/szq0816/scMMN.

Multi-view clustering with adaptive anchor and bipartite graph learning

Multi-view subspace clustering optimizes and integrates the graph structure information of each view. At present, the subspace clustering methods based on the abstract graph have better performance and improve the clustering results. Although the existing clustering algorithms have achieved excellent results, there are still four deficiencies: 1) Expensive time overhead, with most algorithms having a high time complexity; 2) Using fixed anchor points and the separation of anchor graph learning from subsequent graph construction, resulting in inadequate use of underlying graph structure between views; 3) Multi-view data have inevitable noise and in the original high-dimensional space data fusion may cause the loss of important information; 4) Most algorithms require additional post-processing to obtain clustering labels. To solve the above problems, we innovate a novel anchor-adaptive multi-view clustering algorithm based on a bipartite graph . Specifically, anchor graph learning and subspace graph construction are adaptively combined into a unified optimization framework. The projection matrix, consensus anchor matrix, and similarity matrix optimize each other to promote the clustering effect and structure a constrained bipartite graph to describe the relationship between representative points and sample points. At the same time, connectivity constraints are used to ensure that the connected components represent clusters directly. Our method has linear time complexity about sample size. Comparison experiments with the state-of-the-art algorithms demonstrate the effectiveness of the proposed method on various benchmark datasets. Moreover, our method is robust to noisy data and suitable for large-scale datasets.

Elastic deep multi-view autoencoder with diversity embedding

Current research on multi-view clustering (MVC) is pushing the boundaries of knowledge, allowing the extraction of valuable insights from various points of view. Recently, many researchers in this field have turned to deep learning techniques, particularly autoencoders (AE). These methods are highly effective at recognizing nonlinear and complex structures, making them a popular choice in this area. However, while they effectively handle Gaussian noise through the use of the Frobenius norm, they exhibit sensitivity to outlier data and Laplacian noise. Additionally, existing multi-view methodologies tend to usually rely on shared information across multi-view data, this means they often miss out on the diverse and complementary information each view can provide, leading in an inadequate utilization of the information available. In this paper, we present an innovative solution to address these challenges. Our novel Elastic Deep Multi-view Autoencoder with Diversity Embedding (EDMVAE-DE) method incorporates an Elastic Deep AE to enhance its robustness. Furthermore, we integrate an exclusivity constraint term to enhance the diversity of specific representations across distinct views, increasing both modeling consistency and diversity within a unified framework. To address problems arising from incorrectly classified neighbors, we also incorporate a graph constraint term. Our experiments, performed on various multi-view datasets, clearly show that EDMVAE-DE achieves the best performance in its class1.

NeRF-OR: neural radiance fields for operating room scene reconstruction from sparse-view RGB-D videos.

RGB-D cameras in the operating room (OR) provide synchronized views of complex surgical scenes. Assimilation of this multi-view data into a unified representation allows for downstream tasks such as object detection and tracking, pose estimation, and action recognition. Neural radiance fields (NeRFs) can provide continuous representations of complex scenes with limited memory footprint. However, existing NeRF methods perform poorly in real-world OR settings, where a small set of cameras capture the room from entirely different vantage points. In this work, we propose NeRF-OR, a method for 3D reconstruction of dynamic surgical scenes in the OR. Where other methods for sparse-view datasets use either time-of-flight sensor depth or dense depth estimated from color images, NeRF-OR uses a combination of both. The depth estimations mitigate the missing values that occur in sensor depth images due to reflective materials and object boundaries. We propose to supervise with surface normals calculated from the estimated depths, because these are largely scale invariant. We fit NeRF-OR to static surgical scenes in the 4D-OR dataset and show that its representations are geometrically accurate, where state of the art collapses to sub-optimal solutions. Compared to earlier work, NeRF-OR grasps fine scene details while training 30 faster. Additionally, NeRF-OR can capture whole-surgery videos while synthesizing views at intermediate time values with an average PSNR of 24.86dB. Last, we find that our approach has merit in sparse-view settings beyond those in the OR, by benchmarking on the NVS-RGBD dataset that contains as few as three training views. NeRF-OR synthesizes images with a PSNR of 26.72dB, a 1.7% improvement over state of the art. Our results show that NeRF-OR allows for novel view synthesis with videos captured by a small number of cameras with entirely different vantage points, which is the typical camera setting in the OR. Code is available via: github.com/Beerend/NeRF-OR .