Multi-view Data Research Articles

A self-supervised spatio-temporal attention network for video-based 3D infant pose estimation

General movement and pose assessment of infants is crucial for the early detection of cerebral palsy (CP). Nevertheless, most human pose estimation methods, in 2D or 3D, focus on adults due to the lack of large datasets and pose annotations on infants. To solve these problems, here we present a model known as YOLO-infantPose, which has been fine-tuned, for infant pose estimation in 2D. We further propose a self-supervised model called STAPose3D for 3D infant pose estimation based on videos. We employ multi-view video data during the training process as a strategy to address the challenge posed by the absence of 3D pose annotations. STAPose3D combines temporal convolution, temporal attention, and graph attention to jointly learn spatio-temporal features of infant pose. Our methods are summarized into two stages: applying YOLO-infantPose on input videos, followed by lifting these 2D poses along with respective confidences for every joint to 3D. The employment of the best-performing 2D detector in the first stage significantly improves the precision of 3D pose estimation. We reveal that fine-tuned YOLO-infantPose outperforms other models tested on our clinical dataset as well as two public datasets MINI-RGBD and YouTube-Infant dataset. Results from our infant movement video dataset demonstrate that STAPose3D effectively comprehends the spatio-temporal features among different views and significantly improves the performance of 3D infant pose estimation in videos. Finally, we explore the clinical application of our method for general movement assessment (GMA) in a clinical dataset annotated as normal writhing movements or abnormal monotonic movements according to the GMA standards. We show that the 3D pose estimation results produced by our STAPose3D model significantly boost the GMA prediction performance than 2D pose estimation. Our code is available at github.com/wwYinYin/STAPose3D.

View-interactive attention information alignment-guided fusion for incomplete multi-view clustering

Multi-view data from real applications is usually incomplete, leading to unsatisfactory performance in clustering tasks. Existing incomplete multi-view clustering (IMVC) methods still encounter the following issues: (1) They focus on learning consistency features between different views but often struggle to recover missing data based on the complementary features between views. (2) Differing from complete multi-view data, the misaligned characteristics of incomplete multi-view data may introduce noises in feature learning, thereby affecting the performance of the IMVC models. To address these issues, we propose a novel view-interactive attention information alignment-guided fusion (VAIAF) method for clustering incomplete multi-view data. Firstly, it utilizes an autoencoder to extract feature information from each view. Then we construct an information interactive attention (IIA) module to learn multi-view information by introducing the self-attention mechanism. This module not only extracts consistent features from different views but also utilizes the feature complementary relationships between views to recover missing data to a certain extent. Therefore, it can enhance the capability of the model to learn the consistency and complementarity of incomplete multi-view data. The proposed VAIAF method also constructs a discriminator for each view to facilitate missing data recovery. In the feature fusion stage, we implement an alignment-guided strategy to prevent noise caused by misaligned feature fusion. Extensive experiments on four incomplete multi-view datasets demonstrate the effectiveness of the proposed VAIAF method in clustering tasks.

Low-rank representation induced missing-view recovery for incomplete multi-view clustering

In practical applications, multi-view data suffers from incompleteness, which refers to the absence of some samples in each view. Incomplete multi-view clustering is developed to capture the common cluster structure across incomplete views. However, some existing IMC methods ignore the missing samples or the latent relationship between the available and the unavailable samples. We propose a novel missing-view recovery strategy to overcome these two drawbacks. Considering that each missing sample and part of the observable samples belong to the same category, we recover each missing sample via the linear combination of observed samples within each view. Based on the semantic consistency, we extract a unified self-representation graph from complete views and impose a low-rank constraint on it. Furthermore, we learn a comprehensive representation from complete multi-view data and introduce the dynamic weights to consider the importance of various views. The experimental results on several datasets demonstrate that the proposed method outperforms the state-of-the-art IMC approaches.

Clean and robust multi-level subspace representations learning for deep multi-view subspace clustering

Deep multi-view subspace clustering has achieved increasing attention owing to its encouraging ability to address the nonlinear multi-view data. Despite significant progress, most existing deep multi-view subspace clustering methods still suffer from the following drawbacks. First, they usually focus on utilizing the last-level feature from the last encoder layer of the auto-encoder related to each view, yet often overlook exploiting the earlier-level features from the earlier encoder layers as well as integrating the multi-level features from multiple encoder layers. Second, when learning the subspace representation from the self-expressive layer, they rarely consider the cleanliness and robustness of the learned subspace representation. To overcome these drawbacks, this paper proposes a novel clean and robust multi-level subspace representations learning for deep multi-view subspace clustering (CRMSRL). Specifically, the auto-encoder is adopted to nonlinearly map each view into the multi-level features, and then multiple self-expressive layers are added between the encoder layers and their corresponding decoder layers to provide multiple information flow paths through the auto-encoder and learn the multi-level subspace representations corresponding to the multi-level features. Consequently, the intricate hierarchical information embedded in the multi-level features is passed to the corresponding multi-level subspace representations. Moreover, with the idea of robust principal component analysis (RPCA), the learned multi-level subspace representations can be further purified to achieve the cleaner and more robust ones. In addition, to excavate the structural consistency among different views, the common layer is designed to interact with different view-specific self-expressive layers and achieve the high-quality common subspace representation, which can be applied to the spectral clustering algorithm to obtain the final clustering results. Experimental results show that CRMSRL outperforms twelve state-of-the-art baseline methods on six benchmark datasets. In particular, CRMSRL achieves 4.5% improvements on the MSRCV1 dataset in terms of accuracy, compared with the best baseline method.

A fast scene geometric modeling approach for digital twins combining neural rendering and model retrieval

ABSTRACT Digital twin technology is a crucial driving technology for the realization of Industry 4.0, which enables simulation, analysis, and prediction by constructing geometric scenes that correspond to physical scenes. Current automation in scene modeling is limited, relying heavily on costly and inefficient manual operations. This hinders digital twin technology progress. This paper proposes a framework for a fast scene geometric modeling method that combines neural rendering and model retrieval for digital twins. Neural rendering techniques first train the collected image data, and the point cloud data of physical entities are rendered. Then based on the semantic mapping algorithm between the point cloud data of physical entities and the multi-view image data of 3D CAD models proposed in this study, the corresponding geometric models are retrieved from the geometric asset library by inputting the point cloud data, and all the retrieved geometric models are embedded in the geometric scene to complete the scene geometric modeling. Finally, in a case study of the digital twin-based scene construction for decommissioned lithium battery dismantling, the effectiveness of this method is demonstrated, which can improve the speed and reduce the cost of scene geometric modeling.

Contrastive BiLSTM-enabled Health Representation Learning for Remaining Useful Life Prediction

Remaining useful life (RUL) prediction is of vital significance in prognostics health management tasks. Due to powerful learning capabilities, deep learning methods, particularly long short-term memory (LSTM) have been widely applied in RUL prediction. However, many existing deep learning approaches overlook the inherent ordered relationship between samples in the direct mapping from sliced data to RUL pattern. To capture the faithful and ordered health representation of a given system, a Contrastive Bidirectional LSTM-enabled Health Representation Learning (CBHRL) framework is proposed. Firstly, the supervised contrastive regression loss (SupCR) is implemented to extract continuous health representation. The SupCR is designed to rank the similarity among health representations from different samples, prompting them highly correlated with linear RUL label. Among the process of contrastive learning, the series odd-even decomposition (SOED) method is devised to construct multi-view degradation data, which improves generalization ability. Finally, since the health representation is constructed on basis of similarity, a new similarity prediction method is proposed as the complement of regression prediction method. Experimental results show the health representations extracted by CBHRL achieve improved ratio ranging from a minimum of 17.19% to a maximum of 291.30% in monotonicity, smoothness and trendability.

Multi-view fair-augmentation contrastive graph clustering with reliable pseudo-labels

Graph contrastive clustering (GCC) has achieved numerous advantageous results due to the information mining capability of self-supervised learning. Multi-view attribute graph clustering, as a means of addressing complex attribute graph data with multi-view relations or features, has gained significant attention in recent years. However, only a few GCC methods are applicable to multi-view graph data, limiting the full utilization of the rich information contained in such data. Furthermore, most GCC methods rely on random augmentation strategies, leading to structural unfairness in the augmentation process across views. To address these issues, we propose multi-view fair-augmentation contrastive graph clustering with reliable pseudo-labels (MFCGC), which combines contrastive learning and fully utilizes the node-level and cluster-level information within each view of the multi-view data, ensuring the consistency of information across different views. Moreover, we designed a carefully crafted node-degree-based augmentation method called fair augmentation, which preserves the partial topology structure of multi-view data. Finally, we propose a reliable pseudo-label selection mechanism that integrates reliable pseudo-labels from multi-view graph to improve the quality of sample pairs. Our proposed MFCGC has been extensively evaluated on real-world datasets, demonstrating its superiority over state-of-the-art algorithms in two categories of multi-view graph data and attesting to its significant effectiveness.

Deep embedding based tensor incomplete multi-view clustering

The majority of multi-view data are extracted from real life, which often lose information in some views. To solve this problem, existing incomplete multi-view clustering algorithms explore the valuable information from incomplete data while populating the missing information. Nevertheless, they suffer from the following two limitations: 1) The non-linear relationships of high-dimensional available data are not considered. 2) Although some methods utilize information from different views to fill in missing information, they still cannot precisely explore the missing information of each view. To this end, this article proposes a novel one-step incomplete multi-view framework, referred to as deep embedding based tensor incomplete multi-view clustering (DETIMC). Concretely, in this framework, the high-dimensional available data are projected into the low-dimensional embedding space by deep non-negative matrix factorization (NMF), which can obtain a clean space while capturing the complex non-linear relationship of available data. Moreover, a novel double-enhanced missing-view inferring strategy is developed, in which the weighted higher-order information of different views and the clustering structure information are simultaneously exploited. Comprehensive experiments on several benchmark datasets exhibit the superiority of DETIMC over conventional and state-of-the-art algorithms.

Composite attention mechanism network for deep contrastive multi-view clustering

Contrastive learning-based deep multi-view clustering methods have become a mainstream solution for unlabeled multi-view data. These methods usually utilize a basic structure that combines autoencoder, contrastive learning, or/and MLP projectors to generate more representative latent representations for the final clustering stage. However, existing deep contrastive multi-view clustering ignores two key points: (i) the latent representations projecting from one or more layers of MLP or new representations directly obtained from autoencoder fail to mine inherent relationship inner-view or cross-views; (ii) more existing frameworks only employ a one or dual-contrastive learning module, i.e., view- or/and category-oriented, which may result in the lack of communication between latent representations and clustering assignments. This paper proposes a new composite attention framework for contrastive multi-view clustering to address the above two challenges. Our method learns latent representations utilizing composite attention structure, i.e., Hierarchical Transformer for each view and Shared Attention for all views, rather than simple MLP. As a result, the learned representations can simultaneously preserve important features inside the view and balance the contributions across views. In addition, we add a new communication loss in our new dual contrastive framework. The common semantics will be brought into clustering assignments by pushing clustering assignments closer to the fused latent representations. Therefore, our method will provide a higher quality of clustering assignments for the segmentation problem of unlabeled multi-view data. The extensive experiments on several real data demonstrate that the proposed method can achieve superior performance over many state-of-the-art clustering algorithms, especially the significant improvement of an average of 10% on datasets Caltech and its subsets according to accuracy.

Comprehensive Multiview Representation Learning via Deep Autoencoder-Like Nonnegative Matrix Factorization.

Learning a comprehensive representation from multiview data is crucial in many real-world applications. Multiview representation learning (MRL) based on nonnegative matrix factorization (NMF) has been widely adopted by projecting high-dimensional space into a lower order dimensional space with great interpretability. However, most prior NMF-based MRL techniques are shallow models that ignore hierarchical information. Although deep matrix factorization (DMF)-based methods have been proposed recently, most of them only focus on the consistency of multiple views and have cumbersome clustering steps. To address the above issues, in this article, we propose a novel model termed deep autoencoder-like NMF for MRL (DANMF-MRL), which obtains the representation matrix through the deep encoding stage and decodes it back to the original data. In this way, through a DANMF-based framework, we can simultaneously consider the multiview consistency and complementarity, allowing for a more comprehensive representation. We further propose a one-step DANMF-MRL, which learns the latent representation and final clustering labels matrix in a unified framework. In this approach, the two steps can negotiate with each other to fully exploit the latent clustering structure, avoid previous tedious clustering steps, and achieve optimal clustering performance. Furthermore, two efficient iterative optimization algorithms are developed to solve the proposed models both with theoretical convergence analysis. Extensive experiments on five benchmark datasets demonstrate the superiority of our approaches against other state-of-the-art MRL methods.

Energy Conservation in WVSNs With Multiview Data Streaming via Joint Encoding in Measurement Domain

Energy Conservation in WVSNs With Multiview Data Streaming via Joint Encoding in Measurement Domain

NOODLE: Joint Cross-View Discrepancy Discovery and High-Order Correlation Detection for Multi-View Subspace Clustering

Benefiting from the effective exploration of the valuable topological pair-wise relationship of data points across multiple views, multi-view subspace clustering (MVSC) has received increasing attention in recent years. However, we observe that existing MVSC approaches still suffer from two limitations that need to be further improved to enhance the clustering effectiveness. Firstly, previous MVSC approaches mainly prioritize extracting multi-view consistency, often neglecting the cross-view discrepancy that may arise from noise, outliers, and view-inherent properties. Secondly, existing techniques are constrained by their reliance on pair-wise sample correlation and pair-wise view correlation, failing to capture the high-order correlations that are enclosed within multiple views. To address these issues, we propose a novel MVSC framework via joi N t cr O ss-view discrepancy disc O very an D high-order corre L ation d E tection ( NOODLE ), seeking an informative target subspace representation compatible across multiple features to facilitate the downstream clustering task. Specifically, we first exploit the self-representation mechanism to learn multiple view-specific affinity matrices, which are further decomposed into cohesive factors and incongruous factors to fit the multi-view consistency and discrepancy, respectively. Additionally, an explicit cross-view sparse regularization is applied to incoherent parts, ensuring the consistency and discrepancy to be precisely separated from the initial subspace representations. Meanwhile, the multiple cohesive parts are stacked into a three-dimensional tensor associated with a tensor-Singular Value Decomposition (t-SVD) based weighted tensor nuclear norm constraint, enabling effective detection of the high-order correlations implicit in multi-view data. Our proposed method outperforms state-of-the-art methods for multi-view clustering on six benchmark datasets, demonstrating its effectiveness.

Interface investigations of 2A14Al/Nb explosively welded composites with 1060Al and Ag interlayers: Morphologies, grains, and mechanical properties

Explosive welding (EW) with an interlayer is considered a suitable technology for producing Al/Nb layered composite, but the fundamental process behaviors of this method remain insufficient discussion. Here, contrast EW experiments respectively using 1060Al and Ag sheets as the interlayers were carried out to prepare 2A14Al/Nb composites, and the complex interfacial phenomena and the underlying mechanisms were thoroughly examined by advanced characterizations and theoretical analyses. The microstructure results indicated that the interlayer materials had significant influence on the bonding morphologies, but the defects of the two composites shared the same geometry and distribution characteristics. Combining the microstructures and thermodynamic history, the formation mechanisms of the different defects were revealed. The 2A14Al/Ag/Nb and 2A14Al/1060Al/Nb composites had the advantages of no intermetallic compounds and few defects, respectively, but the latter showed a higher bond strength, suggesting that the geometric defects had more negative effects on mechanical properties than intermetallic compounds. The multi-view EBSD data provided insights into the intricate grain evolutions induced by the EW process, including equiaxed 2A14Al grains and columnar 1060Al grains at the 2A14Al/1060Al interface, and nanoscale Al3Nb grains at the 1060Al/Nb interface. Finally, the correlations between the grain structures and mechanical properties were established by nanoindentation tests.

Adaptive multi-view subspace learning based on distributed optimization

As the rapid development of Internet of Things (IoT), information is collected from different sensors and stored in distributed devices which can be regarded as the multi-view data. There are currently numerous clustering algorithms designed to handle multi-view data. However, most of these algorithms still suffer from the following problems: They are designed to operate directly on raw data, which preserves excessive redundant information. They primarily focus on pairwise relationships between views, neglecting the intricate high-order connections among multiple views. The prior information of singular values is not taken into account in multiple views and different views are considered to have equal contributions for clustering. To efficiently address above problems, adaptive multi-view subspace learning based on distributed optimization (AMSLDO) is proposed in this paper. Specifically, the original multi-view data is projected to a low-dimensional space for subspace representation, and multiple representation matrices are stacked in a tensor with weighted tensor nuclear norm to obtain high-order correlations and discover the prior information of singular values. Furthermore, the consensus matrix is learned from different representation matrices through adaptive weights. Meanwhile, samples are partitioned into the ideal number of clusters through Laplacian rank constraint. An efficient distributed optimization algorithm based on the Alternating Direction Method of Multipliers (ADMM) framework is designed to solve the proposed model. Extensive experiments are conducted on six datasets, demonstrating the superiority of the proposed model compared with eleven state-of-the-art methods.

Comprehensive multi-view self-representations for clustering

Subspace learning-based methods have shown excellent performance for multi-view clustering, yet have the following problems: (1) most existing methods obtain the subspace representation from the original space, which might contain noises and cannot guarantee a clean enough subspace representation; (2) existing methods mainly focus on the consistency of the subspace representation, while the unique information of each view is not sufficiently exploited. To solve these two problems, we propose a novel multi-view subspace clustering method called comprehensive multi-view self-representations (CMSR). Specifically, we learn the original coefficient matrix of each view through the self-representation, which can reduce the noise of the original space to some extent. Then, we learn the subspace representation of the original coefficient matrix and decompose it into a consistent coefficient matrix and multiple diverse coefficient matrices, which can exploit the consistent and complementary information of multi-view data. Further, we impose the Schatten p-norm constraint on the consistent coefficient matrix to capture robust consistent information. Finally, the comprehensive results on eight real datasets demonstrate the versatility and effectiveness of the proposed method.

Text-image conditioned diffusion for consistent text-to-3D generation

By lifting the pre-trained 2D diffusion models into Neural Radiance Fields (NeRFs), text-to-3D generation methods have made great progress. Many state-of-the-art approaches usually apply score distillation sampling (SDS) to optimize the NeRF representations, which supervises the NeRF optimization with pre-trained text-conditioned 2D diffusion models such as Imagen. However, the supervision signal provided by such pre-trained diffusion models only depends on text prompts and does not constrain the multi-view consistency. To inject cross-view consistency into diffusion priors, some recent works finetune the 2D diffusion model via multi-view data, but still lack fine-grained view coherence. To tackle this challenge, we incorporate multi-view image conditions into the supervision signal of NeRF optimization, which explicitly enforces fine-grained view consistency. With such stronger supervision, our proposed text-to-3D method effectively mitigates the generation of floaters (due to excessive densities) and completely empty spaces (due to insufficient densities). Our quantitative evaluations on the T3Bench dataset demonstrate that our method achieves state-of-the-art performance over existing text-to-3D methods. We will make the code publicly available.

Dual-level feature assessment for unsupervised multi-view feature selection with latent space learning

Recently, numerous unsupervised multi-view feature selection methods have been presented. However, these methods assess the significance of data features in each view individually or jointly evaluate the significance of data features across multiple views by concatenating data features from all data views. So, they disregard either inter-view or intra-view feature connections of data. Moreover, they ignore the complementary information from multiple insufficient views, each of which only captures partial information of multi-view data and is insufficient to characterize data distribution. To address these problems, this paper proposes a model named Dual-level Feature Assessment for Unsupervised Multi-view Feature Selection with Latent Space Learning (DFA-LSL). It first investigates the underlying complementary information from multiple views in a latent space while simultaneously learning latent representations for data. Compared with the original data representations, the latent representations contain the entire information of multi-view data. Then, it explores connections and distinguishing ability of features at two levels, namely the inter-view and intra-view levels, by jointly assessing the significance of features in the latent space and individually assessing the significance of features in each view concurrently. Following that, an effective optimization algorithm is presented. Extensive experiments demonstrate that the proposed work surpasses several state-of-the-art models.

Optimal and Multi-View Strategic Hybrid Deep Learning for Old Landslide Detection in the Loess Plateau, Northwest China

Old landslides in the Loess Plateau, Northwest China usually occurred over a relatively long period, and their sizes are usually smaller compared to old landslides in the alpine valley areas of Sichuan, Yunnan, and Southeast Tibet. These landslide areas may have been changed either partially or greatly, and they are usually covered with vegetation and similar to their surrounding environment. Therefore, it is a great challenge to detect them using high-resolution remote sensing images with only orthophoto view. This paper proposes the optimal-view and multi-view strategic hybrid deep learning (OMV-HDL) method for old loess landslide detection. First, the optimal-view dataset in the Yan’an area (YA-OP) was established to solve the problem of insufficient optical features in orthophoto images. Second, in order to make the process of interpretation more labor-saving, the optimal-view and multi-view (OMV) strategy was proposed. Third, hybrid deep learning with weighted boxes fusion (HDL-WBF) was proposed to detect old loess landslides effectively. The experimental results with the constructed optimal-view dataset and multi-view data show that the proposed method has excellent performance among the compared methods—the F1 score and AP (mean) of the proposed method were improved by about 30% compared with the single detection model using traditional orthophoto-view data—and that it has good detection performance on multi-view data with the recall of 81.4%.

Research on multi-view clustering algorithm based on sequential three-way decision

Multi-view clustering has achieved many applications in recent years. But the existing multi-view clustering methods face two problems, firstly, the traditional multi-view clustering uses a hard clustering method, which cannot describe the uncertainty between the samples and the clusters, and secondly how to perform effective incremental learning on multi-view data when the number of views increases. Therefore to address the above two problems, we firstly propose a three-way fuzzy spectral clustering algorithm based on fuzzy clustering, which can perform three-way clustering on multi-views and get soft clustering results, thus describing the uncertainty between samples and clusters. Then based on the sequential decision making approach, an incremental learning mechanism is designed for multi-view clustering when the number of views increases. Finally, these two works are combined to propose a multi-view clustering algorithm based on sequential three-way decision making. The experimental results demonstrate that the method proposed in this paper has better clustering accuracy and efficiency compared to the traditional multi-view clustering algorithm.

Information granule optimization and co-training based on kernel method

Co-training was originally designed for multi-view data. Subsequent theoretical research has extended co-training to the application of single-view data. The construction of a feature subspace is one of the methods to expand single-view data into multi-view data, so the establishment of a feature subspace is the key to this approach. In this paper, the kernel method is used to form an implicit feature subspace, to perform multi-view co-training on single-view data. Because the attribute value in the feature subspace is not known, the subspace is a pseudo-view. To make the view adapt to the base classifier, this paper uses the neighborhood classifier as the base classifier and proposes an adaptive kernel function and three kernel parameter optimization methods according to the characteristics of the neighborhood classifier to build the feature subspace adapted to the neighborhood classifier. The decision of the neighborhood classifier needs to be made based on information granules generated by unlabeled objects. In the iteration of the feature subspace, we can continuously learn and optimize the information granule, and finally form what we expect, to get the implicit feature space corresponding to the granule and improve the accuracy of the base classifier. Finally, taking five data sets from UCI, and using accuracy and F1-score as evaluation indicators, we conduct a comparative experiment. The experimental results show that an adaptive kernel function, three kernel parameter optimization methods, and the co-training method presented in this paper are effective.