Structure Of High-dimensional Data Research Articles

Change point detection in high dimensional covariance matrix using Pillai’s statistics

AbstractThis research proposes a method to test and estimate change points in the covariance structure of high-dimensional multivariate series data. Our method uses the trace of the beta matrix, known as Pillai’s statistics, to test the change in covariance matrix at each time point. We study the asymptotic normality of Pillai’s statistics for testing the equality of two covariance matrices when both sample size and dimension increase at the same rate. We test the existence of a single change point in a given time period using Cauchy combination test, the test using an weighted sum of Cauchy transformed p-values, and estimate the change point as the point whose statistic is the greatest. To test and estimate multiple change points, we use the idea of the wild binary segmentation and repeatedly apply the procedure for a single change point to each segmented period until no significant change point exists. We numerically provide the size and power of our method. We finally apply our procedure to finding abnormal behavior in the investment of a private equity fund.

High-Dimensional Block Diagonal Covariance Structure Detection Using Singular Vectors

The assumption of independent subvectors arises in many aspects of multivariate analysis. In most real-world applications, however, we lack prior knowledge about the number of subvectors and the specific variables within each subvector. Yet, testing all these combinations is not feasible. For example, for a data matrix containing 15 variables, there are already 1 382 958 545 possible combinations. Given that zero correlation is a necessary condition for independence, independent subvectors exhibit a block diagonal covariance matrix. This paper focuses on the detection of such block diagonal covariance structures in high-dimensional data and therefore also identifies uncorrelated subvectors. Our approach exploits the fact that the structure of the covariance matrix is mirrored by the structure of its eigenvectors. However, the true block diagonal structure is masked by noise in the sample case. To address this problem, we propose to use sparse approximations of the sample eigenvectors to reveal the sparse structure of the population eigenvectors. Notably, the right singular vectors of a data matrix with an overall mean of zero are identical to the sample eigenvectors of its covariance matrix. Using sparse approximations of these singular vectors instead of the eigenvectors makes the estimation of the covariance matrix obsolete. We demonstrate the performance of our method through simulations and provide real data examples. Supplementary materials for this article are available online.

Shrinkage estimation of gene interaction networks in single-cell RNA sequencing data

BackgroundGene interaction networks are graphs in which nodes represent genes and edges represent functional interactions between them. These interactions can be at multiple levels, for instance, gene regulation, protein-protein interaction, or metabolic pathways. To analyse gene interaction networks at a large scale, gene co-expression network analysis is often applied on high-throughput gene expression data such as RNA sequencing data. With the advance in sequencing technology, expression of genes can be measured in individual cells. Single-cell RNA sequencing (scRNAseq) provides insights of cellular development, differentiation and characteristics at the transcriptomic level. High sparsity and high-dimensional data structures pose challenges in scRNAseq data analysis.ResultsIn this study, a sparse inverse covariance matrix estimation framework for scRNAseq data is developed to capture direct functional interactions between genes. Comparative analyses highlight high performance and fast computation of Stein-type shrinkage in high-dimensional data using simulated scRNAseq data. Data transformation approaches also show improvement in performance of shrinkage methods in non-Gaussian distributed data. Zero-inflated modelling of scRNAseq data based on a negative binomial distribution enhances shrinkage performance in zero-inflated data without interference on non zero-inflated count data.ConclusionThe proposed framework broadens application of graphical model in scRNAseq analysis with flexibility in sparsity of count data resulting from dropout events, high performance, and fast computational time. Implementation of the framework is in a reproducible Snakemake workflow https://github.com/calathea24/ZINBGraphicalModel and R package ZINBStein https://github.com/calathea24/ZINBStein.

Quantitative characterization and application of deep learning carbonate reservoir

Abstract Quantitative reservoir characterization is an important research task in carbonate reservoir development. At present, the comprehensive evaluation of drilling, logging, seismic, and productivity testing data is usually used, which requires a large number of test data and tedious manual data collation. Deep learning technology can independently learn complex reservoir characteristics, has a strong ability for high-dimensional data structure mining, and has a strong advantage in solving complex nonlinear reservoir description problems closely related to various geological characteristics. Therefore, this paper applies deep learning technology to reservoir characterization. First, a deep neural network model is constructed, and then seven characteristic parameters such as fracture density, fracture opening, fracture porosity, matrix porosity, total porosity, effective thickness, and specific oil production index are taken as input variables. After model training and verification, the effective permeability distribution map of the reservoir in the target area can be automatically generated. The results show that the deep learning method has strong adaptability and practicability for the quantitative characterization of complex reservoirs, and provides a new way of thinking for the quantitative characterization of reservoirs.

Model-based multifacet clustering with high-dimensional omics applications.

High-dimensional omics data often contain intricate and multifaceted information, resulting in the coexistence of multiple plausible sample partitions based on different subsets of selected features. Conventional clustering methods typically yield only one clustering solution, limiting their capacity to fully capture all facets of cluster structures in high-dimensional data. To address this challenge, we propose a model-based multifacet clustering (MFClust) method based on a mixture of Gaussian mixture models, where the former mixture achieves facet assignment for gene features and the latter mixture determines cluster assignment of samples. We demonstrate superior facet and cluster assignment accuracy of MFClust through simulation studies. The proposed method is applied to three transcriptomic applications from postmortem brain and lung disease studies. The result captures multifacet clustering structures associated with critical clinical variables and provides intriguing biological insights for further hypothesis generation and discovery.

A Comparative Analysis of Machine Learning Algorithms for Identifying Cultural and Technological Groups in Archaeological Datasets through Clustering Analysis of Homogeneous Data

Machine learning algorithms have revolutionized data analysis by uncovering hidden patterns and structures. Clustering algorithms play a crucial role in organizing data into coherent groups. We focused on K-Means, hierarchical, and Self-Organizing Map (SOM) clustering algorithms for analyzing homogeneous datasets based on archaeological finds from the middle phase of Pre-Pottery B Neolithic in Southern Levant (10,500–9500 cal B.P.). We aimed to assess the repeatability of these algorithms in identifying patterns using quantitative and qualitative evaluation criteria. Thorough experimentation and statistical analysis revealed the pros and cons of each algorithm, enabling us to determine their appropriateness for various clustering scenarios and data types. Preliminary results showed that traditional K-Means may not capture datasets’ intricate relationships and uncertainties. The hierarchical technique provided a more probabilistic approach, and SOM excelled at maintaining high-dimensional data structures. Our research provides valuable insights into balancing repeatability and interpretability for algorithm selection and allows professionals to identify ideal clustering solutions.

Reconstruction and denoising of high-dimensional seismic data via Frobenius-nuclear mixed norm constraints

Abstract The seismic data acquisition design with ‘two-wide and one-high’ geometry effectively improves the imaging quality of seismic records. However, when data are acquired in the field, complex near surface conditions and environmental factors can introduce a variety of noises and gaps in seismic data, impacting the accuracy of seismic imaging. Currently, the method of low-rank matrix/tensor completion is commonly employed for data reconstruction after normal moveout (NMO). In a complex subsurface medium, common midpoint data processed with NMO may not satisfy the linear or quasi-linear assumptions within local data windows. Therefore, this paper exploits the inherent low-rank structure of high-dimensional data to propose a high-dimensional tensor completion method under the Frobenius-nuclear mixed norm constraint (FN-TC). This method unfolds the 4D data tensor into the frequency-space domain along its modes (m, n) and subsequently imposes a non-convex Frobenius-nuclear mixed norm constraint on the unfolded approximate matrices. This approach closely approximates the rank function of the factor matrices, thereby enhancing the accuracy of data modeling. Theoretical and practical studies demonstrate that the novel FN-TC approach can effectively reconstruct high-dimensional seismic data and suppress noise, thereby providing data support for subsequent high-precision seismic imaging.

Joint Projected Fuzzy Neighborhood Preserving C-means Clustering with Local Adaptive Learning

Graph-based dimensionality reduction approaches are extensively applied for preserving the intrinsic structures of high-dimensional data. To capture discriminative information from data clusters that lack a clear manifold structure, previous studies have combined clustering algorithms with local manifold structure preservation. These methods assume that two neighbor data points share the same cluster label. However, the assumption may not hold for Fuzzy C-Means which may assign two nearby data points to different clusters depending on the distance between data points and cluster centroids. This inconsistency degrades the performance in clustering and dimensionality reduction. To tackle this issue, we propose a novel approach for dimensionality reduction, termed joint Projected Fuzzy Neighborhood Preserving C-means Clustering with Local Adaptive Learning (PFNPCM). PFNPCM learns cluster centroids, sparse membership grades, embedded subspaces, and structured graphs simultaneously and extracts both global and local structures of data. Through clustering data that preserves the local neighborhood structure in a low-dimensional space instead of the original one, PFNPCM can effectively capture both global and local information from the inherent manifold structure and cluster structure. The employment of ℓ2,1-norm as the distance metric in the clustering and the preservation of neighborhood structure further ensure the robustness of clustering results against noise or outliers. To achieve the consistency of the captured global and local discriminative information, PFNPCM adopts a novel fuzzy local similarity measure based on the manifold structure, and the manifold assumption thus holds when partitioning unlabeled samples into subgroups. Massive experiments on diverse well-known datasets demonstrate that PFNPCM is superior to some state-of-the-art methods.

Nonparametric High-Dimensional Multi-Sample Tests based on Graph Theory

High-dimensional data pose unique challenges for data processing in an era of ever-increasing amounts of data availability. Graph theory can provide a structure of high-dimensional data. We introduce two key properties desirable for graphs in testing homogeneity. Roughly speaking, these properties may be described as: unboundedness of edge counts under the same distribution and boundedness of edge counts under different distributions. It turns out that the minimum spanning tree violates these properties but the shortest Hamiltonian path posses them. Based on the shortest Hamiltonian path, we propose two combinations of edge counts in multiple samples to test for homogeneity. We give the permutation null distributions of proposed statistics when sample sizes go to infinity. The power is analyzed by assuming both sample sizes and dimensionality tend to infinity. Simulations show that our new tests behave very well overall in comparison with various competitors. Real data analysis of tumors and images further convince the value of our proposed tests. Software implementing the test is available in the R package GRelevance. Supplemental materials for this article are available online.

Sparse multi-view image clustering with complete similarity information

Multi-view image clustering aims to efficiently divide the collection of images by studying the characteristics of different views. Many studies performed Laplacian dimensionality reduction on the original image to avoid noise interference in constructing the similarity matrix. However, they ignored the problem that local similarity information and isolated image information are lost due to dimensionality reduction. These problems will result in the similarity matrix being insufficient to accurately describe the similarity between images, which in turn affects the clustering accuracy. Therefore, we propose a sparse multi-view spectral clustering model with complete similarity information between images (SMSC-CSI). The model combines the original image space and the low-dimensional spectral embedding space based on the adaptive neighbors method to learn the initial similarity matrices of each view jointly. On the one hand, the original space can retain all the similarity information between images so that the initial similarity matrix can accurately describe the similarity between images, which is conducive to accurate clustering. On the other hand, the low-dimensional space can avoid noise interference and retain the main structure of high-dimensional data, improving the robustness of the initial similarity matrix. Meanwhile, to ensure consistency among the views, the model minimizes the difference between the initial similarity matrix and the central fusion matrix by alternately updating to obtain the optimal weights and central fusion matrix for each view. Finally, we can obtain ideal clustering results directly with low model complexity and without post-processing steps such as K-means by adding non-negative Laplace rank constraints and ℓ0-norm constraints to the model objective function. Experimental results on different real image datasets show that SMSC-CSI can outperform some traditional clustering models and recent models on multi-view image clustering.

Topo: Towards a fine-grained topological data processing framework on Tianhe-3 supercomputer

Big data frameworks are widely deployed in supercomputers for analyzing large-scale datasets. Topological data processing is an emerging approach that focuses on analyzing the topological structures in high-dimensional scientific data. However, incorporating topological data processing into current big data frameworks presents three main challenges: (1) The frequent data exchange poses challenges to the traditional coarse-grained parallelism. (2) The spatial topology makes parallel programming harder using oversimplified MapReduce APIs. (3) The massive intermediate data and NUMA architecture hinder resource utilization and scalability on novel supercomputers and many-core processors.In this paper, we present Topo, a generic distributed framework that enhances topological data processing on many-core supercomputers. Topo relies on three concepts. (1) It employs fine-grained parallelism, with awareness of topological structures in datasets, to support interactions among collaborative workers before each shuffle phase. (2) It provides intuitive APIs for topological data operations. (3) It implements efficient collective I/O and NUMA-aware dynamic task scheduling to improve multi-threading and load balancing. We evaluate Topo's performance on the Tianhe-3 supercomputer, which utilizes state-of-the-art ARM many-core processors. Experimental results of execution time show that compared to popular frameworks, Topo achieves an average speedup of 5.3× and 6.3×, with a maximum speedup of 8.4× and 20×, on HPC workloads and big data benchmarks, respectively. Topo further reduces total execution time on processing skewed datasets by 41%.

Nonconvex submodule clustering via joint sliced sparse gradient and cluster-aware approach

Most existing subspace clustering methods preprocess image data by converting them into vectors, which lacks exploration of the spatial structure of high-dimensional data. Therefore, we proposes a nonconvex submodule clustering model (NSSGCA) via joint sliced sparse gradient and cluster-aware approach. NSSGCA arranges each 2D image as lateral slices of a 3rd-order tensor, and utilizes the t-product under the model of the union of free submodules to represent 3rd-order tensor samples, thereby exploring the latent spatial structure of samples. To more accurately approximate tensor rank, a nonconvex Schatten p-norm constraint is imposed on the rotated representation tensor. Under the submodule framework, a consistent gradient matrix is derived based on the δ-nearest neighbor adjacency graph to construct sliced sparse gradient (SSG) regularization, which is more conducive to clustering tasks. NSSGCA learns representation tensor with clearer block-like structure based on ℓq norm and cluster-aware attention mechanism. The convergence of the constructed sequence to the stationary Karush–Kuhn–Tucker (KKT) point is proven. Experimental results on real-world image datasets confirm the effectiveness of NSSGCA.

Optical fiber sensing and tensor AP clustering for high-speed road tunnel vehicle detection

Aiming at the problems of signal acquisition, feature extraction and vehicle recognition in highway tunnel vehicle detection, a new tunnel vehicle detection method is proposed by combining optical time-domain reflectometry distributed fiber sensing technology and tensor affine propagation clustering algorithm. Firstly, the distributed optical fiber system designed by optical time domain reflectometry was used to collect the running signals of tunnel vehicles and obtain the measurement data. Secondly, a high-order tensor sample set is constructed by using the spatial resolution of optical fiber as channel number and combining the feature number, time domain and frequency domain. Finally, tensor affine propagation clustering method and other clustering methods are used to test the accuracy. The test results show that the proposed method can better classify vehicles without destroying the original high-dimensional data structure. Meanwhile, the unsupervised clustering algorithm also reduces manual intervention in the identification process, increases the intelligence level of the whole vehicle detection model, and effectively improves the detection accuracy rate of tunnel vehicles.

Exploiting Data Geometry in Machine Learning

A key challenge in Machine Learning (ML) is the identification of geometric structure in high-dimensional data. Most algorithms assume that data lives in a high-dimensional vector space; however, many applications involve non-Euclidean data, such as graphs, strings and matrices, or data whose structure is determined by symmetries in the underlying system. Here, we discuss methods for identifying geometric structure in data and how leveraging data geometry can give rise to efficient ML algorithms with provable guarantees.

Covariance Matrix Estimation for High-Throughput Biomedical Data with Interconnected Communities

Estimating a covariance matrix is central to high-dimensional data analysis. Empirical analyses of high-dimensional biomedical data, including genomics, proteomics, microbiome, and neuroimaging, among others, consistently reveal strong modularity in the dependence patterns. In these analyses, intercorrelated high-dimensional biomedical features often form communities or modules that can be interconnected with others. While the interconnected community structure has been extensively studied in biomedical research (e.g., gene co-expression networks), its potential to assist in the estimation of covariance matrices remains largely unexplored. To address this gap, we propose a procedure that leverages the commonly observed interconnected community structure in high-dimensional biomedical data to estimate large covariance and precision matrices. We derive the uniformly minimum variance unbiased estimators for covariance and precision matrices in closed forms and provide theoretical results on their asymptotic properties. Our proposed method enhances the accuracy of covariance- and precision-matrix estimation and demonstrates superior performance compared to the competing methods in both simulations and real data analyses.

Fast Model Selection and Hyperparameter Tuning for Generative Models.

Generative models have gained significant attention in recent years. They are increasingly used to estimate the underlying structure of high-dimensional data and artificially generate various kinds of data similar to those from the real world. The performance of generative models depends critically on a good set of hyperparameters. Yet, finding the right hyperparameter configuration can be an extremely time-consuming task. In this paper, we focus on speeding up the hyperparameter search through adaptive resource allocation, early stopping underperforming candidates quickly and allocating more computational resources to promising ones by comparing their intermediate performance. The hyperparameter search is formulated as a non-stochastic best-arm identification problem where resources like iterations or training time constrained by some predetermined budget are allocated to different hyperparameter configurations. A procedure which uses hypothesis testing coupled with Successive Halving is proposed to make the resource allocation and early stopping decisions and compares the intermediate performance of generative models by their exponentially weighted Maximum Means Discrepancy (MMD). The experimental results show that the proposed method selects hyperparameter configurations that lead to a significant improvement in the model performance compared to Successive Halving for a wide range of budgets across several real-world applications.

Accelerating Hyperbolic t-SNE.

The need to understand the structure of hierarchical or high-dimensional data is present in a variety of fields. Hyperbolic spaces have proven to be an important tool for embedding computations and analysis tasks as their non-linear nature lends itself well to tree or graph data. Subsequently, they have also been used in the visualization of high-dimensional data, where they exhibit increased embedding performance. However, none of the existing dimensionality reduction methods for embedding into hyperbolic spaces scale well with the size of the input data. That is because the embeddings are computed via iterative optimization schemes and the computation cost of every iteration is quadratic in the size of the input. Furthermore, due to the non-linear nature of hyperbolic spaces, euclidean acceleration structures cannot directly be translated to the hyperbolic setting. This article introduces the first acceleration structure for hyperbolic embeddings, building upon a polar quadtree. We compare our approach with existing methods and demonstrate that it computes embeddings of similar quality in significantly less time. Implementation and scripts for the experiments can be found at https://graphics.tudelft.nl/accelerating-hyperbolic-tsne.

Low-Rank Tensor Learning for Incomplete Multiview Clustering

Incomplete multiview clustering (IMVC) is an effective way to identify the underlying structure of incomplete multiview data. Most existing algorithms based on matrix factorization, graph learning or subspace learning have at least one of the following limitations: (1) the global and local structures of high-dimensional data are not effectively explored simultaneously; (2) the high-order correlations among multiple views are ignored. In this paper, we propose a low-rank tensor learning (LRTL) method that learns a consensus low-dimensional embedding matrix for IMVC. We first take advantage of the self-expressiveness property of high-dimensional data to construct sparse similarity matrices for individual views under low-rank and sparsity constraints. Individual low-dimensional embedding matrices can be obtained from the sparse similarity matrices using spectral embedding techniques. This approach simultaneously explores the global and local structures of incomplete multiview data. Then, we present a multiview embedding matrix fusion model that incorporates individual low-dimensional embedding matrices into a third-norm tensor to achieve a consensus low-dimensional embedding matrix. The fusion model exploits complementary information by finding the high-order correlations among multiple views. In addition, the computational cost of an improved fusion strategy is dramatically reduced. Extensive experimental results demonstrate that the proposed LRTL method outperforms several state-of-the-art approaches.

Nonlinear Feature Selection Neural Network via Structured Sparse Regularization.

Feature selection is an important and effective data preprocessing method, which can remove the noise and redundant features while retaining the relevant and discriminative features in high-dimensional data. In real-world applications, the relationships between data samples and their labels are usually nonlinear. However, most of the existing feature selection models focus on learning a linear transformation matrix, which cannot capture such a nonlinear structure in practice and will degrade the performance of downstream tasks. To address the issue, we propose a novel nonlinear feature selection method to select those most relevant and discriminative features in high-dimensional dataset. Specifically, our method learns the nonlinear structure of high-dimensional data by a neural network with cross entropy loss function, and then using the structured sparsity norm such as l2,p -norm to regularize the weights matrix connecting the input layer and the first hidden layer of the neural network model to learn weight of each feature. Therefore, a structural sparse weights matrix is obtained by conducting nonlinear learning based on a neural network with structured sparsity regularization. Then, we use the gradient descent method to achieve the optimal solution of the proposed model. Evaluating the experimental results on several synthetic datasets and real-world datasets shows the effectiveness and superiority of the proposed nonlinear feature selection model.

Unsupervised Adaptive Bipartite Graph Embedding

In traditional graph embedding methods, graph construction is sensitive to high-dimensional data with noise and outliers, making an effective exploration of the neighborhood structure of the data difficult. Besides, with these methods, constructing graphs and reducing dimensions are disconnected and cannot be mutually optimized. To address these problems, we propose an unsupervised dimensionality reduction method based on bipartite graph, named unsupervised adaptive bipartite graph embedding (UABGE). First, the anchors are generated from the raw data by K-means or random sampling. Second, the bipartite graph, which is constructed between the samples and the anchors in the low-dimensional subspace, utilizes the adaptive allocation method to assign neighbors for each sample, so that the local structure of high-dimensional data can be captured effectively. Third, we present an objective function that combines bipartite graph construction and projection matrix learning to achieve mutual optimization between them, which can be solved with an alternating optimization algorithm. Finally, the computational complexity and the convergence of the algorithm are analyzed. Experimental results on synthetic data and publicly available datasets illustrate the effectiveness of the proposed method.