Low Intrinsic Dimensionality Research Articles

Online Categorical Subspace Learning for Sketching Big Data with Misses

With the scale of data growing every day, reducing the dimensionality (a.k.a. sketching) of high-dimensional data has emerged as a task of paramount importance. Relevant issues to address in this context include the sheer volume of data that may consist of categorical samples, the typically streaming format of acquisition, and the possibly missing entries. To cope with these challenges, the present paper develops a novel categorical subspace learning approach to unravel the latent structure for three prominent categorical (bilinear) models, namely, Probit, Tobit, and Logit. The deterministic Probit and Tobit models treat data as quantized values of an analog-valued process lying in a low-dimensional subspace, while the probabilistic Logit model relies on low dimensionality of the data log-likelihood ratios. Leveraging the low intrinsic dimensionality of the sought models, a rank regularized maximum-likelihood estimator is devised, which is then solved recursively via alternating majorization-minimization to sketch high-dimensional categorical data `on the fly.' The resultant procedure alternates between sketching the new incomplete datum and refining the latent subspace, leading to lightweight first-order algorithms with highly parallelizable tasks per iteration. As an extra degree of freedom, the quantization thresholds are also learned jointly along with the subspace to enhance the predictive power of the sought models. Performance of the subspace iterates is analyzed for both infinite and finite data streams, where for the former asymptotic convergence to the stationary point set of the batch estimator is established, while for the latter sublinear regret bounds are derived for the empirical cost. Simulated tests with both synthetic and real-world datasets corroborate the merits of the novel schemes for real-time movie recommendation and chess-game classification.

Principal components analysis and the reported low intrinsic dimensionality of gene expression microarray data.

Principal components analysis (PCA) is a common unsupervised method for the analysis of gene expression microarray data, providing information on the overall structure of the analyzed dataset. In the recent years, it has been applied to very large datasets involving many different tissues and cell types, in order to create a low dimensional global map of human gene expression. Here, we reevaluate this approach and show that the linear intrinsic dimensionality of this global map is higher than previously reported. Furthermore, we analyze in which cases PCA fails to detect biologically relevant information and point the reader to methods that overcome these limitations. Our results refine the current understanding of the overall structure of gene expression spaces and show that PCA critically depends on the effect size of the biological signal as well as on the fraction of samples containing this signal.

Hyperspectral Computational Ghost Imaging via Temporal Multiplexing

Computational ghost imaging has made large progress in both spatial resolution and acquisition efficiency, but so far still cannot resolve the spectral reflectance well. This letter proposes a spectrum encoded acquisition scheme to achieve computational ghost imaging of hyperspectral data. Taking advantage of the speed gap between the extremely fast response of the bucket detector and magnitudes lower spatial illumination modulation, our approach temporally multiplexes a group of diverse spectra into the elapse of each 2-D illumination pattern. The number and the type of the multiplexed spectra are optimized utilizing the low intrinsic dimension of the hyperspectral data and based on the reconstruction quality on a diverse range of nature materials. After data acquisition, we infer the top principal component analysis projections of the hyperspectral image from the demultiplexed correlated measurements and the illumination patterns, and then reconstruct the final hyperspectral data. As far as we know, we are the first one to achieve computational hyperspectral ghost imaging with high accuracy in a few seconds.

Low Dimensional Embeddings of Doubling Metrics

We study several embeddings of doubling metrics into low dimensional normed spaces, in particular into ℓ 2 and ℓ ∞ . Doubling metrics are a robust class of metric spaces that have low intrinsic dimension, and often occur in applications. Understanding the dimension required for a concise representation of such metrics is a fundamental open problem in the area of metric embedding. Here we show that the n-vertex Laakso graph can be embedded into constant dimensional ℓ 2 with the best possible distortion, which has implications for possible approaches to the above problem.Since arbitrary doubling metrics require high distortion for embedding into ℓ 2 and even into ℓ 1, we turn to the ℓ ∞ space that enables us to obtain arbitrarily small distortion. We show embeddings of doubling metrics and their ”snowflakes” into low dimensional ℓ ∞ space that simplify and extend previous results.

Detecting False Data Injection Attacks on Power Grid by Sparse Optimization

State estimation in electric power grid is vulnerable to false data injection attacks, and diagnosing such kind of malicious attacks has significant impacts on ensuring reliable operations for power systems. In this paper, the false data detection problem is viewed as a matrix separation problem. By noticing the intrinsic low dimensionality of temporal measurements of power grid states as well as the sparse nature of false data injection attacks, a novel false data detection mechanism is proposed based on the separation of nominal power grid states and anomalies. Two methods, the nuclear norm minimization and low rank matrix factorization, are presented to solve this problem. It is shown that proposed methods are able to identify proper power system operation states as well as detect the malicious attacks, even under the situation that collected measurement data is incomplete. Numerical simulation results both on the synthetic and real data validate the effectiveness of the proposed mechanism.

Load Curve Data Cleansing and Imputation Via Sparsity and Low Rank

The smart grid vision is to build an intelligent power network with an unprecedented level of situational awareness and controllability over its services and infrastructure. This paper advocates statistical inference methods to robustify power monitoring tasks against the outlier effects owing to faulty readings and malicious attacks, as well as against missing data due to privacy concerns and communication errors. In this context, a novel load cleansing and imputation scheme is developed leveraging the low intrinsic-dimensionality of spatiotemporal load profiles and the sparse nature of "bad data.'' A robust estimator based on principal components pursuit (PCP) is adopted, which effects a twofold sparsity-promoting regularization through an $\ell_1$-norm of the outliers, and the nuclear norm of the nominal load profiles. Upon recasting the non-separable nuclear norm into a form amenable to decentralized optimization, a distributed (D-) PCP algorithm is developed to carry out the imputation and cleansing tasks using networked devices comprising the so-termed advanced metering infrastructure. If D-PCP converges and a qualification inequality is satisfied, the novel distributed estimator provably attains the performance of its centralized PCP counterpart, which has access to all networkwide data. Computer simulations and tests with real load curve data corroborate the convergence and effectiveness of the novel D-PCP algorithm.

Dynamic Anomalography: Tracking Network Anomalies Via Sparsity and Low Rank

In the backbone of large-scale networks, origin-to-destination (OD) traffic flows experience abrupt unusual changes known as traffic volume anomalies, which can result in congestion and limit the extent to which end-user quality of service requirements are met. As a means of maintaining seamless end-user experience in dynamic environments, as well as for ensuring network security, this paper deals with a crucial network monitoring task termed dynamic anomalography. Given link traffic measurements (noisy superpositions of unobserved OD flows) periodically acquired by backbone routers, the goal is to construct an estimated map of anomalies in real time, and thus summarize the network `health state' along both the flow and time dimensions. Leveraging the low intrinsic-dimensionality of OD flows and the sparse nature of anomalies, a novel online estimator is proposed based on an exponentially-weighted least-squares criterion regularized with the sparsity-promoting <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -norm of the anomalies, and the nuclear norm of the nominal traffic matrix. After recasting the non-separable nuclear norm into a form amenable to online optimization, a real-time algorithm for dynamic anomalography is developed and its convergence established under simplifying technical assumptions. For operational conditions where computational complexity reductions are at a premium, a lightweight stochastic gradient algorithm based on Nesterov's acceleration technique is developed as well. Comprehensive numerical tests with both synthetic and real network data corroborate the effectiveness of the proposed online algorithms and their tracking capabilities, and demonstrate that they outperform state-of-the-art approaches developed to diagnose traffic anomalies.

Theoretical and practical considerations on the convergence properties of the Fisher-EM algorithm

The Fisher-EM algorithm has been recently proposed in Bouveyron and Brunet (2012) [5] for the simultaneous visualization and clustering of high-dimensional data. It is based on a latent mixture model which fits the data into a latent discriminative subspace with a low intrinsic dimension. Although the Fisher-EM algorithm is based on the EM algorithm, it does not respect at a first glance all conditions of the EM convergence theory. Its convergence toward a maximum of the likelihood is therefore questionable. The aim of this work is twofold. First, the convergence of the Fisher-EM algorithm is studied from the theoretical point of view. In particular, it is proved that the algorithm converges under weak conditions in the general case. Second, the convergence of the Fisher-EM algorithm is considered from the practical point of view. It is shown that the Fisher criterion can be used as a stopping criterion for the algorithm to improve the clustering accuracy. It is also shown that the Fisher-EM algorithm converges faster than both the EM and CEM algorithm.

Thermal Response Variability of Random Polycrystalline Microstructures

AbstractA data-driven model reduction strategy is presented for the representation of random polycrystal microstructures. Given a set of microstructure snapshots that satisfy certain statistical constraints such as given low-order moments of the grain size distribution, using a non-linear manifold learning approach, we identify the intrinsic low-dimensionality of the microstructure manifold. In addition to grain size, a linear dimensionality reduction technique (Karhunun-Loéve Expansion) is used to reduce the texture representation. The space of viable microstructures is mapped to a low-dimensional region thus facilitating the analysis and design of polycrystal microstructures. This methodology allows us to sample microstructure features in the reduced-order space thus making it a highly efficient, low-dimensional surrogate for representing microstructures (grain size and texture). We demonstrate the model reduction approach by computing the variability of homogenized thermal properties using sparse grid collocation in the reduced-order space that describes the grain size and orientation variability.

Incremental alignment manifold learning

A new manifold learning method, called incremental alignment method (IAM), is proposed for nonlinear dimensionality reduction of high dimensional data with intrinsic low dimensionality. The main idea is to incrementally align low-dimensional coordinates of input data patch-by-patch to iteratively generate the representation of the entire dataset. The method consists of two major steps, the incremental step and the alignment step. The incremental step incrementally searches neighborhood patch to be aligned in the next step, and the alignment step iteratively aligns the low-dimensional coordinates of the neighborhood patch searched to generate the embeddings of the entire dataset. Compared with the existing manifold learning methods, the proposed method dominates in several aspects: high efficiency, easy out-of-sample extension, well metric-preserving, and averting of the local minima issue. All these properties are supported by a series of experiments performed on the synthetic and real-life datasets. In addition, the computational complexity of the proposed method is analyzed, and its efficiency is theoretically argued and experimentally demonstrated.

Exploring space–time structure of human mobility in urban space

Understanding of human mobility in urban space benefits the planning and provision of municipal facilities and services. Due to the high penetration of cell phones, mobile cellular networks provide information for urban dynamics with a large spatial extent and continuous temporal coverage in comparison with traditional approaches. The original data investigated in this paper were collected by cellular networks in a southern city of China, recording the population distribution by dividing the city into thousands of pixels. The space–time structure of urban dynamics is explored by applying Principal Component Analysis (PCA) to the original data, from temporal and spatial perspectives between which there is a dual relation. Based on the results of the analysis, we have discovered four underlying rules of urban dynamics: low intrinsic dimensionality, three categories of common patterns, dominance of periodic trends, and temporal stability. It implies that the space–time structure can be captured well by remarkably few temporal or spatial predictable periodic patterns, and the structure unearthed by PCA evolves stably over time. All these features play a critical role in the applications of forecasting and anomaly detection.

Adapting indexing trees to data distribution in feature spaces

Fast similarity retrieval is critical for content-based image retrieval systems. Tree indexing is a classical technique for fast retrieval, but the practical performance increase offered by the indexing tree depends on the intrinsic dimension of the data. Data with a low intrinsic dimension can be indexed more efficiently than data with high intrinsic dimension. This suggests that an indexing tree that is adapted to the data distribution may be more efficient. This paper proposes two adaptation procedures that are guaranteed to improve indexing efficiency. The procedures are based on a formula for average number of node tests incurred during the retrieval. The formula clearly shows how indexing performance varies with the distribution of feature points and the query. Greedy and optimal tree adaptation procedures are derived based on the formula. Both procedures explicitly enhance the retrieval performance of indexing trees. The optimally adapted tree carries the mathematical guarantee that it is the best performing tree in a set of possible trees obtained by node elimination. The adaptation procedures are applied to kdb-trees and hierarchical clustering trees for indexing synthetic as well as real data sets in medical image databases. Experimental results validate the claim that adaptation procedures increase retrieval efficiency.

Classifying functional time series

We consider the problem of classifying a high-dimensional time series into a number of disjoint classes defined by training data. Techniques of this type are an important component of a number of emerging technologies. These include the use of dense sensor arrays for condition monitoring, brain–computer interfaces for communications and control, the detection of moving pedestrians from sequences of images and the study of cognitive function using high-resolution electroencephalography (EEG). We propose a novel approach to problems of this type using the parameters of an underlying functional auto-regression model. We compare the performance of this approach using two contrasting data sets. The first is based on simulated series with different characteristics and sampling schemes and a second based on high-dimensional times series generated by multi-channel EEG. Both experiments show that our approach outperforms conventional time series methods by exploiting low-intrinsic dimensionality (smoothness). In addition, our simulation experiments show that good performance can be maintained for data generated by non-stationary sampling schemes, the latter causing large reductions in the performance of conventional procedures. These experiments suggest that meaningful information can be extracted from high-resolution EEG.

K-nearest neighbors directed noise injection in multilayer perceptron training

The relation between classifier complexity and learning set size is very important in discriminant analysis. One of the ways to overcome the complexity control problem is to add noise to the training objects, increasing in this way the size of the training set. Both the amount and the directions of noise injection are important factors which determine the effectiveness for classifier training. In this paper the effect is studied of the injection of Gaussian spherical noise and -nearest neighbors directed noise on the performance of multilayer perceptrons. As it is impossible to provide an analytical investigation for multilayer perceptrons, a theoretical analysis is made for statistical classifiers. The goal is to get a better understanding of the effect of noise injection on the accuracy of sample-based classifiers. By both empirical as well as theoretical studies, it is shown that the -nearest neighbors directed noise injection is preferable over the Gaussian spherical noise injection for data with low intrinsic dimensionality.

Infrared properties of exotic superconductors

The infrared spectra of the non-traditional superconductors share certain common features. The lack of a gap signature at 2 Δ and the residual conductivity are the consequence of a d-wave order parameter. The high T c materials, the organic conductors and the heavy Fermion materials have a strong mid-infrared absorption band which can be interpreted as strong coupling of the carriers to electronic degrees of freedom, which leads to a breakdown of the Fermi liquid picture. The cuprates and the organic charge transfer salts are unique in possessing an intrinsic low dimensionality. The charge transport normal to the highly conducting direction is incoherent down to the lowest temperatures and frequencies.