Randomized Dimensionality Reduction Research Articles

Robust Adaptive Beamforming Based on Covariance Matrix Reconstruction with Gaussian Random Dimensionality Reduction

Robust Adaptive Beamforming Based on Covariance Matrix Reconstruction with Gaussian Random Dimensionality Reduction

Tricking the Hashing Trick: A Tight Lower Bound on the Robustness of CountSketch to Adaptive Inputs

CountSketch and Feature Hashing (the ``hashing trick'') are popular randomized dimensionality reduction methods that support recovery of l2 -heavy hitters and approximate inner products. When the inputs are not adaptive (do not depend on prior outputs), classic estimators applied to a sketch of size O(l / epsilon) are accurate for a number of queries that is exponential in l. When inputs are adaptive, however, an adversarial input can be constructed after O(l) queries with the classic estimator and the best known robust estimator only supports ~O(l^2) queries. In this work we show that this quadratic dependence is in a sense inherent: We design an attack that after O(l^2) queries produces an adversarial input vector whose sketch is highly biased. Our attack uses ``natural'' non-adaptive inputs (only the final adversarial input is chosen adaptively) and universally applies with any correct estimator, including one that is unknown to the attacker. In that, we expose inherent vulnerability of this fundamental method.

Near-neighbor preserving dimension reduction via coverings for doubling subsets of ℓ1

Randomized dimensionality reduction has been recognized as one of the cornerstones in handling high-dimensional data, originating in various foundational works such as the celebrated Johnson-Lindenstrauss Lemma. More specifically, nearest neighbor-preserving embeddings exist for ℓ2 (Euclidean) and ℓ1 (Manhattan) metrics, as well as doubling subsets of ℓ2, where doubling dimension is today the most effective way of capturing intrinsic dimensionality, as well as input structure in various applications. These randomized embeddings bound the distortion only for distances between the query point and a point set. Motivated by the foundational character of fast Approximate Nearest Neighbor search in ℓ1, this paper settles an important missing case, namely that of doubling subsets of ℓ1. In particular, we introduce a randomized dimensionality reduction by means of a near neighbor-preserving embedding, which is related to the decision-with-witness problem. The input set gets represented with a carefully chosen covering point set; in a second step, the algorithm randomly projects the latter. In order to obtain the covering point sets, we leverage either approximate r-nets or randomly shifted grids, with different tradeoffs between preprocessing time and target dimension. We exploit Cauchy random variables, and derive a concentration bound of independent interest. Our algorithms are rather simple and should therefore be useful in practice.

Estimating Leverage Scores via Rank Revealing Methods and Randomization

We study algorithms for estimating the statistical leverage scores of rectangular dense or sparse matrices of arbitrary rank. Our approach is based on combining rank revealing methods with compositions of dense and sparse randomized dimensionality reduction transforms. We first develop a set of fast novel algorithms for rank estimation, column subset selection and least squares preconditioning. We then describe the design and implementation of leverage score estimators based on these primitives. These estimators are also effective for rank deficient input, which is frequently the case in data analytics applications. We provide detailed complexity analyses for all algorithms as well as meaningful approximation bounds and comparisons with the state-of-the-art. We conduct extensive numerical experiments to evaluate our algorithms and to illustrate their properties and performance using synthetic and real world data sets.

Ensembles of classifiers based on dimensionality reduction

We present a novel approach for the construction of ensemble classifiers based on dimensionality reduction. Dimensionality reduction methods represent datasets using a small number of attributes while preserving the information conveyed by the original dataset. The ensemble members are trained based on dimension-reduced versions of the training set. These versions are obtained by applying dimensionality reduction to the original training set using different values of the input parameters. This construction meets both the diversity and accuracy criteria which are required to construct an ensemble classifier where the former criterion is obtained by the various input parameter values and the latter is achieved due to the decorrelation and noise reduction properties of dimensionality reduction. In order to classify a test sample, it is first embedded into the dimension reduced space of each individual classifier by using an out-of-sample extension algorithm. Each classifier is then applied to the embedded sample and the classification is obtained via a voting scheme. We present three variations of the proposed approach based on the Random Projections, the Diffusion Maps and the Random Subspaces dimensionality reduction algorithms. We also present a multi-strategy ensemble which combines AdaBoost and Diffusion Maps. A comparison is made with the Bagging, AdaBoost, Rotation Forest ensemble classifiers and also with the base classifier which does not incorporate dimensionality reduction. Our experiments used seventeen benchmark datasets from the UCI repository. The results obtained by the proposed algorithms were superior in many cases to other algorithms.

Sparse classification of rotating machinery faults based on compressive sensing strategy

Rotating machinery is integral to production processes and is composed of several components susceptible to malfunction. Safety and stable operation necessitate early identification of potential faults. Fault identification and classifications for rotating machinery are investigated in this study utilizing expanded monitoring data. A representation classification strategy for rotating machinery faults is developed based on a newly developed compressive sensing theory focusing on extraction and classification of fault features through sparse representation combined with random dimensionality reduction mapping. Original characteristics of a vibration signal are sampled and preserved by applying a small number of random projections and a learning redundant dictionary then constructed to sparsely represent the vibration signal. Fault signal impulse information is determined through an optimization strategy with sparsity promoting. A rolling bearing is utilized as an example in the study with simulations and experiments indicating that fault characteristics may be directly extracted from a small number of random projections without ever reconstructing the vibration signal completely and with minimal prior fault characteristic frequency knowledge required.

Efficient Dimensionality Reduction for Canonical Correlation Analysis

We present a fast algorithm for approximate canonical correlation analysis (CCA). Given a pair of tall-and-thin matrices, the proposed algorithm first employs a randomized dimensionality reduction transform to reduce the size of the input matrices, and then applies any CCA algorithm to the new pair of matrices. The algorithm computes an approximate CCA to the original pair of matrices with provable guarantees while requiring asymptotically fewer operations than the state-of-the-art exact algorithms.

Abstract PR4: Prioritizing SNPs for gene-environment interaction studies.

Abstract Background: In the post-GWAS era, investigating gene-environment interactions for disease in large-scale observational studies poses new challenges. Existing single studies are not originally powered to investigate gene-environment interactions when genome-wide variation is taken into account. Pooling studies still may not offer enough power, and introduces the problem of heterogeneity in study designs and specifically in the measurement of environmental characteristics. Therefore, current approaches are multistage, with the first stage aimed at prioritizing SNPs, thereby reducing the number of SNPs before testing their interactions with environmental factors. Methods: Current strategies for prioritizing SNPs in gene-environment interaction studies 1) with available genome-wide data for genetic variation, or 2) without readily available genome-wide data for genetic variation as in The Netherlands Cohort Study on diet and cancer are described and discussed. Results: In current gene-environment interaction studies with genome-wide data for genetic variation, several approaches are used to prioritize SNPs to be tested in further gene-environment interactions. Two screening test statistics proposed are either marginal genetic association or correlation based. The underlying idea is that SNPs likely to show interactions with environment will at least show a slight association with outcome or correlation with the environment. Gene-environment interactions are then usually first tested on a SNP by SNP basis. In the Netherlands Cohort Study on diet and cancer (NLCS), genome-wide data on genetic variants are not readily available. For ongoing projects within the NLCS, the strategies for SNP reduction to a priority list of up to 100 SNPs, depend on the hypothesis to be tested. First, genes are selected from pathways thought to be involved in the association between the environmental factor and the outcome. After gene selection using the KEGG database and literature, further prioritizing of SNPs is conducted using literature through MEDLINE and the HuGE Navigator using a pre-defined search strategy focusing on a particular pathway. The search strategy includes the SNP and either the outcome, related outcomes (that share similar etiology), or environmental factors. Inclusion criteria for SNPs further include a minor allele frequency over 10% in North Europeans or Caucasians, a positive SNP validation status according to dbSNP, and a position within 10 kb upstream of the 5'untranslated region and 10 kb downstream of the 3'untranslated region of the gene. SNPs are then ranked according to a pre-defined scoring strategy based on the number of significant associations observed in the literature with outcome and exposure. Gene-environment interactions can be explored using several complementary approaches, both exploratory (random forests and multifactor dimensionality reduction techniques) and parametric methods. Conclusions: Different strategies for prioritizing SNPs for gene-environment interaction studies exist. They show some similarity in that prioritizing is based on the association between the SNP and the outcome and/or the SNP and the environment, either in own data (availability of tag SNPs) or in the literature (e.g. NLCS pathway based approach). There is a need for a formal discussion and a more uniform strategy for prioritizing SNPs in gene-environment interaction studies especially when focusing on specific pathways. This proffered talk is also presented as Poster 53. Citation Format: Matty P. Weijenberg, Colinda C.J.M. Simons, Leo J. Schouten, Milan S. Geybels, Bas A. Verhage, Janneke G. Hogervorst, Andras P. Keszei, Roger W. Godschalk, Frederik-Jan van Schooten, Piet A. van den Brandt. Prioritizing SNPs for gene-environment interaction studies. [abstract]. In: Proceedings of the AACR Special Conference on Post-GWAS Horizons in Molecular Epidemiology: Digging Deeper into the Environment; 2012 Nov 11-14; Hollywood, FL. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2012;21(11 Suppl):Abstract nr PR4.

Spectral compressive sensing

Compressive sensing (CS) is a new approach to simultaneous sensing and compression of sparse and compressible signals based on randomized dimensionality reduction. To recover a signal from its compressive measurements, standard CS algorithms seek the sparsest signal in some discrete basis or frame that agrees with the measurements. A great many applications feature smooth or modulated signals that are frequency-sparse and can be modeled as a superposition of a small number of sinusoids; for such signals, the discrete Fourier transform (DFT) basis is a natural choice for CS recovery. Unfortunately, such signals are only sparse in the DFT domain when the sinusoid frequencies live precisely at the centers of the DFT bins; when this is not the case, CS recovery performance degrades significantly. In this paper, we introduce the spectral CS (SCS) recovery framework for arbitrary frequency-sparse signals. The key ingredients are an over-sampled DFT frame and a restricted union-of-subspaces signal model that inhibits closely spaced sinusoids. We demonstrate that SCS significantly outperforms current state-of-the-art CS algorithms based on the DFT while providing provable bounds on the number of measurements required for stable recovery. We also leverage line spectral estimation methods (specifically Thomsonʼs multitaper method and MUSIC) to further improve the performance of SCS recovery.

Randomized SVD Methods in Hyperspectral Imaging

We present a randomized singular value decomposition (rSVD) method for the purposes of lossless compression, reconstruction, classification, and target detection with hyperspectral (HSI) data. Recent work in low‐rank matrix approximations obtained from random projections suggests that these approximations are well suited for randomized dimensionality reduction. Approximation errors for the rSVD are evaluated on HSI, and comparisons are made to deterministic techniques and as well as to other randomized low‐rank matrix approximation methods involving compressive principal component analysis. Numerical tests on real HSI data suggest that the method is promising and is particularly effective for HSI data interrogation.

Robust and Secure Image Hashing via Non-Negative Matrix Factorizations

In this paper, we propose the use of non-negative matrix factorization (NMF) for image hashing. In particular, we view images as matrices and the goal of hashing as a randomized dimensionality reduction that retains the essence of the original image matrix while preventing intentional attacks of guessing and forgery. Our work is motivated by the fact that standard-rank reduction techniques, such as QR and singular value decomposition, produce low-rank bases which do not respect the structure (i.e., non-negativity for images) of the original data. We observe that NMFs have two very desirable properties for secure image hashing applications: 1) The additivity property resulting from the non-negativity constraints results in bases that capture local components of the image, thereby significantly reducing misclassification and 2) the effect of geometric attacks on images in the spatial domain manifests (approximately) as independent identically distributed noise on NMF vectors, allowing the design of detectors that are both computationally simple and, at the same time, optimal in the sense of minimizing error probabilities. Receiver operating characteristics analysis over a large image database reveals that the proposed algorithms significantly outperform existing approaches for image hashing.