Use Of Random Projections Research Articles

Sparse multiple kernel learning: Minimax rates with random projection

In kernel-based learning, the random projection method, also called random sketching, has been successfully used in kernel ridge regression to reduce the computational burden in the big data setting, and at the same time retain the minimax convergence rate. In this work, we consider its use in sparse multiple kernel learning problems where a closed-form optimizer is not available, which poses significant technical challenges, for which the existing results do not carry over directly. Even when random projection is not used, our risk bound improves on the existing results in several aspects. We also illustrate the use of random projection via some numerical examples.

Multivariate functional outlier detection using the fast massive unsupervised outlier detection indices

We present definitions and properties of the fast massive unsupervised outlier detection (FastMUOD) indices, used for outlier detection (OD) in functional data. FastMUOD detects outliers by computing, for each curve, an amplitude, magnitude, and shape index meant to target the corresponding types of outliers. Some methods adapting FastMUOD to outlier detection in multivariate functional data are then proposed. These include applying FastMUOD on the components of the multivariate data and using random projections. Moreover, these techniques are tested on various simulated and real multivariate functional datasets. Compared with the state of the art in multivariate functional OD, the use of random projections showed the most effective results with similar, and in some cases improved, OD performance. Based on the proportion of random projections that flag each multivariate function as an outlier, we propose a new graphical tool, the magnitude‐shape‐amplitude (MSA) plot, useful for visualizing the magnitude, shape and amplitude outlyingness of multivariate functional data.

Bioinspired random projections for robust, sparse classification

Inspired by the use of random projections in biological sensing systems, we present a new algorithm for processing data in classification problems. This is based on observations of the human brain and the fruit fly's olfactory system and involves randomly projecting data into a space of greatly increased dimension before applying a cap operation to truncate the smaller entries. This leads to an algorithm that achieves a sparse representation with minimal loss in classification accuracy and is also more robust in the sense that classification accuracy is improved when noise is added to the data. This is demonstrated with numerical experiments, which supplement theoretical results demonstrating that the resulting signal transform is continuous and invertible, in an appropriate sense.

On the Non-Gaussianity of the Height of Sea Waves

The objective of this paper is to prove that the sea wave height is not a Gaussian process. This is contrary to the common belief, as the height of a sea wave is generally considered a Gaussian process. With this aim in mind, an empirical study of the buoys along the US coast at a random day is pursued. The analysis differs from those in the literature in that we study the Gaussianity of the process as a whole and not just of its one-dimensional marginal. This is done by making use of random projections and a variety of tests that are powerful against different types of alternatives. The study has resulted in a rejection of the Gaussianity in over 96% of the studied cases.

SIGEST

The SIGEST article in this issue, “Randomized Projection for Rank-Revealing Matrix Factorizations and Low-Rank Approximations,” by Jed A. Duersch and Ming Gu, concerns the use of randomization to reduce the bottleneck in a central numerical linear algebra kernel. The authors consider QR factorization, which is a key component in algorithms for constructing stable bases, for solving least-squares problems, and generally for summarizing or extracting insights from data. Column pivoting is the standard technique for dealing with cases where the target matrix is not of full rank, or where the rank is in doubt. However, for large-scale problems column pivoting requires substantial communication between different levels of computer memory, which severely limits the overall efficiency. The authors therefore propose a strategy for choosing pivots based on randomization rather than exhaustive search. This approach reduces communication complexity while increasing uncertainty in the quality of the factorization in a controlled manner. The article makes use of clear pseudocode listings in order to summarize both the original deterministic strategies and the newly developed randomized approaches; see Algorithms 1 to 6. The authors also explain how a Bayesian framework can be set up to quantify the uncertainty introduced by the use of random projections. Extensive, large-scale tests are conducted using Fortran with OpenMP, an application programming interface that supports shared-memory multiprocessing. The original article appeared in the SIAM Journal on Scientific Computing in 2017, and the main idea has subsequently proved to be effective for a number of related tasks. In creating this highlighted article for the SIGEST section, the authors have revised the introductory material, providing additional context; reorganized the document structure and added new visualizations; updated and extended the computational experiments, while making this section more compact and easier to interpret; and discussed recent developments in the field and provided references to the growing literature.

Rigorous restricted isometry property of low-dimensional subspaces

Dimensionality reduction is in demand to reduce the complexity of solving large-scale problems with data lying in latent low-dimensional structures in machine learning and computer version. Motivated by such need, in this work we study the Restricted Isometry Property (RIP) of Gaussian random projections for low-dimensional subspaces in $\mathbb{R}^N$, and rigorously prove that the projection Frobenius norm distance between any two subspaces spanned by the projected data in $\mathbb{R}^n$ ($n<N$) remain almost the same as the distance between the original subspaces with probability no less than $1 - {\rm e}^{-\mathcal{O}(n)}$. Previously the well-known Johnson-Lindenstrauss (JL) Lemma and RIP for sparse vectors have been the foundation of sparse signal processing including Compressed Sensing. As an analogy to JL Lemma and RIP for sparse vectors, this work allows the use of random projections to reduce the ambient dimension with the theoretical guarantee that the distance between subspaces after compression is well preserved.

Support vector machines resilient against training data integrity attacks

Support Vector Machines (SVMs) are vulnerable to integrity attacks, where malicious attackers distort the training data in order to compromise the decision boundary of the learned model. With increasing real-world applications of SVMs, malicious data that is classified as innocuous may have harmful consequences. This paper presents a novel framework that utilizes adversarial learning, nonlinear data projections, and game theory to improve the resilience of SVMs against such training-data-integrity attacks. The proposed approach introduces a layer of uncertainty through the use of random projections on top of the learners, making it challenging for the adversary to guess the specific configurations of the learners. To find appropriate projection directions, we introduce novel indices that ensure the contraction of the data and maximize the detection accuracy. Experiments with benchmark data sets show increases in detection rates up to 13.5% for OCSVMs and up to 14.1% for binary SVMs under different attack algorithms when compared with the respective base algorithms.

Computational efficient multidimensional singular spectrum analysis for prestack seismic data reconstruction

We have evaluated a fast and memory efficient implementation of the multidimensional singular spectrum analysis (MSSA) method for seismic data reconstruction. The new algorithm makes use of random projections and the structure of Hankel matrices to avoid the construction of large Hankel trajectory matrices. Through tests with synthetic and real data examples, we find that the presented algorithm significantly decreases the computational costs of MSSA seismic data recovery without compromising its accuracy.

Randomized Linear Algebra Approaches to Estimate the Von Neumann Entropy of Density Matrices.

The von Neumann entropy, named after John von Neumann, is an extension of the classical concept of entropy to the field of quantum mechanics. From a numerical perspective, von Neumann entropy can be computed simply by computing all eigenvalues of a density matrix, an operation that could be prohibitively expensive for large-scale density matrices. We present and analyze three randomized algorithms to approximate von Neumann entropy of real density matrices: our algorithms leverage recent developments in the Randomized Numerical Linear Algebra (RandNLA) literature, such as randomized trace estimators, provable bounds for the power method, and the use of random projections to approximate the eigenvalues of a matrix. All three algorithms come with provable accuracy guarantees and our experimental evaluations support our theoretical findings showing considerable speedup with small loss in accuracy.

Restricted Isometry Property of Gaussian Random Projection for Finite Set of Subspaces

Dimension reduction plays an essential role when decreasing the complexity of solving large-scale problems. The well-known Johnson-Lindenstrauss (JL) Lemma and Restricted Isometry Property (RIP) admit the use of random projection to reduce the dimension while keeping the Euclidean distance, which leads to the boom of Compressed Sensing and the field of sparsity related signal processing. Recently, successful applications of sparse models in computer vision and machine learning have increasingly hinted that the underlying structure of high dimensional data looks more like a union of subspaces (UoS). In this paper, motivated by JL Lemma and an emerging field of Compressed Subspace Clustering (CSC), we study for the first time the RIP of Gaussian random matrices for the compression of two subspaces based on the generalized projection $F$-norm distance. We theoretically prove that with high probability the affinity or distance between two projected subspaces are concentrated around their estimates. When the ambient dimension after projection is sufficiently large, the affinity and distance between two subspaces almost remain unchanged after random projection. Numerical experiments verify the theoretical work.

The use of random projections for the analysis of mass spectrometry imaging data.

The ‘curse of dimensionality’ imposes fundamental limits on the analysis of the large, information rich datasets that are produced by mass spectrometry imaging. Additionally, such datasets are often too large to be analyzed as a whole and so dimensionality reduction is required before further analysis can be performed. We investigate the use of simple random projections for the dimensionality reduction of mass spectrometry imaging data and examine how they enable efficient and fast segmentation using k-means clustering. The method is computationally efficient and can be implemented such that only one spectrum is needed in memory at any time. We use this technique to reveal histologically significant regions within MALDI images of diseased human liver. Segmentation results achieved following a reduction in the dimensionality of the data by more than 99% (without peak picking) showed that histologic changes due to disease can be automatically visualized from molecular images. Graphical ᅟ Electronic supplementary materialThe online version of this article (doi:10.1007/s13361-014-1024-7) contains supplementary material, which is available to authorized users.

Fast GRAPPA reconstruction with random projection.

To address the issue of computational complexity in generalized autocalibrating partially parallel acquisition (GRAPPA) when several calibration data are used. GRAPPA requires fully sampled data for accurate calibration with increasing data needed for higher reduction factors to maintain accuracy, which leads to longer computational time, especially in a three-dimensional (3D) setting and with higher channel count coils. Channel reduction methods have been developed to address this issue when massive array coils are used. In this study, the complexity problem was addressed from a different prospective. Instead of compressing to fewer channels, we propose the use of random projections to reduce the dimension of the linear equation in the calibration phase. The equivalence before and after the reduction is supported by the Johnson-Lindenstrauss lemma. The proposed random projection method can be integrated with channel reduction sequentially for even higher computational efficiency. Experimental results show that GRAPPA with random projection can achieve comparable image quality with much less computational time when compared with conventional GRAPPA without random projection. The proposed random projection method is able to reduce the computational time of GRAPPA, especially in a 3D setting, without compromising the image quality, or to improve the reconstruction quality by allowing more data for calibration when the computational time is a limiting factor. Magn Reson Med 74:71-80, 2015. © 2014 Wiley Periodicals, Inc.

Signal Processing With Direct Computations on Compressively Sensed Data

Sparsity is characteristic of a signal that potentially allows us to represent information efficiently. We present an approach that enables efficient representations based on sparsity to be utilized throughout a signal processing system, with the aim of reducing the energy and/or resources required for computation, communication, and storage. The representation we focus on is compressive sensing. Its benefit is that compression is achieved with minimal computational cost through the use of random projections; however, a key drawback is that reconstruction is expensive. We focus on inference frameworks for signal analysis. We show that reconstruction can be avoided entirely by transforming signal processing operations (e.g., wavelet transforms, finite impulse response filters, etc.) such that they can be applied directly to the compressed representations. We present a methodology and a mathematical framework that achieve this goal and also enable significant computational-energy savings through operations over fewer input samples. This enables explicit energy-versus-accuracy tradeoffs that are under the control of the designer. We demonstrate the approach through two case studies. First, we consider a system for neural prosthesis that extracts wavelet features directly from compressively sensed spikes. Through simulations, we show that spike sorting can be achieved with 54× fewer samples, providing an accuracy of 98.63% in spike count, 98.56% in firing-rate estimation, and 96.51% in determining the coefficient of variation; this compares with a baseline Nyquist-domain detector with corresponding performance of 98.97%, 99.69%, and 97.09%, respectively. Second, we consider a system for detecting epileptic seizures by extracting spectral-energy features directly from compressively sensed electroencephalogram. Through simulations of the end-to-end algorithm, we show that detection can be achieved with 21× fewer samples, providing a sensitivity of 94.43%, false alarm rate of 0.1543/h, and latency of 4.70 s; this compares with a baseline Nyquist-domain detector with corresponding performance of 96.03%, 0.1471/h, and 4.59 s, respectively.