Johnson-Lindenstrauss Lemma Research Articles

Adaptive Compressed Spectrum Sensing Based on Cross Validation in WideBand Cognitive Radio System

This paper proposes an adaptive compressed spectrum sensing (CSS) algorithm to detect the spectral holes in wideband cognitive radio (CR) system without the a priori information on the sparsity of received signal. Utilizing Johnson–Lindenstrauss lemma and cross validation, it is proved that the recovery error of wideband analog signal can be estimated by the recovery error of testing measurements in random demodulated compressed sensing. Then, taking the recovery error of wideband analog signal estimated as a stopping rule, an adaptive CSS algorithm is proposed to detect the spectral holes in the wideband spectrum. Furthermore, the parameters are optimized in the algorithm to maximize system throughput. Finally, the numerical results show the effectiveness of the algorithm and the optimization of parameters. For a wideband channel with an unknown spectrum occupancy, this adaptive CSS can obtain the minimum sampling rate and detect the spectrum holes for CR users. Compared with the traditional CSS and two-step CSS, this adaptive CSS can reduce the sampling resource and improve the throughput in wideband cognitive radio system.

Robustness properties of dimensionality reduction with Gaussian random matrices

In this paper, motivated by the results in compressive phase retrieval, we study the robustness properties of dimensionality reduction with Gaussian random matrices having arbitrarily erased rows. We first study the robustness property against erasure for the almost norm preservation property of Gaussian random matrices by obtaining the optimal estimate of the erasure ratio for a small given norm distortion rate. As a consequence, we establish the robustness property of Johnson-Lindenstrauss lemma and the robustness property of restricted isometry property with corruption for Gaussian random matrices. Secondly, we obtain a sharp estimate for the optimal lower and upper bounds of norm distortion rates of Gaussian random matrices under a given erasure ratio. This allows us to establish the strong restricted isometry property with the almost optimal restricted isometry property (RIP) constants, which plays a central role in the study of phaseless compressed sensing. As a byproduct of our results, we also establish the robustness property of Gaussian random finite frames under erasure.

A data-scalable randomized misfit approach for solving large-scale PDE-constrained inverse problems

A randomized misfit approach is presented for the efficient solution of large-scale PDE-constrained inverse problems with high-dimensional data. The purpose of this paper is to offer a theory-based framework for random projections in this inverse problem setting. The stochastic approximation to the misfit is analyzed using random projection theory. By expanding beyond mean estimator convergence, a practical characterization of randomized misfit convergence can be achieved. The theoretical results developed hold with any valid random projection in the literature. The class of feasible distributions is broad yet simple to characterize compared to previous stochastic misfit methods. This class includes very sparse random projections which provide additional computational benefit. A different proof for a variant of the Johnson–Lindenstrauss lemma is also provided. This leads to a different intuition for the factor in bounds for Johnson–Lindenstrauss results. The main contribution of this paper is a theoretical result showing the method guarantees a valid solution for small reduced misfit dimensions. The interplay between Johnson–Lindenstrauss theory and Morozov’s discrepancy principle is shown to be essential to the result. The computational cost savings for large-scale PDE-constrained problems with high-dimensional data is discussed. Numerical verification of the developed theory is presented for model problems of estimating a distributed parameter in an elliptic partial differential equation. Results with different random projections are presented to demonstrate the viability and accuracy of the proposed approach.

Random Projection Estimation of Discrete-Choice Models with Large Choice Sets

We introduce random projection, an important dimension-reduction tool from machine learning, for the estimation of aggregate discrete-choice models with high-dimensional choice sets. Initially, high-dimensional data are projected into a lower-dimensional Euclidean space using random projections. Subsequently, estimation proceeds using cyclical monotonicity moment inequalities implied by the multinomial choice model; the estimation procedure is semiparametric and does not require explicit distributional assumptions to be made regarding the random utility errors. Our procedure is justified via the Johnson–Lindenstrauss lemma—the pairwise distances between data points are preserved through random projections. The estimator works well in simulations and in an application to a supermarket scanner data set. This paper was accepted by Juanjuan Zhang, marketing.

Accelerating Random Kaczmarz Algorithm Based on Clustering Information

Kaczmarz algorithm is an efficient iterative algorithm to solve overdetermined consistent system of linear equations. During each updating step, Kaczmarz chooses a hyperplane based on an individual equation and projects the current estimate for the exact solution onto that space to get a new estimate.Many vairants of Kaczmarz algorithms are proposed on how to choose better hyperplanes.Using the property of randomly sampled data in high-dimensional space,we propose an accelerated algorithm based on clustering information to improve block Kaczmarz and Kaczmarz via Johnson-Lindenstrauss lemma. Additionally, we theoretically demonstrate convergence improvement on block Kaczmarz algorithm.

A Quantized Johnson–Lindenstrauss Lemma: The Finding of Buffon’s Needle

In 1733, Georges-Louis Leclerc, Comte de Buffon in France, set the ground of geometric probability theory by defining an enlightening problem: what is the probability that a needle thrown randomly on a ground made of equispaced parallel strips lies on two of them? In this paper, we show that the solution to this problem, and its generalization to N dimensions, allows us to discover a quantized form of the Johnson-Lindenstrauss (JL) lemma, i.e., one that combines a linear dimensionality reduction procedure with a uniform quantization of precision δ > 0. In particular, given a finite set S ⊂ ℝ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</sup> of S points and a distortion level ϵ > 0, as soon as M > M <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> = O(ϵ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-2</sup> log S), we can (randomly) construct a mapping from (S, ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> ) to (δℤ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</sup> , ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> ) that approximately preserves the pairwise distances between the points of S. Interestingly, compared with the common JL lemma, the mapping is quasi-isometric and we observe both an additive and a multiplicative distortions on the embedded distances. These two distortions, however, decay as O((log S/M) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sup> ) when M increases. Moreover, for coarse quantization, i.e., for high δ compared with the set radius, the distortion is mainly additive, while for small δ we tend to a Lipschitz isometric embedding. Finally, we prove the existence of a nearly quasi-isometric embedding of (S, ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> ) into (δℤ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</sup> , ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> ). This one involves a non-linear distortion of the ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -distance in S that vanishes for distant points in this set. Noticeably, the additive distortion in this case is slower, and decays as O((log S/M) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/4</sup> ).

Toward a unified theory of sparse dimensionality reduction in Euclidean space

Let $${\Phi\in\mathbb{R}^{m\times n}}$$ be a sparse Johnson–Lindenstrauss transform (Kane and Nelson in J ACM 61(1):4, 2014) with s non-zeroes per column. For a subset T of the unit sphere, $${\varepsilon\in(0,1/2)}$$ given, we study settings for m, s required to ensure $$\mathop{\mathbb{E}}_\Phi \sup_{x\in T}\left|\|\Phi x\|_2^2 - 1\right| < \varepsilon,$$ i.e. so that $${\Phi}$$ preserves the norm of every $${x\in T}$$ simultaneously and multiplicatively up to $${1+\varepsilon}$$ . We introduce a new complexity parameter, which depends on the geometry of T, and show that it suffices to choose s and m such that this parameter is small. Our result is a sparse analog of Gordon’s theorem, which was concerned with a dense $${\Phi}$$ having i.i.d. Gaussian entries. We qualitatively unify several results related to the Johnson–Lindenstrauss lemma, subspace embeddings, and Fourier-based restricted isometries. Our work also implies new results in using the sparse Johnson–Lindenstrauss transform in numerical linear algebra, classical and model-based compressed sensing, manifold learning, and constrained least squares problems such as the Lasso.

Fast GRAPPA reconstruction with random projection.

To address the issue of computational complexity in generalized autocalibrating partially parallel acquisition (GRAPPA) when several calibration data are used. GRAPPA requires fully sampled data for accurate calibration with increasing data needed for higher reduction factors to maintain accuracy, which leads to longer computational time, especially in a three-dimensional (3D) setting and with higher channel count coils. Channel reduction methods have been developed to address this issue when massive array coils are used. In this study, the complexity problem was addressed from a different prospective. Instead of compressing to fewer channels, we propose the use of random projections to reduce the dimension of the linear equation in the calibration phase. The equivalence before and after the reduction is supported by the Johnson-Lindenstrauss lemma. The proposed random projection method can be integrated with channel reduction sequentially for even higher computational efficiency. Experimental results show that GRAPPA with random projection can achieve comparable image quality with much less computational time when compared with conventional GRAPPA without random projection. The proposed random projection method is able to reduce the computational time of GRAPPA, especially in a 3D setting, without compromising the image quality, or to improve the reconstruction quality by allowing more data for calibration when the computational time is a limiting factor. Magn Reson Med 74:71-80, 2015. © 2014 Wiley Periodicals, Inc.

Applying Supervised Learning Algorithms and a New Feature Selection Method to Predict Coronary Artery Disease

From a fresh data science perspective, this thesis discusses the prediction of coronary artery disease based on genetic variations at the DNA base pair level, called Single-Nucleotide Polymorphisms (SNPs), collected from the Ontario Heart Genomics Study (OHGS). First, the thesis explains two commonly used supervised learning algorithms, the k-Nearest Neighbour (k-NN) and Random Forest classifiers, and includes a complete proof that the k-NN classifier is universally consistent in any finite dimensional normed vector space. Second, the thesis introduces two dimensionality reduction steps, Random Projections, a known feature extraction technique based on the Johnson-Lindenstrauss lemma, and a new method termed Mass Transportation Distance (MTD) Feature Selection for discrete domains. Then, this thesis compares the performance of Random Projections with the k-NN classifier against MTD Feature Selection and Random Forest, for predicting artery disease based on accuracy, the F-Measure, and area under the Receiver Operating Characteristic (ROC) curve. The comparative results demonstrate that MTD Feature Selection with Random Forest is vastly superior to Random Projections and k-NN. The Random Forest classifier is able to obtain an accuracy of 0.6660 and an area under the ROC curve of 0.8562 on the OHGS genetic dataset, when 3335 SNPs are selected by MTD Feature Selection for classification. This area is considerably better than the previous high score of 0.608 obtained by Davies et al. in 2010 on the same dataset.

Exponential communication gap between weak and strong classical simulations of quantum communication

The most trivial way to simulate classically the communication of a quantum state is to transmit the classical description of the quantum state itself. However, this requires an infinite amount of classical communication if the simulation is exact. A more intriguing and potentially less demanding strategy would encode the full information about the quantum state into the probability distribution of the communicated variables so that this information is never sent in each single shot. This kind of simulation is called weak, as opposed to strong simulations, where the quantum state is communicated in individual shots. In this paper, we introduce a bounded-error weak protocol for simulating the communication of an arbitrary number of qubits and a subsequent two-outcome measurement consisting of an arbitrary pure state projector and its complement. This protocol requires an amount of classical communication independent of the number of qubits and proportional to ${\ensuremath{\Delta}}^{\ensuremath{-}1}$, where $\ensuremath{\Delta}$ is the error and a free parameter of the protocol. Conversely, a bounded-error strong protocol requires an amount of classical communication growing exponentially with the number of qubits for a fixed error. Our result improves a previous protocol, based on the Johnson-Lindenstrauss lemma, with communication cost scaling as ${\ensuremath{\Delta}}^{\ensuremath{-}2}\mathrm{ln}{\ensuremath{\Delta}}^{\ensuremath{-}1}$.

A variant of the Johnson–Lindenstrauss lemma for circulant matrices

We continue our study of the Johnson–Lindenstrauss lemma and its connection to circulant matrices started in Hinrichs and Vybíral (in press) [7]. We reduce the bound on k from k = Ω ( ε − 2 log 3 n ) proven there to k = Ω ( ε − 2 log 2 n ) . Our technique differs essentially from the one used in Hinrichs and Vybíral (in press) [7]. We employ the discrete Fourier transform and singular value decomposition to deal with the dependency caused by the circulant structure.

The Johnson–Lindenstrauss Lemma Almost Characterizes Hilbert Space, But Not Quite

Let X be a normed space that satisfies the Johnson–Lindenstrauss lemma (J–L lemma, in short) in the sense that for any integer n and any x 1,…,x n ∈X, there exists a linear mapping L:X→F, where F⊆X is a linear subspace of dimension O(log n), such that ‖x i −x j ‖≤‖L(x i )−L(x j )‖≤O(1)⋅‖x i −x j ‖ for all i,j∈{1,…,n}. We show that this implies that X is almost Euclidean in the following sense: Every n-dimensional subspace of X embeds into Hilbert space with distortion $2^{2^{O(\log^{*}n)}}$. On the other hand, we show that there exists a normed space Y which satisfies the J–L lemma, but for every n, there exists an n-dimensional subspace E n ⊆Y whose Euclidean distortion is at least 2Ω(α(n)), where α is the inverse Ackermann function.

A Unified Theorem on SDP Rank Reduction

We consider the problem of finding a low-rank approximate solution to a system of linear equations in symmetric, positive semidefinite matrices, where the approximation quality of a solution is measured by its maximum relative deviation, both above and below, from the prescribed quantities. We show that a simple randomized polynomial-time procedure produces a low-rank solution that has provably good approximation qualities. Our result provides a unified treatment of and generalizes several well-known results in the literature. In particular, it contains as special cases the Johnson–Lindenstrauss lemma on dimensionality reduction, results on low-distortion embeddings into low-dimensional Euclidean space, and approximation results on certain quadratic optimization problems.

Fuzzy ensemble clustering based on random projections for DNA microarray data analysis

Two major problems related the unsupervised analysis of gene expression data are represented by the accuracy and reliability of the discovered clusters, and by the biological fact that the boundaries between classes of patients or classes of functionally related genes are sometimes not clearly defined. The main goal of this work consists in the exploration of new strategies and in the development of new clustering methods to improve the accuracy and robustness of clustering results, taking into account the uncertainty underlying the assignment of examples to clusters in the context of gene expression data analysis. We propose a fuzzy ensemble clustering approach both to improve the accuracy of clustering results and to take into account the inherent fuzziness of biological and bio-medical gene expression data. We applied random projections that obey the Johnson-Lindenstrauss lemma to obtain several instances of lower dimensional gene expression data from the original high-dimensional ones, approximately preserving the information and the metric structure of the original data. Then we adopt a double fuzzy approach to obtain a consensus ensemble clustering, by first applying a fuzzy k-means algorithm to the different instances of the projected low-dimensional data and then by using a fuzzy t-norm to combine the multiple clusterings. Several variants of the fuzzy ensemble clustering algorithms are proposed, according to different techniques to combine the base clusterings and to obtain the final consensus clustering. We applied our proposed fuzzy ensemble methods to the gene expression analysis of leukemia, lymphoma, adenocarcinoma and melanoma patients, and we compared the results with other state of the art ensemble methods. Results show that in some cases, taking into account the natural fuzziness of the data, we can improve the discovery of classes of patients defined at bio-molecular level. The reduction of the dimension of the data, achieved through random projections techniques, is well-suited to the characteristics of high-dimensional gene expression data, thus resulting in improved performance with respect to single fuzzy k-means and with respect to ensemble methods based on resampling techniques. Moreover, we show that the analysis of the accuracy and diversity of the base fuzzy clusterings can be useful to explain the advantages and the limitations of the proposed fuzzy ensemble approach.

On variants of the JohnsonLindenstrauss lemma

The JohnsonLindenstrauss lemma asserts that an n-point set in any Euclidean space can be mapped to a Euclidean space of dimension k = O(e-2 log n) so that all distances are preserved up to a multip...

On variants of the Johnson–Lindenstrauss lemma

AbstractThe Johnson–Lindenstrauss lemma asserts that an n‐point set in any Euclidean space can be mapped to a Euclidean space of dimension k = O(ε‐2 log n) so that all distances are preserved up to a multiplicative factor between 1 − ε and 1 + ε. Known proofs obtain such a mapping as a linear map Rn → Rk with a suitable random matrix. We give a simple and self‐contained proof of a version of the Johnson–Lindenstrauss lemma that subsumes a basic versions by Indyk and Motwani and a version more suitable for efficient computations due to Achlioptas. (Another proof of this result, slightly different but in a similar spirit, was given independently by Indyk and Naor.) An even more general result was established by Klartag and Mendelson using considerably heavier machinery.Recently, Ailon and Chazelle showed, roughly speaking, that a good mapping can also be obtained by composing a suitable Fourier transform with a linear mapping that has a sparse random matrix M; a mapping of this form can be evaluated very fast. In their result, the nonzero entries of M are normally distributed. We show that the nonzero entries can be chosen as random ± 1, which further speeds up the computation. We also discuss the case of embeddings into Rk with the ℓ1 norm. © 2008 Wiley Periodicals, Inc. Random Struct. Alg., 2008

A Simple Proof of the Restricted Isometry Property for Random Matrices

We give a simple technique for verifying the Restricted Isometry Property (as introduced by Candes and Tao) for random matrices that underlies Compressed Sensing. Our approach has two main ingredients: (i) concentration inequalities for random inner products that have recently provided algorithmically simple proofs of the Johnson–Lindenstrauss lemma; and (ii) covering numbers for finite-dimensional balls in Euclidean space. This leads to an elementary proof of the Restricted Isometry Property and brings out connections between Compressed Sensing and the Johnson–Lindenstrauss lemma. As a result, we obtain simple and direct proofs of Kashin’s theorems on widths of finite balls in Euclidean space (and their improvements due to Gluskin) and proofs of the existence of optimal Compressed Sensing measurement matrices. In the process, we also prove that these measurements have a certain universality with respect to the sparsity-inducing basis.

Efficiently Computing the Robinson-Foulds Metric

The Robinson-Foulds (RF) metric is the measure most widely used in comparing phylogenetic trees; it can be computed in linear time using Day's algorithm. When faced with the need to compare large numbers of large trees, however, even linear time becomes prohibitive. We present a randomized approximation scheme that provides, in sublinear time and with high probability, a (1 + epsilon) approximation of the true RF metric. Our approach is to use a sublinear-space embedding of the trees, combined with an application of the Johnson-Lindenstrauss lemma to approximate vector norms very rapidly. We complement our algorithm by presenting an efficient embedding procedure, thereby resolving an open issue from the preliminary version of this paper. We have also improved the performance of Day's (exact) algorithm in practice by using techniques discovered while implementing our approximation scheme. Indeed, we give a unified framework for edge-based tree algorithms in which implementation tradeoffs are clear. Finally, we present detailed experimental results illustrating the precision and running-time tradeoffs as well as demonstrating the speed of our approach. Our new implementation, FastRF, is available as an open-source tool for phylogenetic analysis.

Kernels as features: On kernels, margins, and low-dimensional mappings

Kernel functions are typically viewed as providing an implicit mapping of points into a high-dimensional space, with the ability to gain much of the power of that space without incurring a high cost if the result is linearly-separable by a large margin γ. However, the Johnson-Lindenstrauss lemma suggests that in the presence of a large margin, a kernel function can also be viewed as a mapping to a low-dimensional space, one of dimension only $$\tilde{O}(1/\gamma^2)$$ . In this paper, we explore the question of whether one can efficiently produce such low-dimensional mappings, using only black-box access to a kernel function. That is, given just a program that computes K(x,y) on inputs x,y of our choosing, can we efficiently construct an explicit (small) set of features that effectively capture the power of the implicit high-dimensional space? We answer this question in the affirmative if our method is also allowed black-box access to the underlying data distribution (i.e., unlabeled examples). We also give a lower bound, showing that if we do not have access to the distribution, then this is not possible for an arbitrary black-box kernel function; we leave as an open problem, however, whether this can be done for standard kernel functions such as the polynomial kernel. Our positive result can be viewed as saying that designing a good kernel function is much like designing a good feature space. Given a kernel, by running it in a black-box manner on random unlabeled examples, we can efficiently generate an explicit set of $$\tilde{O}(1/\gamma^2)$$ features, such that if the data was linearly separable with margin γ under the kernel, then it is approximately separable in this new feature space.

The Johnson-Lindenstrauss lemma and the sphericity of some graphs

A simple short proof of the Johnson-Lindenstrauss lemma (concerning nearly isometric embeddings of finite point sets in lower-dimensional spaces) is given. This result is applied to show that if G is a graph on n vertices and with smallest eigenvalue λ then its sphericity sph( G) is less than cλ 2 log n. It is also proved that if G or its complement is a forest then sph( G) ≤ c log n holds.