Weak Sparsity Research Articles

High dimensional discriminant analysis under weak sparsity

. In this work, we consider the discriminant analysis under the weak sparsity where many entries of the parameters are nearly zero. We develop a unified LASSO-typed framework to estimate the parameters for sparse discriminant analysis and derive the general non asymptotic error bound under the weak sparsity condition. As applications, we revisit the sparse linear discriminant analysis and sparse quadratic discriminant analysis. We establish the consistency of the estimators and also the misclassification error rate. These results extend the sparse discriminant analysis methods to weak sparsity setting and refine the existing theoretical results.

Linear and nonlinear signal detection and estimation in high-dimensional nonparametric regression under weak sparsity

Linear and nonlinear signal detection and estimation in high-dimensional nonparametric regression under weak sparsity

A unified precision matrix estimation framework via sparse column-wise inverse operator under weak sparsity

In this paper, we estimate the high-dimensional precision matrix under the weak sparsity condition where many entries are nearly zero. We revisit the sparse column-wise inverse operator estimator and derive its general error bounds under the weak sparsity condition. A unified framework is established to deal with various cases including the heavy-tailed data, the non-paranormal data, and the matrix variate data. These new methods can achieve the same convergence rates as the existing methods and can be implemented efficiently.

Lasso inference for high-dimensional time series

In this paper we develop valid inference for high-dimensional time series. We extend the desparsified lasso to a time series setting under Near-Epoch Dependence (NED) assumptions allowing for non-Gaussian, serially correlated and heteroskedastic processes, where the number of regressors can possibly grow faster than the time dimension. We first derive an error bound under weak sparsity, which, coupled with the NED assumption, means this inequality can also be applied to the (inherently misspecified) nodewise regressions performed in the desparsified lasso. This allows us to establish the uniform asymptotic normality of the desparsified lasso under general conditions, including for inference on parameters of increasing dimensions. Additionally, we show consistency of a long-run variance estimator, thus providing a complete set of tools for performing inference in high-dimensional linear time series models. Finally, we perform a simulation exercise to demonstrate the small sample properties of the desparsified lasso in common time series settings.

On estimation error bounds of the Elastic Net when p ≫ n

We study the estimation property of the Elastic Net estimator in high-dimensional linear regression models where the number of parameters p is comparable to or larger than the sample size n. In such a situation, one often assumes sparsity of the true regression coefficient vector , i.e., assuming that belongs to an -ball with radius , , for some . In this paper, we provide -estimation error bounds for the Elastic Net and naive Elastic Net estimators under a unified framework for high-dimensional analysis of M-estimators proposed by Negahban et al. [A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. Adv Neural Inf Process Syst. 2009;22:1348–1356]. We show that for both cases of exact sparsity and weak sparsity, under the same conditions on the design matrix, the Elastic Net estimator achieves a slightly better error bound than the Lasso estimator by suitably choosing the tuning parameters.

The ML–EM algorithm in continuum: sparse measure solutions

Linear inverse problems Aμ = y with Poisson noise and non-negative unknown μ ⩾ 0 are ubiquitous in applications, for instance in positron emission tomography (PET) in medical imaging. The associated maximum likelihood problem is routinely solved using an expectation–maximisation algorithm (ML–EM). This typically results in images which look spiky, even with early stopping. We give an explanation for this phenomenon. We first regard the image μ as a measure. We prove that if the measurements y are not in the cone {Aμ, μ ⩾ 0}, which is typical of low injected dose, likelihood maximisers must be sparse, i.e., typically a sum of point masses. We also show a weak sparsity result for cluster points of ML–EM. On the other hand, in the low noise regime, we prove that cluster points of ML–EM are optimal measures with full support. Finally, we provide concentration bounds for the probability to be in the sparse case, and a set of numerical experiments supporting our claims.

Random walks in a strongly sparse random environment

The integer points (sites) of the real line are marked by the positions of a standard random walk with positive integer jumps. We say that the set of marked sites is weakly, moderately or strongly sparse depending on whether the jumps of the random walk are supported by a bounded set, have finite or infinite mean, respectively. Focussing on the case of strong sparsity and assuming additionally that the distribution tail of the jumps is regularly varying at infinity we consider a nearest neighbor random walk on the set of integers having jumps ±1 with probability 1∕2 at every nonmarked site, whereas a random drift is imposed at every marked site. We prove new distributional limit theorems for the so defined random walk in a strongly sparse random environment, thereby complementing results obtained recently in Buraczewski et al. (2019) for the case of moderate sparsity and in Matzavinos et al. (2016) for the case of weak sparsity. While the random walk in a strongly sparse random environment exhibits either the diffusive scaling inherent to a simple symmetric random walk or a wide range of subdiffusive scalings, the corresponding limit distributions are non-stable.

High-dimensional Linear Regression for Dependent Data with Applications to Nowcasting

Recent research has focused on $\ell_1$ penalized least squares (Lasso) estimators for high-dimensional linear regressions in which the number of covariates $p$ is considerably larger than the sample size $n$. However, few studies have examined the properties of the estimators when the errors and/or the covariates are serially dependent. In this study, we investigate the theoretical properties of the Lasso estimator for a linear regression with a random design and weak sparsity under serially dependent and/or nonsubGaussian errors and covariates. In contrast to the traditional case, in which the errors are independent and identically distributed and have finite exponential moments, we show that $p$ can be at most a power of $n$ if the errors have only finite polynomial moments. In addition, the rate of convergence becomes slower owing to the serial dependence in the errors and the covariates. We also consider the sign consistency of the model selection using the Lasso estimator when there are serial correlations in the errors or the covariates, or both. Adopting the framework of a functional dependence measure, we describe how the rates of convergence and the selection consistency of the estimators depend on the dependence measures and moment conditions of the errors and the covariates. Simulation results show that a Lasso regression can be significantly more powerful than a mixed-frequency data sampling regression (MIDAS) and a Dantzig selector in the presence of irrelevant variables. We apply the results obtained for the Lasso method to nowcasting with mixed-frequency data, in which serially correlated errors and a large number of covariates are common. The empirical results show that the Lasso procedure outperforms the MIDAS regression and the autoregressive model with exogenous variables in terms of both forecasting and nowcasting.

Random walks in a moderately sparse random environment.

A random walk in a sparse random environment is a model introduced by Matzavinos et al. [Electron. J. Probab. 21, paper no. 72: 2016] as a generalization of both a simple symmetric random walk and a classical random walk in a random environment. A random walk in a sparse random environment is a nearest neighbor random walk on that jumps to the left or to the right with probability 1/2 from every point of and jumps to the right (left) with the random probability λ k+1 (1 - λ k+1) from the point S k , . Assuming that are independent copies of a random vector and the mean is finite (moderate sparsity) we obtain stable limit laws for X n , properly normalized and centered, as n → ∞. While the case ξ ≤ M a.s. for some deterministic M > 0 (weak sparsity) was analyzed by Matzavinos et al., the case (strong sparsity) will be analyzed in a forthcoming paper.

Robust low-rank matrix estimation

Many results have been proved for various nuclear norm penalized estimators of the uniform sampling matrix completion problem. However, most of these estimators are not robust: in most of the cases the quadratic loss function and its modifications are used. We consider robust nuclear norm penalized estimators using two well-known robust loss functions: the absolute value loss and the Huber loss. Under several conditions on the sparsity of the problem (i.e., the rank of the parameter matrix) and on the regularity of the risk function sharp and nonsharp oracle inequalities for these estimators are shown to hold with high probability. As a consequence, the asymptotic behavior of the estimators is derived. Similar error bounds are obtained under the assumption of weak sparsity, that is, the case where the matrix is assumed to be only approximately low-rank. In all of our results, we consider a high-dimensional setting. In this case, this means that we assume $n\leq pq$. Finally, various simulations confirm our theoretical results.

Minimax Optimal Convex Methods for Poisson Inverse Problems Under <inline-formula> <tex-math notation="LaTeX">$\ell_{q}$ </tex-math> </inline-formula>-Ball Sparsity

In this paper, we study the minimax rates and provide an implementable convex algorithm for Poisson inverse problems under weak sparsity and physical constraints. In particular, we assume the model $y_{i} \sim \mathrm {Poisson}(Ta_{i}^{\top }f^{*})$ for $1 \leq i \leq n$ , where $T \in \mathbb {R}_{+}$ is the intensity, and we impose weak sparsity on $f^{*} \in \mathbb {R}^{p}$ by assuming $f^{*}$ lies in an $\ell _{q}$ -ball when rotated according to an orthonormal basis $D \in \mathbb {R}^{p \times p}$ . In addition, since we are modeling real-physical systems, we also impose positivity and flux-preserving constraints on the matrix $A = [a_{1}, a_{2},\ldots, a_{n}]^{\top }$ and the function $f^{*}$ . We prove minimax lower bounds for this model, which scale as $R_{q} ({\log p}/{T})^{1 - ({q}/{2})}$ where it is noticeable that the rate depends on the intensity $T$ and not the sample size $n$ . We also show that an $\ell _{1}$ -based regularized least-squares estimator achieves this minimax lower bound, provided a suitable restricted eigenvalue condition is satisfied. Finally, we prove that provided $n \geq \tilde {K} \log p$ where $\tilde {K} = \Theta (R_{q} ({\log p}/{T})^{- ({q}/{2})})$ represents an approximate sparsity level, and our restricted eigenvalue condition and physical constraints are satisfied for random bounded ensembles. We also provide numerical experiments that validate our mean-squared error bounds. Our results address a number of open issues from prior work on Poisson inverse problems that focuses on strictly sparse models and does not provide guarantees for convex implementable algorithms.

Compressive sampling for spectrally sparse signal recovery via one-bit random demodulator

The one-bit compressive sampling (CS) framework aims at alleviating the quantization burden on analog-to-digital converters by quantizing each sample to one bit, i.e., capturing just the signs of samples. Motivated by one-bit CS theory, this paper addresses a new type of data acquisition system to recover spectrally sparse signals. This system is composed of a random demodulator and a one-bit quantizer. The former yields the signal compressed samples while the latter records the sign of each sample. With the observation sign data, the signal is eventually recovered by using the binary iterative hard thresholding algorithm. Through numerical experiments, we demonstrate that our scheme is high-efficient for spectrally sparse signal recovery in the situations of low signal-to-noise ratio, stringent bit budget and weak sparsity.

Oracle inequalities, variable selection and uniform inference in high-dimensional correlated random effects panel data models

In this paper we study high-dimensional correlated random effects panel data models. Our setting is useful as it allows including time invariant covariates as under random effects yet allows for correlation between covariates and unobserved heterogeneity as under fixed effects. We use the Mundlak–Chamberlain device to model this correlation. Allowing for a flexible correlation structure naturally leads to a high-dimensional model in which least squares estimation easily becomes infeasible with even a moderate number of explanatory variables.Imposing a combination of sparsity and weak sparsity on the parameters of the model we first establish an oracle inequality for the Lasso. This is valid even when the error terms are heteroskedastic and no structure is imposed on the time series dependence of the error terms.Next, we provide upper bounds on the sup-norm estimation error of the Lasso. As opposed to the classical ℓ1- and ℓ2-bounds the sup-norm bounds do not directly depend on the unknown degree of sparsity and are thus well suited for thresholding the Lasso for variable selection. We provide sufficient conditions under which thresholding results in consistent model selection. Pointwise valid asymptotic inference is established for a post-thresholding estimator. Finally, we show how the Lasso can be desparsified in the correlated random effects setting and how this leads to uniformly valid inference even in the presence of heteroskedasticity and autocorrelated error terms.

A sparse Bayesian learning based scheme for multi-movement recognition using sEMG.

This paper proposed a feature extraction scheme based on sparse representation considering the non-stationary property of surface electromyography (sEMG). Sparse Bayesian learning was introduced to extract the feature with optimal class separability to improve recognition accuracy of multi-movement patterns. The extracted feature, sparse representation coefficients (SRC), represented time-varying characteristics of sEMG effectively because of the compressibility (or weak sparsity) of the signal in some transformed domains. We investigated the effect of the proposed feature by comparing with other fourteen individual features in offline recognition. The results demonstrated the proposed feature revealed important dynamic information in the sEMG signals. The multi-feature sets formed by the SRC and other single feature yielded more superior performance on recognition accuracy, compared with the single features. The best average recognition accuracy of 94.33% was gained by using SVM classifier with the multi-feature set combining the feature SRC, Williston amplitude (WAMP), wavelength (WL) and the coefficients of the fourth order autoregressive model (ARC4) via multiple kernel learning framework. The proposed feature extraction scheme (known as SRC + WAMP + WL + ARC4) is a promising method for multi-movement recognition with high accuracy.

Compressive Sensing SAR Image Reconstruction Based on Bayesian Framework and Evolutionary Computation

Compressive sensing (CS) is a theory that one may achieve an exact signal reconstruction from sufficient CS measurements taken from a sparse signal. However, in practical applications, the transform coefficients of SAR images usually have weak sparsity. Exactly reconstructing these images is very challenging. A new Bayesian evolutionary pursuit algorithm (BEPA) is proposed in this paper. A signal is represented as the sum of a main signal and some residual signals, and the generalized Gaussian distribution (GGD) is employed as the prior of the main signal and the residual signals. BEPA decomposes the residual iteratively and estimates the maximum a posteriori of the main signal and the residual signals by solving a sequence of subproblems to achieve the approximate CS reconstruction of the signal. Under the assumption of GGD with the parameter 0 < p < 1, the evolutionary algorithm (EA) is introduced to CS reconstruction for the first time. The better reconstruction performance can be achieved by searching the global optimal solutions of subproblems with EA. Numerical experiments demonstrate that the important features of SAR images (e.g., the point and line targets) can be well preserved by our algorithm, and the superior reconstruction performance can be obtained at the same time.