Sparsity Assumption Research Articles

L2-Boosting for sensitivity analysis with dependent inputs

This paper is dedicated to the study of an estimator of the generalized Hoeffding decomposition. We build such an estimator using an empirical Gram-Schmidt approach and derive a consistency rate in a large dimensional setting. We then apply a greedy algorithm with these previous estimators to a sensitivity analysis. We also establish the consistency of this L2-boosting under sparsity assumptions of the signal to be analyzed. The paper concludes with numerical experiments, that demonstrate the low computational cost of our method, as well as its efficiency on the standard benchmark of sensitivity analysis.

Low noise regimes for ℓ regularization : continuous and discrete settings

AbstractWe focus on support recovery for signal deconvolution with sparsity assumption. We adopt the continuous setting defined by several recent works and we try to reconstruct a sum of Dirac masses from its low frequencies (possibly perturbed by some noise), by using a total variation prior for Radon measures (i.e. the generalization to measures of the ℓ1 norm). We show that, under a non degenerate source condition, there exists a small noise regime in which the model recovers exactly the same number of spikes as the original signal, and the spikes converge to those of the original signal as the noise vanishes. This continuous setting, by allowing the spikes to “move”, provides robust support recovery for signals composed of well separated spikes.In a discrete setting, where the spikes are reconstructed on a grid, similar low noise regimes which guarantee the exact recovery of the support also exist (see [3]). Yet, this property only concerns a small class of signals. Considering the asymptotics of the discrete problems as the size of the grid tends to zero, we show that the support of the original signal cannot be stable on thin grids, and that the discrete models actually reconstruct pairs of spikes near each original spike. (© 2014 Wiley‐VCH Verlag GmbH & Co. KGaA, Weinheim)

The Dantzig Discriminant Analysis with High Dimensional Data

It is well known that linear discriminant analysis (LDA) works well and is asymptotically optimal under fixed-p-large-n situations. But Bickel and Levina (2004) showed that the LDA is as bad as random guessing when p > n. This article studies the sparse discriminant analysis via Dantzig penalized least squares. Our method avoids estimating the high-dimensional covariance matrix and does not need the sparsity assumption on the inverse of the covariance matrix. We show that the new discriminant analysis is asymptotically optimal theoretically. Simulation and real data studies show that the classifier performs better than the existing sparse methods.

Generalized Cross Validation in variable selection with and without shrinkage

This paper investigates two types of results that support the use of Generalized Cross Validation (GCV) for variable selection under the assumption of sparsity. The first type of result is based on the well established links between GCV on one hand and Mallows’s Cp and Stein Unbiased Risk Estimator (SURE) on the other hand. The result states that GCV performs as well as Cp or SURE in a regularized or penalized least squares problem as an estimator of the prediction error for the penalty in the neighborhood of its optimal value. This result can be seen as a refinement of an earlier result in GCV for soft thresholding of wavelet coefficients. The second novel result concentrates on the behavior of GCV for penalties near zero. Good behavior near zero is of crucial importance to ensure successful minimization of GCV as a function of the regularization parameter. Understanding the behavior near zero is important in the extension of GCV from ℓ1 towards ℓ0 regularized least squares, i.e., for variable selection without shrinkage, or hard thresholding. Several possible implementations of GCV are compared with each other and with SURE and Cp. These simulations illustrate the importance of the fact that GCV has an implicit and robust estimator of the observational variance.

Robust Multi-image Processing with Optimal Sparse Regularization

Sparse modeling can be used to characterize outlier type noise. Thanks to sparse recovery theory, it was shown that 1-norm super-resolution is robust to outliers if enough images are captured. Moreover, sparse modeling of signals is a way to overcome ill-posedness of under-determined problems. This naturally leads to this question: does an added sparsity assumption on the signal improve the robustness to outliers of the 1-norm super-resolution, and if yes, how strong should this assumption be? In this article, we review and extend results of the literature to the robustness to outliers of overdetermined signal recovery problems under sparse regularization, with a convex variational formulation. We then apply them to general random matrices, and show how the regularization parameter acts on the robustness to outliers. Finally, we show that in the case of multi-image processing, the structure of the support of signal and noise must be studied precisely. We show that the sparsity assumption improves robustness if outliers do not overlap with signal jumps, and determine how the regularization parameter can be chosen.

Blind identification of the underdetermined mixing matrix based on K-weighted hyperline clustering

Blind identification of the underdetermined mixing matrix is an emerging problem in the area of sparse component analysis (SCA). Traditionally, the K-hyperLine clustering (K-HLC) learning algorithm is employed to solve it, but this method is designed under the strict sparse assumption on the source signals. In order to deal with the blind identification problem of multiple dominant, a discriminatory clustering algorithm, K-weighted hyperline clustering (K-WHLC), is developed via weighting scheme. The Gaussian membership function is offered as the weight factor for the hyperline clustering, together with an optimal selection in relation to the involved threshold. As shown in the paper, this discriminatory clustering scheme is efficient and especially suitable for the hyperline identification against the multiple dominant SCA problem. Also, the developed algorithm has higher accuracy than the traditional K-HLC, and it is with lower computational cost in the medium or large-scale problem. Numerical simulations are provided finally to verify the advantages of our clustering scheme.

Concurrent multiresponse non-linear screening: Robust profiling of webpage performance

Profiling engineered data with robust mining methods continues attracting attention in knowledge engineering systems. The purpose of this article is to propose a simple technique that deals with non-linear multi-factorial multi-characteristic screening suitable for knowledge discovery studies. The method is designed to proactively seek and quantify significant information content in engineered mini-datasets. This is achieved by deploying replicated fractional-factorial sampling schemes. Compiled multi-response data are converted to a single master-response effectuated by a series of distribution-free transformations and multi-compressed data fusions. The resulting amalgamated master response is deciphered by non-linear multi-factorial stealth stochastics intended for saturated schemes. The stealth properties of our method target processing datasets which might be overwhelmed by a lack of knowledge about the nature of reference distributions at play. Stealth features are triggered to overcome restrictions regarding the data normality conformance, the effect sparsity assumption and the inherent collapse of the ‘unexplainable error’ connotation in saturated arrays. The technique is showcased by profiling four ordinary controlling factors that influence webpage content performance by collecting data from a commercial browser monitoring service on a large scale web host. The examined effects are: (1) the number of Cascading Style Sheets files, (2) the number of JavaScript files, (3) the number of Image files, and (4) the Domain Name System Aliasing. The webpage performance level was screened against three popular characteristics: (1) the time to first visual, (2) the total loading time, and (3) the customer satisfaction. Our robust multi-response data mining technique is elucidated for a ten-replicate run study dictated by an L9(34) orthogonal array scheme where any uncontrolled noise embedded contribution has not been necessarily excluded.

Audio-visual underdetermined blind source separation algorithm based on Gaussian potential function

Most existing algorithms for the underdetermined blind source separation (UBSS) problem are two-stage algorithm, i.e., mixing parameters estimation and sources estimation. In the mixing parameters estimation, the previously proposed traditional clustering algorithms are sensitive to the initializations of the mixing parameters. To reduce the sensitiveness to the initialization, we propose a new algorithm for the UBSS problem based on anechoic speech mixtures by employing the visual information, i.e., the interaural time difference (ITD) and the interaural level difference (ILD), as the initializations of the mixing parameters. In our algorithm, the video signals are utilized to estimate the distances between microphones and sources, and then the estimations of the ITD and ILD can be obtained. With the sparsity assumption in the time-frequency domain, the Gaussian potential function algorithm is utilized to estimate the mixing parameters by using the ITDs and ILDs as the initializations of the mixing parameters. And the time-frequency masking is used to recover the sources by evaluating the various ITDs and ILDs. Experimental results demonstrate the competitive performance of the proposed algorithm compared with the baseline algorithms.

Posterior contraction in sparse Bayesian factor models for massive covariance matrices

Sparse Bayesian factor models are routinely implemented for parsimonious dependence modeling and dimensionality reduction in high-dimensional applications. We provide theoretical understanding of such Bayesian procedures in terms of posterior convergence rates in inferring high-dimensional covariance matrices where the dimension can be larger than the sample size. Under relevant sparsity assumptions on the true covariance matrix, we show that commonly-used point mass mixture priors on the factor loadings lead to consistent estimation in the operator norm even when $p\gg n$. One of our major contributions is to develop a new class of continuous shrinkage priors and provide insights into their concentration around sparse vectors. Using such priors for the factor loadings, we obtain similar rate of convergence as obtained with point mass mixture priors. To obtain the convergence rates, we construct test functions to separate points in the space of high-dimensional covariance matrices using insights from random matrix theory; the tools developed may be of independent interest. We also derive minimax rates and show that the Bayesian posterior rates of convergence coincide with the minimax rates upto a $\sqrt{\log n}$ term.

Adaptive Variable Selection in Nonparametric Sparse Regression

We study the problem of exact recovery of an unknown multivariate function f observed in the continuous regression model. It is assumed that, in addition to some smoothness constraints, f possesses an additive sparse structure determined by sparsity index β ∈ (0, 1). As a consequence of the additive sparsity assumption, the recovery problem transforms to a variable selection problem. Conditions for exact variable selection are provided and a family of asymptotically minimax variable selection procedures is constructed. The procedures are adaptive with respect to the sparsity index β. Bibliography: 18 titles.

Variance estimation in high-dimensional linear models

The residual variance and the proportion of explained variation are important quantities in many statistical models and model fitting procedures. They play an important role in regression diagnostics and model selection procedures, as well as in determining the performance limits in many problems. In this paper we propose new method-of-moments-based estimators for the residual variance, the proportion of explained variation and other related quantities, such as the ℓ2 signal strength. The proposed estimators are consistent and asymptotically normal in high-dimensional linear models with Gaussian predictors and errors, where the number of predictors d is proportional to the number of observations n; in fact, consistency holds even in settings where d/n → ∞. Existing results on residual variance estimation in high-dimensional linear models depend on sparsity in the underlying signal. Our results require no sparsity assumptions and imply that the residual variance and the proportion of explained variation can be consistently estimated even when d>n and the underlying signal itself is nonestimable. Numerical work suggests that some of our distributional assumptions may be relaxed. A real-data analysis involving gene expression data and single nucleotide polymorphism data illustrates the performance of the proposed methods.

Convergence rates in ℓ1-regularization when the basis is not smooth enough

Sparsity promoting regularization is an important technique for signal reconstruction and several other ill-posed problems. Theoretical investigation typically bases on the assumption that the unknown solution has a sparse representation with respect to a fixed basis. We drop this sparsity assumption and provide error estimates for nonsparse solutions. After discussing a result in this direction published earlier by one of the authors and co-authors, we prove a similar error estimate under weaker assumptions. Two examples illustrate that this set of weaker assumptions indeed covers additional situations which appear in applications.

Underdetermined DOA estimation and blind separation of non-disjoint sources in time-frequency domain based on sparse representation method

This paper deals with the blind separation of nonstation-ary sources and direction-of-arrival (DOA) estimation in the under-determined case, when there are more sources than sensors. We assume the sources to be time-frequency (TF) disjoint to a certain extent. In particular, the number of sources presented at any TF neighborhood is strictly less than that of sensors. We can identify the real number of active sources and achieve separation in any TF neighborhood by the sparse representation method. Compared with the subspace-based algorithm under the same sparseness assumption, which suffers from the extra noise effect since it cannot estimate the true number of active sources, the proposed algorithm can estimate the number of active sources and their corresponding TF values in any TF neighborhood simultaneously. Another contribution of this paper is a new estimation procedure for the DOA of sources in the underdetermined case, which combines the TF sparseness of sources and the clustering technique. Simulation results demonstrate the validity and high performance of the proposed algorithm in both blind source separation (BSS) and DOA estimation.

Bayes Variable Selection in Semiparametric Linear Models

There is a rich literature on Bayesian variable selection for parametric models. Our focus is on generalizing methods and asymptotic theory established for mixtures of g-priors to semiparametric linear regression models having unknown residual densities. Using a Dirichlet process location mixture for the residual density, we propose a semiparametric g-prior which incorporates an unknown matrix of cluster allocation indicators. For this class of priors, posterior computation can proceed via a straightforward stochastic search variable selection algorithm. In addition, Bayes’ factor and variable selection consistency is shown to result under a class of proper priors on g even when the number of candidate predictors p is allowed to increase much faster than sample size n, while making sparsity assumptions on the true model size.

Stochastic change-point ARX-GARCH models and their applications to econometric time series

This paper shows that the commonly encountered volatility persistence in fitting GARCH models to financial time series can arise if the possibility of structural changes is not incorporated in the time series model. To avoid spurious memory in modeling volatilities of econometric time series, we consider two time-scales and use the short time-scale to define GARCH dynamics and the long time-scale to incorporate parameter jumps. This leads to a Bayesian change- point ARX-GARCH model, whose unknown parameters can undergo occasional changes at unspecified times and can be estimated by explicit recursive formulas when the hyperparameters of the Bayesian model are specified. Efficient estimators of the hyperparameters of the Bayesian model are developed, yielding empirical Bayes estimates of the piecewise constant parameters in the stochastic change- point model. The empirical Bayes approach is applied to the frequentist problem of partitioning the time series into segments under sparsity assumptions on the change-points. Simulation and empirical studies of its performance are also given.

Optimization of the Regularization in Background and Foreground Modeling

Background and foreground modeling is a typical method in the application of computer vision. The current general “low-rank + sparse” model decomposes the frames from the video sequences into low-rank background and sparse foreground. But the sparse assumption in such a model may not conform with the reality, and the model cannot directly reflect the correlation between the background and foreground either. Thus, we present a novel model to solve this problem by decomposing the arranged data matrixDinto low-rank backgroundLand moving foregroundM. Here, we only need to give the priori assumption of the background to be low-rank and let the foreground be separated from the background as much as possible. Based on this division, we use a pair of dual norms, nuclear norm and spectral norm, to regularize the foreground and background, respectively. Furthermore, we use a reweighted function instead of the normal norm so as to get a better and faster approximation model. Detailed explanation based on linear algebra about our two models will be presented in this paper. By the observation of the experimental results, we can see that our model can get better background modeling, and even simplified versions of our algorithms perform better than mainstream techniques IALM and GoDec.

Gene regulatory network inference by point-based Gaussian approximation filters incorporating the prior information

The extended Kalman filter (EKF) has been applied to inferring gene regulatory networks. However, it is well known that the EKF becomes less accurate when the system exhibits high nonlinearity. In addition, certain prior information about the gene regulatory network exists in practice, and no systematic approach has been developed to incorporate such prior information into the Kalman-type filter for inferring the structure of the gene regulatory network. In this paper, an inference framework based on point-based Gaussian approximation filters that can exploit the prior information is developed to solve the gene regulatory network inference problem. Different point-based Gaussian approximation filters, including the unscented Kalman filter (UKF), the third-degree cubature Kalman filter (CKF3), and the fifth-degree cubature Kalman filter (CKF5) are employed. Several types of network prior information, including the existing network structure information, sparsity assumption, and the range constraint of parameters, are considered, and the corresponding filters incorporating the prior information are developed. Experiments on a synthetic network of eight genes and the yeast protein synthesis network of five genes are carried out to demonstrate the performance of the proposed framework. The results show that the proposed methods provide more accurate inference results than existing methods, such as the EKF and the traditional UKF.

Effect of Heredity and Sparsity on Second-Order Screening Design Performance

Two recent and powerful screening designs are compared. The comparisons focus on how robust each design is with respect to assumptions of model heredity and sparsity. Four truth models having varied aspects of heredity and sparsity are used; each examined at four levels of random noise. Each screening method is analyzed against the random data using the method's proposed analysis approach and compared in terms of correctly identified model components, incorrectly identified model components (type I error), and missed model components (type II error). Copyright © 2013 John Wiley & Sons, Ltd.

On the interplay of basis smoothness and specific range conditions occurring in sparsity regularization

The convergence rates results in ℓ1-regularization when the sparsity assumption is narrowly missed, presented by Burger et al (2013 Inverse Problems 29 025013), are based on a crucial condition which requires that all basis elements belong to the range of the adjoint of the forward operator. Partly it was conjectured that such a condition is very restrictive. In this context, we study sparsity-promoting varieties of Tikhonov regularization for linear ill-posed problems with respect to an orthonormal basis in a separable Hilbert space using ℓ1 and sublinear penalty terms. In particular, we show that the corresponding range condition is always satisfied for all basis elements if the problems are well-posed in a certain weaker topology and the basis elements are chosen appropriately related to an associated Gelfand triple. The Radon transform, Symm’s integral equation and linear integral operators of Volterra type are examples for such behaviour, which allows us to apply convergence rates results for non-sparse solutions, and we further extend these results also to the case of non-convex ℓq-regularization with 0 < q < 1.

Better subset regression

This paper studies the relationship between model fitting and screening performance to find efficient screening methods for high-dimensional linear regression models. Under a sparsity assumption we show in a general asymptotic setting that a subset that includes the true submodel always yields a smaller residual sum of squares than those that do not. To seek such a subset, we consider the optimization problem associated with best subset regression. An em algorithm, known as orthogonalizing subset screening, and its accelerated version are proposed for searching for the best subset. Although the algorithms do not always find the best subset, their monotonicity makes the subset fit the data better than initial subsets, and thus the subset can have better screening performance asymptotically. Simulation results show that our methods are very competitive.