Sparse Linear Regression Problem Research Articles

Sparse regression for discovery of constitutive models from oscillatory shear measurements

We propose sparse regression as an alternative to neural networks for the discovery of parsimonious constitutive models (CMs) from oscillatory shear experiments. Symmetry and frame invariance are strictly imposed by using tensor basis functions to isolate and describe unknown nonlinear terms in the CMs. We generate synthetic experimental data using the Giesekus and Phan-Thien Tanner CMs and consider two different scenarios. In the complete information scenario, we assume that the shear stress, along with the first and second normal stress differences, is measured. This leads to a sparse linear regression problem that can be solved efficiently using l1 regularization. In the partial information scenario, we assume that only shear stress data are available. This leads to a more challenging sparse nonlinear regression problem, for which we propose a greedy two-stage algorithm. In both scenarios, the proposed methods fit and interpolate the training data remarkably well. Predictions of the inferred CMs extrapolate satisfactorily beyond the range of training data for oscillatory shear. They also extrapolate reasonably well to flow conditions like startup of steady and uniaxial extension that are not used in the identification of CMs. We discuss ramifications for experimental design, potential algorithmic improvements, and implications of the non-uniqueness of CMs inferred from partial information.

On Sparse Linear Regression in the Local Differential Privacy Model

In this paper, we study the sparse linear regression problem under the Local Differential Privacy (LDP) model. We first show that polynomial dependency on the dimensionality <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> of the space is unavoidable for the estimation error in both non-interactive and sequential interactive local models, if the privacy of the whole dataset needs to be preserved. Similar limitations also exist for other types of error measurements and in the relaxed local models. This indicates that differential privacy in high dimensional space is unlikely achievable for the problem. With the understanding of this limitation, we then present two algorithmic results. The first one is a sequential interactive LDP algorithm for the low dimensional sparse case, called Locally Differentially Private Iterative Hard Thresholding (LDP-IHT), which achieves a near optimal upper bound. This algorithm is actually rather general and can be used to solve quite a few other problems, such as (Local) DP-ERM with sparsity constraints and sparse regression with non-linear measurements. The second one is for the restricted (high dimensional) case where only the privacy of the responses (labels) needs to be preserved. For this case, we show that the optimal rate of the error estimation can be made logarithmically dependent on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> (i.e., <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\log p$ </tex-math></inline-formula> ) in the local model, where an upper bound is obtained by a label-privacy version of LDP-IHT. Experiments on real world and synthetic datasets confirm our theoretical analysis.

Sparse Wavelet Networks

A wavelet network (WN) is a feed-forward neural network that uses wavelets as activation functions for the neurons in its hidden layer. By predetermining the wavelet positions and dilations, the WN can turn into a linear regression model. The common approach for the construction of these WN families is to use least-squares type algorithms. In this letter, we propose a novel approach by formulating a WN as a sparse linear regression problem, which we call a sparse wavelet network (SWN). In this WN, the problem of calculating the unknown inner parameters of the network becomes that of finding the sparse solution of an under-determined system of linear equations. Our sparse solution algorithm is a non-convex sparse relaxation approach inspired by smoothed L0 (SL0), a distinguished sparse recovery algorithm. The proposed SWN can be applied as a tool for the prediction and identification of dynamical systems.

Sparse Green’s Functions Estimation Using Orthogonal Matching Pursuit: Application to Aeroacoustic Beamforming

The Paper presents a new methodology for the numerical estimation of the Green’s functions in complex external aeroacoustic configurations. Computational aeroacoustics is used to propagate multifrequency signals from focus points to microphones. The method takes advantage of the sparsity of the Green’s functions in the time domain to minimize the simulation time. It leads to a complex sparse linear regression problem. To solve it, the orthogonal matching pursuit algorithm is adapted. The method is first applied on the case of the diffraction by a rigid sphere. Results are studied both in terms of Green’s function estimation and aeroacoustic beamforming. They show that the Green’s functions are obtained with a good accuracy and enable localization of acoustic sources placed behind the diffracting object. The methodology is then applied on a NACA0012 two-dimensional wing in a potential flow for which the Green’s function is not known analytically. The use of the reverse-flow reciprocity principle enables reduction of the complexity of the estimation problem when there are more scan points than microphones. It is shown that it is possible to take advantage of the presence of diffracting objects to improve the capability of detection of a sensor array.

Accelerated Sparse Linear Regression via Random Projection

In this paper, we present an accelerated numerical method based on random projection for sparse linear regression. Previous studies have shown that under appropriate conditions, gradient-based methods enjoy a geometric convergence rate when applied to this problem. However, the time complexity of evaluating the gradient is as large as $\mathcal{O}(nd)$, where $n$ is the number of data points and $d$ is the dimensionality, making those methods inefficient for large-scale and high-dimensional dataset. To address this limitation, we first utilize random projection to find a rank-$k$ approximator for the data matrix, and reduce the cost of gradient evaluation to $\mathcal{O}(nk+dk)$, a significant improvement when $k$ is much smaller than $d$ and $n$. Then, we solve the sparse linear regression problem via a proximal gradient method with a homotopy strategy to generate sparse intermediate solutions. Theoretical analysis shows that our method also achieves a global geometric convergence rate, and moreover the sparsity of all the intermediate solutions are well-bounded over the iterations. Finally, we conduct experiments to demonstrate the efficiency of the proposed method.

A Sparse Bayesian Approach to the Identification of Nonlinear State-Space Systems

This technical note considers the identification of nonlinear discrete-time systems with additive process noise but without measurement noise. In particular, we propose a method and its associated algorithm to identify the system nonlinear functional forms and their associated parameters from a limited number of time-series data points. For this, we cast this identification problem as a sparse linear regression problem and take a Bayesian viewpoint to solve it. As such, this approach typically leads to nonconvex optimisations. We propose a convexification procedure relying on an efficient iterative re-weighted $\ell_1$-minimisation algorithm that uses general sparsity inducing priors on the parameters of the system and marginal likelihood maximisation. Using this approach, we also show how convex constraints on the parameters can be easily added to our proposed iterative re-weighted $\ell_1$-minimisation algorithm. In the supplementary material \cite{appendix}, we illustrate the effectiveness of the proposed identification method on two classical systems in biology and physics, namely, a genetic repressilator network and a large scale network of interconnected Kuramoto oscillators.

A Dirty Model for Multiple Sparse Regression

The task of sparse linear regression consists of finding an unknown sparse vector from linear measurements. Solving this task even under “high-dimensional” settings, where the number of samples is fewer than the number of variables, is now known to be possible via methods such as the LASSO. We consider the multiple sparse linear regression problem, where the task consists of recovering several related sparse vectors at once. A simple approach to this task would involve solving independent sparse linear regression problems, but a natural question is whether one can reduce the overall number of samples required by leveraging partial sharing of the support sets, or nonzero patterns, of the signal vectors. A line of recent research has studied the use of l1/lq norm block-regularizations with q > 1 for such problems. However, depending on the level of sharing, these could actually perform worse in sample complexity when compared to solving each problem independently. We present a new “adaptive” method for multiple sparse linear regression that can leverage support and parameter overlap when it exists, but not pay a penalty when it does not. We show how to achieve this using a very simple idea: decompose the parameters into two components and regularize these differently. We show, theoretically and empirically, that our method strictly and noticeably outperforms both l1 or l1/lq methods, over the entire range of possible overlaps (except at boundary cases, where we match the best method), even under high-dimensional scaling.

Sound Field Reproduction using the Lasso

Reproducing a sampled sound field using an array of loudspeakers is a problem with well-appreciated applications to acoustics and ultrasound treatment. Loudspeaker signal design has traditionally relied on (possibly regularized) least-squares (LS) criteria. In many cases however, the desired sound field can be reproduced using only a few loudspeakers, which are sparsely distributed in space. To exploit this feature, the fresh look advocated here permeates benefits from advances in variable selection and compressive sampling to sound field synthesis by formulating a sparse linear regression problem that is solved using the least-absolute shrinkage and selection operator (Lasso). An efficient implementation of the Lasso for the problem at hand is developed based on a coordinate descent iteration. Analysis and simulations demonstrate that Lasso-based sound field reproduction yields better performance than LS especially at high frequencies and for reproduction of under-sampled sound fields. In addition, Lasso-based synthesis enables judicious placement of loudspeaker arrays.

Sparsity-Aware Estimation of CDMA System Parameters

The number of active users, their timing offsets, and their (possibly dispersive) channels with the access point are performance-critical parameters for wireless code division multiple access (CDMA). Estimating them as accurately as possible using as short as possible training sequences can markedly improve error performance as well as the capacity of CDMA systems. The fresh look advocated here permeates benefits from recent advances in variable selection and compressive sampling approaches to multiuser communications by casting estimation of these parameters as a sparse linear regression problem. Novel estimators are developed by exploiting two forms of sparsity present: the first emerging from user (in)activity, and the second from the uncertainty on user delays and channel taps. Simulated tests demonstrate a large gain in performance when sparsity-aware estimators of CDMA parameters are compared to sparsity-agnostic standard least-squares-based alternatives.

A linear programming approach for estimating the structure of a sparse linear genetic network from transcript profiling data

BackgroundA genetic network can be represented as a directed graph in which a node corresponds to a gene and a directed edge specifies the direction of influence of one gene on another. The reconstruction of such networks from transcript profiling data remains an important yet challenging endeavor. A transcript profile specifies the abundances of many genes in a biological sample of interest. Prevailing strategies for learning the structure of a genetic network from high-dimensional transcript profiling data assume sparsity and linearity. Many methods consider relatively small directed graphs, inferring graphs with up to a few hundred nodes. This work examines large undirected graphs representations of genetic networks, graphs with many thousands of nodes where an undirected edge between two nodes does not indicate the direction of influence, and the problem of estimating the structure of such a sparse linear genetic network (SLGN) from transcript profiling data.ResultsThe structure learning task is cast as a sparse linear regression problem which is then posed as a LASSO (l1-constrained fitting) problem and solved finally by formulating a Linear Program (LP). A bound on the Generalization Error of this approach is given in terms of the Leave-One-Out Error. The accuracy and utility of LP-SLGNs is assessed quantitatively and qualitatively using simulated and real data. The Dialogue for Reverse Engineering Assessments and Methods (DREAM) initiative provides gold standard data sets and evaluation metrics that enable and facilitate the comparison of algorithms for deducing the structure of networks. The structures of LP-SLGNs estimated from the INSILICO1, INSILICO2 and INSILICO3 simulated DREAM2 data sets are comparable to those proposed by the first and/or second ranked teams in the DREAM2 competition. The structures of LP-SLGNs estimated from two published Saccharomyces cerevisae cell cycle transcript profiling data sets capture known regulatory associations. In each S. cerevisiae LP-SLGN, the number of nodes with a particular degree follows an approximate power law suggesting that its degree distributions is similar to that observed in real-world networks. Inspection of these LP-SLGNs suggests biological hypotheses amenable to experimental verification.ConclusionA statistically robust and computationally efficient LP-based method for estimating the topology of a large sparse undirected graph from high-dimensional data yields representations of genetic networks that are biologically plausible and useful abstractions of the structures of real genetic networks. Analysis of the statistical and topological properties of learned LP-SLGNs may have practical value; for example, genes with high random walk betweenness, a measure of the centrality of a node in a graph, are good candidates for intervention studies and hence integrated computational – experimental investigations designed to infer more realistic and sophisticated probabilistic directed graphical model representations of genetic networks. The LP-based solutions of the sparse linear regression problem described here may provide a method for learning the structure of transcription factor networks from transcript profiling and transcription factor binding motif data.