Least Angle Regression Research Articles

Medical image fusion based on sparse representation of classified image patches

Abstract Medical image fusion is one of the hot research in the field of medical imaging and radiation medicine, and is widely recognized by medical and engineering fields. In this paper, a new fusion scheme for medical images based on sparse representation of classified image patches is proposed. In this method, first, the registered source images are divided into classified patches according to the patch geometrical direction, from which the corresponding sub-dictionary is trained via the online dictionary learning (ODL) algorithm, and the least angle regression (LARS) algorithm is used to sparsely code each patch; second, the sparse coefficients are combined with the “choose-max” fusion rule; Finally, the fused image is reconstructed from the combined sparse coefficients and the corresponding sub-dictionary. The experimental results showed that the proposed method outperforms other methods in terms of both visual perception and objective evaluation metrics.

A speech enhancement method based on sparse reconstruction on log-spectra

ABSTRACTA new speech enhancement method using sparse reconstruction of the log-spectra is presented. Similar to the traditional sparse coding methods, the proposed algorithm makes use of the least angle regression (LARS) with a coherence criterion (LARC) algorithm to reconstruct the log power spectrum of clean speech. However, a new stopping criterion is introduced to allow the LARC algorithm to adapt to various background noise environments. In addition, a modified two-step noise reduction with a log-MMSE filter is applied which solves the bias of estimated a-priori signal-to-noise ratio (SNR). A notable improvement in the proposed algorithm over traditional speech enhancement methods is its adaptability to the changes in the SNR of noisy speech. The performance of the proposed algorithm is evaluated using standard measures based on a large set of speech and noise signals. The results show that a significant improvement is achieved compared to traditional approaches, especially in non-stationary noise en...

Basis-Adaptive Sparse Polynomial Chaos Expansion for Probabilistic Power Flow

This paper introduces the basis-adaptive sparse polynomial chaos (BASPC) expansion to perform the probabilistic power flow (PPF) analysis in power systems. The proposed method takes advantage of three state-of-the-art uncertainty quantification methodologies reasonably: the hyperbolic scheme to truncate the infinite polynomial chaos (PC) series; the least angle regression (LARS) technique to select the optimal degree of each univariate PC series; and the Copula to deal with nonlinear correlations among random input variables. Consequently, the proposed method brings appealing features to PPF, including the ability to handle the large-scale uncertainty sources; to tackle the nonlinear correlation among the random inputs; to analytically calculate representative statistics of the desired outputs; and to dramatically alleviate the computational burden as of traditional methods. The accuracy and efficiency of the proposed method are verified through either quantitative indicators or graphical results of PPF on both the IEEE European Low Voltage Test Feeder and the IEEE 123 Node Test Feeder, in the presence of more than 100 correlated uncertain input variables.

A Robust Multivariate EWMA Control Chart for Detecting Sparse Mean Shifts

In multivariate statistical process control (MSPC) applications, process mean shifts sometimes occur in only a few components. To solve this MSPC problem, many control charts were proposed in the literature. Most of these charts assumed that the multivariate quality characteristics are normally distributed. Among them, the control chart proposed by Zou and Qiu (2009), incorporating the least absolute shrinkage and selection operator (LASSO) method into the EWMA scheme, has the best overall performance. In this paper, we extend the classical multivariate LASSO control chart to a robust version that has an affine-invariance property and is distribution free under the family of elliptical direction distributions, indicating that the in-control run-length distribution is the same for any continuous distribution in this family and the control limit can be acquired from the multivariate standard normal distribution. Our simulation results show that the proposed method is very efficient in detecting various sparse shifts under heavy-tailed and skewed multivariate distributions. In addition, it is easy to implement with an iterative algorithm and the least angle regression (LARS) algorithm. White-wine data illustrates that the proposed control chart performs quite well in applications.

On scaling of soft-thresholding estimator

LASSO is known to have a problem of excessive shrinkage at a sparse representation. To analyze this problem in detail, in this paper, we consider a positive scaling for soft-thresholding estimators that are LASSO estimators in an orthogonal regression problem. We especially consider a non-parametric orthogonal regression problem which includes wavelet denosing. We first gave a risk (generalization error) of LARS (least angle regression) based soft-thresholding with a single scaling parameter. We then showed that an optimal scaling value that minimizes the risk under a sparseness condition is 1+O(logn/n), where n is the number of samples. The important point is that the optimal value of scaling is larger than one. This implies that expanding soft-thresholding estimator shows a better generalization performance compared to a naive soft-thresholding. This also implies that a risk of LARS-based soft-thresholding with the optimal scaling is smaller than without scaling. We then showed their difference is O(logn/n). This also shows an effectiveness of the introduction of scaling. Through simple numerical experiments, we found that LARS-based soft-thresholding with scaling can improve both of sparsity and generalization performance compared to a naive soft-thresholding.

Uncertainty Quantification Using Sparse Approximation for Models With a High Number of Parameters: Application to a Magnetoelectric Sensor

To face the curse of dimensionality met in uncertainty quantification problems when the model has a high number of random parameters, methods based on sparse approximation, such as the least angle regression (LAR) method, should be used. In this paper, we propose an extension of the LAR method, and we apply it to quantify the impact of uncertainties on the characteristics of the material on the magnetoelectric sensor performance. The sensor response is represented by a 2-D finite-element model with ten random parameters. A global sensitivity analysis is carried out in order to determine the most influential parameters.

Variable Selection With Prior Information for Generalized Linear Models via the Prior LASSO Method

LASSO is a popular statistical tool often used in conjunction with generalized linear models that can simultaneously select variables and estimate parameters. When there are many variables of interest, as in current biological and biomedical studies, the power of LASSO can be limited. Fortunately, so much biological and biomedical data have been collected and they may contain useful information about the importance of certain variables. This article proposes an extension of LASSO, namely, prior LASSO (pLASSO), to incorporate that prior information into penalized generalized linear models. The goal is achieved by adding in the LASSO criterion function an additional measure of the discrepancy between the prior information and the model. For linear regression, the whole solution path of the pLASSO estimator can be found with a procedure similar to the least angle regression (LARS). Asymptotic theories and simulation results show that pLASSO provides significant improvement over LASSO when the prior information is relatively accurate. When the prior information is less reliable, pLASSO shows great robustness to the misspecification. We illustrate the application of pLASSO using a real dataset from a genome-wide association study. Supplementary materials for this article are available online.

SURROGATE MODELING FOR STOCHASTIC DYNAMICAL SYSTEMS BY COMBINING NONLINEAR AUTOREGRESSIVE WITH EXOGENOUS INPUT MODELS AND POLYNOMIAL CHAOS EXPANSIONS

The application of polynomial chaos expansions (PCEs) to the propagation of uncertainties in stochastic dynamical models is well-known to face challenging issues. The accuracy of PCEs degenerates quickly in time. Thus maintaining a sufficient level of long-term accuracy requires the use of high-order polynomials. In numerous cases, it is even infeasible to obtain accurate metamodels with regular PCEs because PCEs cannot represent the dynamics. To overcome the problem, an original numerical approach was recently proposed that combines PCEs and non-linear autoregressive with exogenous input (NARX) models, which are a universal tool in the field of system identification. The approach relies on using NARX models to mimic the dynamical behavior of the system and dealing with the uncertainties using PCEs. The PC-NARX model was built by means of heuristic genetic algorithms. This paper aims at introducing the least angle regression (LARS) technique for computing PC-NARX models, which consists of solving two linear regression problems. The proposed approach is validated with structural mechanics case studies, in which uncertainties arising from both structures and excitations are taken into account. Comparison with Monte Carlo simulation and regular PCEs is also carried out to demonstrate the effectiveness of the proposed approach.

Prediction of enteric methane emissions from Holstein dairy cows fed various forage sources

Milk fatty acid (FA) profile has been previously used as a predictor of enteric CH4output in dairy cows fed diets supplemented with plant oils, which can potentially impact ruminal fermentation. The objective of this study was to investigate the relationships between milk FA and enteric CH4 emissions in lactating dairy cows fed different types of forages in the context of commonly fed diets. A total of 81 observations from three separate 3×3 Latin square design (32-day periods) experiments including a total of 27 lactating cows (96±27 days in milk; mean±SD) were used. Dietary forages were included at 60% of ration dry matter and were as follows: (1) 100% corn silage, (2) 100% alfalfa silage, (3) 100% barley silage, (4) 100% timothy silage, (5) 50 : 50 mix of corn and alfalfa silages, (6) 50 : 50 mix of barley and corn silages and (7) 50 : 50 mix of timothy and alfalfa silages. Enteric CH4output was measured using respiration chambers during 3 consecutive days. Milk was sampled during the last 7 days of each period and analyzed for components and FA profile. Test variables included dry matter intake (DMI; kg/day), NDF (%), ether extract (%), milk yield (kg/day), milk components (%) and individual milk FA (% of total FA). Candidate multivariate models were obtained using the Least Absolute Shrinkage and Selection Operator and Least-Angle Regression methods based on the Schwarz Bayesian Criterion. Data were then fitted into a random regression using the MIXED procedure including the random effects of cow, period and study. A positive correlation was observed between CH4 and DMI (r=0.59,P<0.001), whereas negative associations were observed between CH4 and cis9-17:1 (r=−0.58, P<0.001), and trans8, cis13-18:2 (r=−0.51,P<0.001). Three different candidate models were selected and the best fit candidate model predicted CH4 with a coefficient of determination of 0.84 after correction for cow, period and study effects and was: CH4 (g/day)=319.7−57.4×15:0−13.8×cis9-17:1−39.5×trans10-18:1−59.9×cis11-18:1−253.1×trans8, cis12-18:2−642.7×trans8, cis13-18:2−195.7×trans11, cis15-18:2+16.5×DMI. Overall and linear prediction biases of all models were not significant (P>0.19). Milk FA profile and DMI can be used to predict CH4emissions in dairy cows across a wide range of dietary forage sources

A comparative study for least angle regression on NIR spectra analysis to determine internal qualities of navel oranges

Internal qualities of navel oranges are the key factors for their market value and of major concern to customers. Unlike traditional subjective quality assessment, near infrared (NIR) spectroscopy based techniques are quantitative, convenient and non-destructive. Various machine learning methods have been applied to NIR spectra analysis to determine the fruit qualities. NIR spectra are usually of very high dimension. Explicit or implicit variable selection is essential to ensure prediction performance. Least angle regression (LAR) is a relatively new and efficient machine learning algorithm for regression analysis and is good for variable selection. We investigate the potential of the LAR algorithm for NIR spectra analysis to determine the internal qualities of navel oranges. A total of 1535 navel orange samples from 15 origins were prepared for NIR spectra collection and quality parameters measurement. Spectra are of 1500 dimensions with wavelengths ranging from 1000 nm to 2499 nm. The LAR was compared with the most widely used linear and nonlinear methods in three aspects: prediction accuracy, computational efficiency, and model interpretability. The results showed that the prediction performance of LAR was better than that of PLS, while slightly inferior to that of least squares support vector machines (LS-SVM). LAR was computationally more efficient than both PLS and LS-SVM. By concentrating on the most important predictors, LAR is much easier to reveal the most relevant predictors than PLS; LS-SVM was hardly interpretable because of its nonlinear kernel.

ConvexLAR: An Extension of Least Angle Regression

The least angle regression (LAR) was proposed by Efron, Hastie, Johnstone and Tibshirani in the year 2004 for continuous model selection in linear regression. It is motivated by a geometric argument and tracks a path along which the predictors enter successively and the active predictors always maintain the same absolute correlation (angle) with the residual vector. Although it gains popularity quickly, its extensions seem rare compared to the penalty methods. In this expository article, we show that the powerful geometric idea of LAR can be generalized in a fruitful way. We propose a ConvexLAR algorithm that works for any convex loss function and naturally extends to group selection and data adaptive variable selection. After simple modification, it also yields new exact path algorithms for certain penalty methods such as a convex loss function with lasso or group lasso penalty. Variable selection in recurrent event and panel count data analysis, Ada-Boost, and Gaussian graphical model is reconsidered from the ConvexLAR angle. Supplementary materials for this article are available online.

Unbiased Estimation for Linear Regression When n &lt; v

In this paper a new method is proposed for solving the linear regression problem when the number of observations $n$ is smaller than the number of predictors v. This method uses the idea of graphical models and provides unbiased parameter estimates under certain conditions, while existing methods such as ridge regression, LASSO and least angle regression (LARS) give biased estimates. Also the new method can provide a detailed graphical correlation structure for the predictors, therefore the real causal relationship between predictors and response could be identified. In contrast, existing methods often cannot identify the real important predictors which have possible causal effects on the response variable. Unlike the existing methods based on graphical models, the proposed method can identify the potential networks while doing regression even if the data do not follow a multivariate distribution. The new method is compared with some existing methods such as ridge regression, LASSO and LARS by using simulated and real data sets. Our experiments reveal that the new method outperforms all the other methods when n<v.

Optimizing the echo state network with a binary particle swarm optimization algorithm

The echo state network (ESN) is a novel and powerful method for the temporal processing of recurrent neural networks. It has tremendous potential for solving a variety of problems, especially real-valued, time-series modeling tasks. However, its complicated topologies and random reservoirs are difficult to implement in practice. For instance, the reservoir must be large enough to capture all data features given that the reservoir is generated randomly. To reduce network complexity and to improve generalization ability, we present a novel optimized ESN (O-ESN) based on binary particle swarm optimization (BPSO). Because the optimization of output weights connection structures is a feature selection problem and PSO has been used as a promising method for feature selection problems, BPSO is employed to determine the optimal connection structures for output weights in the O-ESN. First, we establish and train an ESN with sufficient internal units using training data. The connection structure of output weights, i.e., connection or disconnection, is then optimized through BPSO with validation data. Finally, the performance of the O-ESN is evaluated through test data. This performance is demonstrated in three different types of problems, namely, a system identification and two time-series benchmark tasks. Results show that the O-ESN outperforms the classical feature selection method, least angle regression (LAR) method in that its architecture is simpler than that of LAR.

Efficient Lasso training from a geometrical perspective

The Lasso (L1-penalized regression) has drawn great interests in machine learning and statistics due to its robustness and high accuracy. A variety of methods have been proposed for solving the Lasso. But for large scale problems, the presence of L1 norm constraint significantly impedes the efficiency. Inspired by recent theoretical and practical contributions on the close relation between Lasso and SVMs, we reformulate the Lasso as a problem of finding the nearest point in a polytope to the origin, which circumvents the L1 norm constraint. This problem can be solved efficiently from a geometric perspective using the Wolfe׳s method. Comparing with least angle regression (LARS), which is a conventional method to solve Lasso, the proposed algorithm is advantageous in both efficiency and numerical stability. Experimental results show that the proposed approach is competitive with other state-of-the-art Lasso solvers on large scale problems.

LARS NETWORK FILTRATION IN THE STUDY OF EEG BRAIN CONNECTIVITY.

In a brain network, weak and non-significant edge weights between nodes signal spurious connections and are often discarded by thresholding the weights. Traditional practice of thresholding edge weights at an arbitrary value can be problematic. Network filtration provides an alternative by summarizing the changes in the network topology with respect to a broad range of thresholds. A well established network filtration approach depends on the graphical-LASSO (least absolute shrinkage and selection operator) model, where a sequence of binary networks are obtained based on non-zero sparse inverse covariance (IC) estimates of partial correlations at a range of sparsity parameters. The limitation of the graphical-LASSO network model is that it relies on the structural information rather than actual entries of the sparse IC matrices and therefore can only yield approximate dynamic topological changes in the network. In the current study, we propose a new network filtration approach based on least angle regression (LARS) that gives exact filtration values at which the filtration changes and apply it to study brain connectivity in response to emotional stimuli across different age groups via electroencephalographic (EEG) data.

A new surrogate modeling technique combining Kriging and polynomial chaos expansions – Application to uncertainty analysis in computational dosimetry

In numerical dosimetry, the recent advances in high performance computing led to a strong reduction of the required computational time to assess the specific absorption rate (SAR) characterizing the human exposure to electromagnetic waves. However, this procedure remains time-consuming and a single simulation can request several hours. As a consequence, the influence of uncertain input parameters on the SAR cannot be analyzed using crude Monte Carlo simulation. The solution presented here to perform such an analysis is surrogate modeling. This paper proposes a novel approach to build such a surrogate model from a design of experiments. Considering a sparse representation of the polynomial chaos expansions using least-angle regression as a selection algorithm to retain the most influential polynomials, this paper proposes to use the selected polynomials as regression functions for the universal Kriging model. The leave-one-out cross validation is used to select the optimal number of polynomials in the deterministic part of the Kriging model. The proposed approach, called LARS-Kriging-PC modeling, is applied to three benchmark examples and then to a full-scale metamodeling problem involving the exposure of a numerical fetus model to a femtocell device. The performances of the LARS-Kriging-PC are compared to an ordinary Kriging model and to a classical sparse polynomial chaos expansion. The LARS-Kriging-PC appears to have better performances than the two other approaches. A significant accuracy improvement is observed compared to the ordinary Kriging or to the sparse polynomial chaos depending on the studied case. This approach seems to be an optimal solution between the two other classical approaches. A global sensitivity analysis is finally performed on the LARS-Kriging-PC model of the fetus exposure problem.

Forward and backward least angle regression for nonlinear system identification

A forward and backward least angle regression (LAR) algorithm is proposed to construct the nonlinear autoregressive model with exogenous inputs (NARX) that is widely used to describe a large class of nonlinear dynamic systems. The main objective of this paper is to improve model sparsity and generalization performance of the original forward LAR algorithm. This is achieved by introducing a replacement scheme using an additional backward LAR stage. The backward stage replaces insignificant model terms selected by forward LAR with more significant ones, leading to an improved model in terms of the model compactness and performance. A numerical example to construct four types of NARX models, namely polynomials, radial basis function (RBF) networks, neuro fuzzy and wavelet networks, is presented to illustrate the effectiveness of the proposed technique in comparison with some popular methods.

LARSEN-ELM: Selective ensemble of extreme learning machines using LARS for blended data

Extreme learning machine (ELM) as a neural network algorithm has shown its good performance, such as fast speed, simple structure etc, but also, weak robustness is an unavoidable defect in original ELM for blended data. We present a new machine learning framework called LARSEN-ELM for overcoming this problem. In our paper, we would like to show two key steps in LARSEN-ELM. In the first step, preprocessing, we select the input variables highly related to the output using least angle regression (LARS). In the second step, training, we employ Genetic Algorithm (GA) based selective ensemble and original ELM. In the experiments, we apply a sum of two sines and four datasets from UCI repository to verify the robustness of our approach. The experimental results show that compared with original ELM and other methods such as OP-ELM, GASEN-ELM and LSBoost, LARSEN-ELM significantly improve robustness performance while keeping a relatively high speed.

Improved simple deterministically constructed Cycle Reservoir Network with Sensitive Iterative Pruning Algorithm

Abstract Reservoir Computing (RC) is an effective approach to design and train recurrent neural networks, which is successfully and widely applied in real-valued time series modeling tasks. However, RC has been criticized for not being principled enough, namely the reservoir which is unlikely to be optimal because the reservoir connectivity and weight structure are created randomly. A new Simple Cycle Reservoir Network (SCRN) with deterministically constructed connectivity and weight structure can yield performance competitive with standard Echo State Network (ESN). In order to determine the proper size of the reservoir and improve generalization ability of SCRN, a Sensitive Iterated Pruning Algorithm (SIPA), in which a larger than necessary reservoir is employed firstly and then its size is reduced by pruning out the least sensitive internal units, is proposed to optimize the reservoir size and weights of SCRN. A system identification and two time-series benchmark tasks are applied to demonstrate the feasibility and superiority of SIPA. The results show that the SIPA method significantly outperforms a Least Angle Regression (LAR) method and SIPA is able to improve the generalization performance of SCRN. Besides, two well known reservoir characterizations, i.e. pseudo-Lyapunov exponent of the reservoir dynamics and Memory Capacity, and the impact of SIPA on two characterizations are investigated.

Unbalance identification using the least angle regression technique

The present investigation proposes a robust procedure for unbalance identification using the equivalent load method based on sparse vibration measurements. The procedure is demonstrated and benchmarked on an example rotor at constant speed. Since the number of measuring positions is much smaller than the number of possible fault locations, performing unbalance identification leads to an ill-posed problem. This problem was tried to be overcome previously with modal expansion in the time domain and with several linear regressions in the frequency domain. However, since the solution to the problem is a sparse equivalent force vector, these methods cannot provide a robust identification procedure. A robust identification can only be achieved by providing a-priori information on the number of unbalances to be identified. The presently proposed procedure achieves more precise unbalance identification without the need of a-priori information by incorporating a regularization technique. A well-known technique for producing sparse solutions is the Least Absolute Shrinkage and Selection Operator (LASSO). The proposed procedure is based on the generalized technique Least Angle Regression (LAR) which finds all the solutions of LASSO. A comparison of the time-domain approach, the frequency-domain approach and the proposed technique is made and the superiority of the latter technique in identifying the number of possible fault locations is highlighted. The selection of the threshold of the convergence algorithm of LAR as well as the selection of the value of the Lagrangian multiplier is discussed in some detail.