Lasso Problem Research Articles

Modified Double Inertial Extragradient-like Approaches for Convex Bilevel Optimization Problems with VIP and CFPP Constraints

Convex bilevel optimization problems (CBOPs) exhibit a vital impact on the decision-making process under the hierarchical setting when image restoration plays a key role in signal processing and computer vision. In this paper, a modified double inertial extragradient-like approach with a line search procedure is introduced to tackle the CBOP with constraints of the CFPP and VIP, where the CFPP and VIP represent a common fixed point problem and a variational inequality problem, respectively. The strong convergence analysis of the proposed algorithm is discussed under certain mild assumptions, where it constitutes both sections that possess a mutual symmetry structure to a certain extent. As an application, our proposed algorithm is exploited for treating the image restoration problem, i.e., the LASSO problem with the constraints of fractional programming and fixed-point problems. The illustrative instance highlights the specific advantages and potential infect of the our proposed algorithm over the existing algorithms in the literature, particularly in the domain of image restoration.

A graph decomposition-based approach for the graph-fused lasso

We propose a new algorithm for solving the graph-fused lasso (GFL), a regularized model that operates under the assumption that the signal tends to be locally constant over a predefined graph structure. The proposed method applies a novel decomposition of the objective function for the alternating direction method of multipliers (ADMM) algorithm. While ADMM has been widely used in fused lasso problems, existing works such as the network lasso decompose the objective function into the loss function component and the total variation penalty component. In contrast, based on the graph matching technique in graph theory, we propose a new method of decomposition that separates the objective function into two components, where one component is the loss function plus part of the total variation penalty, and the other component is the remaining total variation penalty. We develop an exact convergence rate of the proposed algorithm by developing a general theory on the local convergence of ADMM. Compared with the network lasso algorithm, our algorithm has a faster exact linear convergence rate (although in the same order as for the network lasso). It also enjoys a smaller computational cost per iteration, thus converges overall faster in most numerical examples.

Modified subgradient extragradient method for solution of split variational inequality problem with application to generalized Nash equilibrium problem

In this research, we propose modified subgradient extragradient method to find the common solution of split variational inequality problem and common fixed point problem of a finite family of demicontractive mappings without knowing norm of bounded linear operator and Lipschitz constants of the associated pseudomonotone operators. We obtain the strong convergence result for the proposed method under certain mild conditions. This result extends and generalizes many recent results in this direction. Furthermore, we demonstrate how our findings may be used to solve a generalized Nash equilibrium problem and LASSO problem. Finally, we give numerical experiments to validate the performance of our algorithm and compare it with other existing methods.

Relaxed Inertial Projective Forward-backward Splitting Algorithms for Regularized Least Square Problems

This study presents flexible conditions of the inertial extrapolation parameter for easy implementation that is added to the algorithm in faster convergence. Modified inertial forward-backward splitting algorithms for solving variational inclusion problems are introduced to apply to solve the LASSO problem for image restoration and signal recovery, and the elastic net model for classification problem. Projection methods are used in the final step for narrowing down the search field, resulting in a better solution.

Grid-free algorithms for direction-of-arrival trajectory localization.

Direction-of-arrival (DOA) estimation algorithms are crucial in localizing acoustic sources. Traditional localization methods rely on block-level processing to extract the directional information from multiple measurements processed together. However, these methods assume that DOA remains constant throughout the block, which may not be true in practical scenarios. Also, the performance of localization methods is limited when the true parameters do not lie on the parameter search grid. In this paper, two trajectory models are proposed, namely the polynomial and harmonic trajectory models, to capture the DOA dynamics. To estimate trajectory parameters, two gridless algorithms are adopted: (i) Sliding Frank-Wolfe (SFW), which solves the Beurling LASSO problem, and (ii) Newtonized orthogonal matching pursuit (NOMP), which is improved over orthogonal matching pursuit (OMP) using cyclic refinement. Furthermore, our analysis is extended to include multi-frequency processing. The proposed models and algorithms are validated using both simulated and real-world data. The results indicate that the proposed trajectory localization algorithms exhibit improved performance compared to grid-based methods in terms of resolution, robustness to noise, and computational efficiency.

Fixed-time neural networks with time-invariant and time-varying coefficients for mixed variational inequalities

This paper develops several fixed-time neural networks for solving mixed variational inequalities (MVIs). The proposed networks are highly efficient and with fixed-time convergence. First, based on the conventional forward-backward-forward neural network (FNN) and sliding mode control technique, a time-invariant fixed-time FNN (FxTFNN) is designed. Next, the Euclidean norm of FNN is introduced into FxTFNN to design the modified FxTFNN (MFxTFNN). It is shown that the proposed FxTFNN and MFxTFNN have fixed-time convergence properties and their settling-time functions are independent of the initial values. The proposed FxTFNN and MFxTFNN can be used to solve the Lasso problem and apply sparse signal reconstruction and image reconstruction. In addition, by introducing time-varying coefficients based on FxTFNN and MFxTFNN, time-varying FxTFNN (TFxTFNN) and time-varying MFxTFNN (TMFxTFNN) are developed. Finally, experimental results of numerical example, signal reconstruction, and image reconstruction are used to verify the effectiveness and superiority of the proposed neural networks.

Algorithm Research on Generalized Lasso Problems with Linear Equality Constraints

Algorithm Research on Generalized Lasso Problems with Linear Equality Constraints

Event-Triggered Distribution System State Estimation: Sparse Kalman Filtering With Reinforced Coupling

A novel distribution system state estimation (DSSE) method is proposed for power distribution networks with low-observablity, where the measurements come from only a few distribution-level phasor measurement units (D-PMUs). The proposed DSSE method is event-triggered, which means the state variables are updated based on the information that is extracted from the events in the power distribution system. In this regard, the estimations of the state variables during the previous events are used as priori information to predict the state variables at the current event. Accordingly, a novel data-driven method based on elastic net regression analysis is proposed to learn the event-triggered state transition matrix. The DSSE problem is formulated as a generalized group Lasso problem, which is augmented based on the knowledge on the sparsity patterns of the state variables that are extracted from the analysis of the events. Here, in the absence of direct power measurements, we enhance our ability in sparse recovery by developing a new reinforced physics-based coupling method among the state variables, in which we add a novel set of linear differential power flow equations to the DSSE problem formulation in forms of virtual measurements. Finally, two different approaches are proposed to solve the formulated sparse event-triggered DSSE problem. The first approach is exact but computationally expensive, as it requires conducting a batch alternating direction method of multipliers (ADMM) analysis. The second approach is approximate, but it is much faster as it works based on a novel modified Kalman filter/smoother in the presence of ℓ1-norm.

A Fast Solution to the Lasso Problem with Equality Constraints

The equality-constrained lasso problem augments the standard lasso by imposing additional structure on regression coefficients. Despite the broad utilities of the equality-constrained lasso, existing algorithms are typically computationally inefficient and only applicable to linear and logistic models. In this article, we devise a fast solution to the equality-constrained lasso problem with a two-stage algorithm: first obtaining candidate covariate subsets of increasing size from unconstrained lasso problems and then leveraging an efficient combined alternating direction method of multipliers/Newton-Raphson algorithm. Our proposed algorithm leads to substantial speedups in getting the solution path of the constrained lasso and can be easily adapted to generalized linear models and Cox proportional hazards models. We conduct extensive simulation studies to demonstrate the computational advantage of the proposed method over existing solvers. To further show the unique utility of our method, we consider two real-world data examples: a microbiome regression analysis and a myeloma survival analysis; neither example could be solved by naively fitting the constrained lasso problem on the full predictor set. Supplementary materials for this article are available online.

Multichannel seismic data attenuation compensation via curvelet‐based sparsity promotion

AbstractDue to subsurface viscosity and heterogeneity, the vertical resolution of observed seismic data is decreased after wave propagation, generating nonstationary seismic data with amplitude attenuation and phase distortion. Inverse Q filtering techniques are always used to enhance the vertical resolution of seismic data. However, the majority of inverse Q filtering methods treat attenuation compensation trace by trace, which may produce non‐robust compensation results with poor transverse continuity and amplify noise energy in noisy cases. Thus, we develop a novel sparsity‐promoting inversion‐based multichannel seismic data attenuation compensation approach by introducing a sparse constraint for curvelet coefficients of multichannel compensated data, which takes the transverse continuity of compensated data into account. Besides, the proposed method with a sparse constraint for curvelet coefficients has a better noise‐resistance property, which can attenuate the noise energy in noisy cases during attenuation compensation, improving compensation accuracy and robustness. To improve its computational efficiency, a fast iterative shrinkage–thresholding algorithm is adopted to solve the established lasso problem. Synthetic data examples with different noise levels and two post‐stack field data examples validate the effectiveness of the proposed multichannel method. Its compensation results have superior vertical resolution, transverse continuity and noise robustness in comparison to the conventional single‐channel compensation method using a damped least squares algorithm.

An Inexact Semismooth Newton-Based Augmented Lagrangian Algorithm for Multi-Task Lasso Problems

This paper is concerned with the [Formula: see text]-norm ball constrained multi-task learning problem, which has received extensive attention in many research areas such as machine learning, cognitive neuroscience, and signal processing. To address the challenges of solving large-scale multi-task Lasso problems, this paper develops an inexact semismooth Newton-based augmented Lagrangian (Ssnal) algorithm. When solving the inner problems in the Ssnal algorithm, the semismooth Newton (Ssn) algorithm with superlinear or even quadratic convergence is applied. Theoretically, this paper presents the global and asymptotically superlinear local convergence of the Ssnal algorithm under standard conditions. Computationally, we derive an efficient procedure to construct the generalized Jacobian of the projector onto [Formula: see text]-norm ball, which is an important component of the Ssnal algorithm, making the computational cost in the Ssn algorithm very cheap. Comprehensive numerical experiments on the multi-task Lasso problems demonstrate that the Ssnal algorithm is more efficient and robust than several existing state-of-the-art first-order algorithms.

The reciprocal elastic net

A new penalized likelihood method (reciprocal elastic net) is put forward for regularization and variable selection. Our proposal is based on a new class of reciprocal penalty functions, combining the strengths of the reciprocal LASSO regularization and ridge regression. We formulate the reciprocal elastic net problem as an equivalent reciprocal LASSO problem on augmented data, facilitating a direct utilization of the reciprocal LASSO algorithm to generate the entire reciprocal elastic net solution path. We further present the reciprocal adaptive elastic net, fusing the merits of ridge regression with the adaptively weighted reciprocal LASSO regularization. These methods, illustrated through simulated examples and real data analysis, demonstrate satisfactory performance in various diversified scenarios compared to published methods. Finally, we propose Bayesian methods to solve the reciprocal elastic net and reciprocal adaptive elastic net models using Gibbs samplers.

A highly efficient algorithm for solving exclusive lasso problems

The exclusive lasso (also known as elitist lasso) regularizer has become popular recently due to its superior performance on intra-group feature selection. Its complex nature poses difficulties for the computation of high-dimensional machine learning models involving such a regularizer. In this paper, we propose a highly efficient dual Newton method based proximal point algorithm (PPDNA) for solving large-scale exclusive lasso models. As important ingredients, we systematically study the proximal mapping of the weighted exclusive lasso regularizer and the corresponding generalized Jacobian. These results also make popular first-order algorithms for solving exclusive lasso models more practical. Extensive numerical results are presented to demonstrate the superior performance of the PPDNA against other popular numerical algorithms for solving the exclusive lasso problems.

A Dual Semismooth Newton Based Augmented Lagrangian Method for Large-Scale Linearly Constrained Sparse Group Square-Root Lasso Problems

A Dual Semismooth Newton Based Augmented Lagrangian Method for Large-Scale Linearly Constrained Sparse Group Square-Root Lasso Problems

On the selection of predictors by using greedy algorithms and information theoretic criteria

SummaryWe discuss the use of the following greedy algorithms in the prediction of multivariate time series: Matching Pursuit Algorithm (MPA), Orthogonal Matching Pursuit (OMP), Relaxed Matching Pursuit (RMP), Frank–Wolfe Algorithm (FWA) and Constrained Matching Pursuit (CMP). The last two are known to be solvers for the lasso problem. Some of the algorithms are well‐known (e.g. OMP), while others are less popular (e.g. RMP). We provide a unified presentation of all the algorithms, and evaluate their computational complexity for the high‐dimensional case and for the big data case. We show how 12 information theoretic (IT) criteria can be used jointly with the greedy algorithms. As part of this effort, we derive new theoretical results that allow modification of the IT criteria such that to be compatible with RMP. The prediction capabilities are tested in experiments with two data sets. The first one involves air pollution data measured in Auckland (New Zealand) and the second one concerns the House Price Index in England (the United Kingdom).

Accelerate the warm-up stage in the Lasso computation via a homotopic approach

In optimization of the least absolute shrinkage and selection operator (Lasso) problem, the fastest algorithm has a convergence rate of O(1/ϵ). This polynomial order of 1/ϵ is caused by the undesirable behavior of the absolute function at the origin. To expedite the convergence, an algorithm called homotopy shrinkage yielding (HOSKY) is proposed. It helps expedite the warm-up stage of the existing algorithms. With the acceleration by HOSKY in the warm-up stage, one can get a provable convergence rate lower than O(1/ϵ). The main idea of the proposed HOSKY algorithm is to use a sequence of surrogate functions to approximate the ℓ1 penalty that is used in Lasso. This sequence of surrogate functions, on the one hand, gets closer and closer to the ℓ1 penalty; on the other hand, they are strictly convex and well-conditioned, which enables a provable exponential rate of convergence by gradient-based approaches. As the proof shows, the convergence rate of the HOSKY algorithm is O([log⁡(1/ϵw)]2), where ϵw is the precision used in the warm-up stage (ϵw↛0). Additionally, the numerical simulations also show that HOSKY empirically performs better in the warm-up stage and accelerates the overall convergence rate.

SF-SGL: Solver-Free Spectral Graph Learning From Linear Measurements

This work introduces a highly-scalable spectral graph densification framework (SGL) for learning resistor networks with linear measurements, such as node voltages and currents. We show that the proposed graph learning approach is equivalent to solving the classical graphical Lasso problems with Laplacian-like precision matrices. We prove that given $O(\log N)$ pairs of voltage and current measurements, it is possible to recover sparse $N$-node resistor networks that can well preserve the effective resistance distances on the original graph. In addition, the learned graphs also preserve the structural (spectral) properties of the original graph, which can potentially be leveraged in many circuit design and optimization tasks. To achieve more scalable performance, we also introduce a solver-free method (SF-SGL) that exploits multilevel spectral approximation of the graphs and allows for a scalable and flexible decomposition of the entire graph spectrum (to be learned) into multiple different eigenvalue clusters (frequency bands). Such a solver-free approach allows us to more efficiently identify the most spectrally-critical edges for reducing various ranges of spectral embedding distortions. Through extensive experiments for a variety of real-world test cases, we show that the proposed approach is highly scalable for learning sparse resistor networks without sacrificing solution quality. We also introduce a data-driven EDA algorithm for vectorless power/thermal integrity verifications to allow estimating worst-case voltage/temperature (gradient) distributions across the entire chip by leveraging a few voltage/temperature measurements.

Regularization Solver Guided FISTA for Electrical Impedance Tomography.

Electrical impedance tomography (EIT) is non-destructive monitoring technology that can visualize the conductivity distribution in the observed area. The inverse problem for imaging is characterized by a serious nonlinear and ill-posed nature, which leads to the low spatial resolution of the reconstructions. The iterative algorithm is an effective method to deal with the imaging inverse problem. However, the existing iterative imaging methods have some drawbacks, such as random and subjective initial parameter setting, very time consuming in vast iterations and shape blurring with less high-order information, etc. To solve these problems, this paper proposes a novel fast convergent iteration method for solving the inverse problem and designs an initial guess method based on an adaptive regularization parameter adjustment. This method is named the Regularization Solver Guided Fast Iterative Shrinkage Threshold Algorithm (RS-FISTA). The iterative solution process under the L1-norm regular constraint is derived in the LASSO problem. Meanwhile, the Nesterov accelerator is introduced to accelerate the gradient optimization race in the ISTA method. In order to make the initial guess contain more prior information and be independent of subjective factors such as human experience, a new adaptive regularization weight coefficient selection method is introduced into the initial conjecture of the FISTA iteration as it contains more accurate prior information of the conductivity distribution. The RS-FISTA method is compared with the methods of Landweber, CG, NOSER, Newton-Raphson, ISTA and FISTA, six different distributions with their optimal parameters. The SSIM, RMSE and PSNR of RS-FISTA methods are 0.7253, 3.44 and 37.55, respectively. In the performance test of convergence, the evaluation metrics of this method are relatively stable at 30 iterations. This shows that the proposed method not only has better visualization, but also has fast convergence. It is verified that the RS-FISTA algorithm is the better algorithm for EIT reconstruction from both simulation and physical experiments.

A derivative‐free projection method for nonlinear equations with non‐Lipschitz operator: Application to LASSO problem

In this paper, we introduce a derivative‐free iterative method for finding the solutions of convex constrained nonlinear equations (CCNE) using the projection strategy. The new approach is free from gradient evaluations at each iteration. Also, the search direction generated by the proposed method satisfies the sufficient descent property, which is independent of the line search. Compared with traditional methods for solving CCNE that assumes Lipschitz continuity and monotonicity to establish the global convergence result, an advantage of our proposed method is that the global convergence result does not require the assumption of Lipschitz continuity. Moreover, the underlying operator is assumed to be pseudomonotone, which is a milder condition than monotonicity. As an applications, we solve the LASSO problem in compressed sensing. Numerical experiments illustrate the performances of our proposed algorithm and provide a comparison with related algorithms.

Efficient feature extraction from highly sparse binary genotype data for cancer prognosis prediction using an auto-encoder.

Genomics involving tens of thousands of genes is a complex system determining phenotype. An interesting and vital issue is how to integrate highly sparse genetic genomics data with a mass of minor effects into a prediction model for improving prediction power. We find that the deep learning method can work well to extract features by transforming highly sparse dichotomous data to lower-dimensional continuous data in a non-linear way. This may provide benefits in risk prediction-associated genotype data. We developed a multi-stage strategy to extract information from highly sparse binary genotype data and applied it for cancer prognosis. Specifically, we first reduced the size of binary biomarkers via a univariable regression model to a moderate size. Then, a trainable auto-encoder was used to learn compact features from the reduced data. Next, we performed a LASSO problem process to select the optimal combination of extracted features. Lastly, we applied such feature combination to real cancer prognostic models and evaluated the raw predictive effect of the models. The results indicated that these compressed transformation features could better improve the model's original predictive performance and might avoid an overfitting problem. This idea may be enlightening for everyone involved in cancer research, risk reduction, treatment, and patient care via integrating genomics data.