Proximal Gradient Research Articles

Sharper Bounds for Proximal Gradient Algorithms with Errors

Sharper Bounds for Proximal Gradient Algorithms with Errors

Delay-tolerant distributed Bregman proximal algorithms

Many problems in machine learning write as the minimization of a sum of individual loss functions over the training examples. These functions are usually differentiable but, in some cases, their gradients are not Lipschitz continuous, which compromises the use of (proximal) gradient algorithms. Fortunately, changing the geometry and using Bregman divergences can alleviate this issue in several applications, such as for Poisson linear inverse problems. However, the Bregman operation makes the aggregation of several points and gradients more involved, hindering the distribution of computations for such problems. In this paper, we propose an asynchronous variant of the Bregman proximal-gradient method, able to adapt to any centralized computing system. In particular, we prove that the algorithm copes with arbitrarily long delays and we illustrate its behaviour on distributed Poisson inverse problems.

Composite Optimization With Coupling Constraints via Penalized Proximal Gradient Method in Asynchronous Networks

In this paper, we consider a composite optimization problem with linear coupling constraints in a multi-agent network. In this problem, the agents cooperatively optimize a strongly convex cost function which is the linear sum of individual cost functions composed of smooth and possibly non-smooth components. To solve this problem, we propose an asynchronous penalized proximal gradient (Asyn-PPG) algorithm, a variant of classical proximal gradient method, with the presence of the asynchronous updates of the agents and uniform communication delays in the network. Specifically, we consider a slot-based asynchronous network (SAN), where the whole time domain is split into sequential time slots and each agent is permitted to execute multiple updates during a slot by accessing the historical state information of the agents. By the Asyn-PPG algorithm, an explicit convergence rate can be guaranteed based on deterministic analysis. The feasibility of the proposed algorithm is verified by solving a consensus-based distributed regression problem and a social welfare optimization problem in the electricity market.

A variable metric proximal stochastic gradient method: An application to classification problems

Due to the continued success of machine learning and deep learning in particular, supervised classification problems are ubiquitous in numerous scientific fields. Training these models typically involves the minimization of the empirical risk over large data sets along with a possibly non-differentiable regularization. In this paper, we introduce a stochastic gradient method for the considered classification problem. To control the variance of the objective's gradients, we use an automatic sample size selection along with a variable metric to precondition the stochastic gradient directions. Further, we utilize a non-monotone line search to automatize step size selection. Convergence results are provided for both convex and non-convex objective functions. Extensive numerical experiments verify that the suggested approach performs on par with state-of-the-art methods for training both statistical models for binary classification and artificial neural networks for multi-class image classification. The code is publicly available at https://github.com/koblererich/lisavm.

Adaptive fixed-time proximal gradient method for non-smooth optimization: the fractional approach

Non-smooth optimization has now played an important role in many fields, such as l1-regulation problem and sparse recovery problem. In this paper, a novel fractional fixed-time proximal gradient method is proposed for solving the composite optimization problems with non-smooth terms. At first, according to the distribution of zeroes of Mittag-Leffler functions, a novel fixed-time convergence theory with a fractional adaptive gain is proposed. Based on the optimal condition analysis of the transformed problem using the augmented Lagrangian function, a novel fixed-time proximal gradient method is then proposed and the fixed-time convergence property is rigorously proven. Furthermore, the proposed algorithm is applied to solve the sparse recovery problem in compressive sensing, and the restricted isometry condition required for the application of the algorithm is analyzed to achieve fixed-time recovery of the signal. Finally, simulation examples are provided to validate all the results.

Convergence of proximal gradient method with alternated inertial step for minimization problem

Convergence of proximal gradient method with alternated inertial step for minimization problem

A proximal gradient method with Bregman distance in multi-objective optimization

A proximal gradient method with Bregman distance in multi-objective optimization

A variant of the proximal gradient method for constrained convex minimization problems

A variant of the proximal gradient method for constrained convex minimization problems

Sparse Approximation of Complex Networks

This paper considers the problem of recovering a sparse approximation A∈Rn×n of an unknown (exact) adjacency matrix Atrue for a network from a corrupted version of a communicability matrix C=exp⁡(Atrue)+N∈Rn×n, where N denotes an unknown “noise matrix”. We consider two methods for determining an approximation A of Atrue: (i) a Newton method with soft-thresholding and line search, and (ii) a proximal gradient method with line search. These methods are applied to compute the solution of the minimization problemarg⁡minA∈Rn×n{‖exp⁡(A)−C‖F2+μ‖vec(A)‖1}, where μ>0 is a regularization parameter that controls the amount of shrinkage. We discuss the effect of μ on the computed solution, conditions for convergence, and the rate of convergence of the methods. Computed examples illustrate their performance when applied to directed and undirected networks.

A stochastic proximal gradient method for linear hyperspectral unmixing

A stochastic proximal gradient method for linear hyperspectral unmixing

Iterative-Weighted Thresholding Method for Group-Sparsity-Constrained Optimization With Applications.

Taking advantage of the natural grouping structure inside data, group sparse optimization can effectively improve the efficiency and stability of high-dimensional data analysis, and it has wide applications in a variety of fields such as machine learning, signal processing, and bioinformatics. Although there has been a lot of progress, it is still a challenge to construct a group sparse-inducing function with good properties and to identify significant groups. This article aims to address the group-sparsity-constrained minimization problem. We convert the problem to an equivalent weighted lp,q -norm ( , ) constrained optimization model, instead of its relaxation or approximation problem. Then, by applying the proximal gradient method, a solution method with theoretical convergence analysis is developed. Moreover, based on the properties proved in the Lagrangian dual framework, the homotopy technique is employed to cope with the parameter tuning task and to ensure that the output of the proposed homotopy algorithm is an L -stationary point of the original problem. The proposed weighted framework, with the central idea of identifying important groups, is compatible with a wide range of support set identification strategies, which can better meet the needs of different applications and improve the robustness of the model in practice. Both simulated and real data experiments demonstrate the superiority of the proposed method in terms of group feature selection accuracy and computational efficiency. Extensive experimental results in application areas such as compressed sensing, image recognition, and classifier design show that our method has great potential in a wide range of applications. Our codes will be available at https://github.com/jianglanfan/HIWT-GSC.

A Fixed-Time Proximal Gradient Neurodynamic Network With Time-Varying Coefficients for Composite Optimization Problems and Sparse Optimization Problems With Log-Sum Function.

This article presents a novel proximal gradient neurodynamic network (PGNN) for solving composite optimization problems (COPs). The proposed PGNN with time-varying coefficients can be flexibly chosen to accelerate the network convergence. Based on PGNN and sliding mode control technique, the proposed time-varying fixed-time proximal gradient neurodynamic network (TVFxPGNN) has fixed-time stability and a settling time independent of the initial value. It is further shown that fixed-time convergence can be achieved by relaxing the strict convexity condition via the Polyak-Lojasiewicz condition. In addition, the proposed TVFxPGNN is being applied to solve the sparse optimization problems with the log-sum function. Furthermore, the field-programmable gate array (FPGA) circuit framework for time-varying fixed-time PGNN is implemented, and the practicality of the proposed FPGA circuit is verified through an example simulation in Vivado 2019.1. Simulation and signal recovery experimental results demonstrate the effectiveness and superiority of the proposed PGNN.

Density Control of Interacting Agent Systems

We consider the problem of controlling the group behavior of a large number of dynamic systems that are constantly interacting with each other. These systems are assumed to have identical dynamics (e.g., flocks of birds, UAV swarms) and their group behavior can be modeled by a distribution. Thus, this problem can be viewed as an optimal control problem over the space of distributions. We propose a novel algorithm to compute a feedback control strategy so that, when adopted by the agents, the distribution of them would be transformed from an initial one to a target one over a finite time window. Our method is built on the optimal transport theory but differs significantly from existing work in this area in that our method models the interactions among agents explicitly. From an algorithmic point of view, our algorithm is based on the generalized proximal gradient descent algorithm and has a convergence guarantee with a sublinear rate. We further extend our framework to account for the scenarios where the agents are from multiple species. In the linear quadratic setting, the solution is characterized by a system of coupled Riccati equations which can be solved in closed form. Finally, several numerical examples are presented to illustrate our framework.

Joint Under-Sampling Pattern and Dual-Domain Reconstruction for Accelerating Multi-Contrast MRI.

Multi-Contrast Magnetic Resonance Imaging (MCMRI) utilizes the short-time reference image to facilitate the reconstruction of the long-time target one, providing a new solution for fast MRI. Although various methods have been proposed, they still have certain limitations. 1) existing methods featuring the preset under-sampling patterns give rise to redundancy between multi-contrast images and limit their model performance; 2) most methods focus on the information in the image domain, prior knowledge in the k-space domain has not been fully explored; and 3) most networks are manually designed and lack certain physical interpretability. To address these issues, we propose a joint optimization of the under-sampling pattern and a deep-unfolding dual-domain network for accelerating MCMRI. Firstly, to reduce the redundant information and sample more contrast-specific information, we propose a new framework to learn the optimal under-sampling pattern for MCMRI. Secondly, a dual-domain model is established to reconstruct the target image in both the image domain and the k-space frequency domain. The model in the image domain introduces a spatial transformation to explicitly model the inconsistent and unaligned structures of MCMRI. The model in the k-space learns prior knowledge from the frequency domain, enabling the model to capture more global information from the input images. Thirdly, we employ the proximal gradient algorithm to optimize the proposed model and then unfold the iterative results into a deep-unfolding network, called MC-DuDoN. We evaluate the proposed MC-DuDoN on MCMRI super-resolution and reconstruction tasks. Experimental results give credence to the superiority of the current model. In particular, since our approach explicitly models the inconsistent structures, it shows robustness on spatially misaligned MCMRI. In the reconstruction task, compared with conventional masks, the learned mask restores more realistic images, even under an ultra-high acceleration ratio ( ×30 ). Code is available at https://github.com/lpcccc-cv/MC-DuDoNet.

Estimation of a Simple Structure in a Multidimensional IRT Model Using Structure Regularization.

We develop a method for estimating a simple matrix for a multidimensional item response theory model. Our proposed method allows each test item to correspond to a single latent trait, making the results easier to interpret. It also enables clustering of test items based on their corresponding latent traits. The basic idea of our approach is to use the prenet (product-based elastic net) penalty, as proposed in factor analysis. For optimization, we show that combining stochastic EM algorithms, proximal gradient methods, and coordinate descent methods efficiently yields solutions. Furthermore, our numerical experiments demonstrate its effectiveness, especially in cases where the number of test subjects is small, compared to methods using the existing L1 penalty.

Proximal Gradient Method with Extrapolation and Line Search for a Class of Non-convex and Non-smooth Problems

Proximal Gradient Method with Extrapolation and Line Search for a Class of Non-convex and Non-smooth Problems

Debiased inference for heterogeneous subpopulations in a high-dimensional logistic regression model

Due to the prevalence of complex data, data heterogeneity is often observed in contemporary scientific studies and various applications. Motivated by studies on cancer cell lines, we consider the analysis of heterogeneous subpopulations with binary responses and high-dimensional covariates. In many practical scenarios, it is common to use a single regression model for the entire data set. To do this effectively, it is critical to quantify the heterogeneity of the effect of covariates across subpopulations through appropriate statistical inference. However, the high dimensionality and discrete nature of the data can lead to challenges in inference. Therefore, we propose a novel statistical inference method for a high-dimensional logistic regression model that accounts for heterogeneous subpopulations. Our primary goal is to investigate heterogeneity across subpopulations by testing the equivalence of the effect of a covariate and the significance of the overall effects of a covariate. To achieve overall sparsity of the coefficients and their fusions across subpopulations, we employ a fused group Lasso penalization method. In addition, we develop a statistical inference method that incorporates bias correction of the proposed penalized method. To address computational issues due to the nonlinear log-likelihood and the fused Lasso penalty, we propose a computationally efficient and fast algorithm by adapting the ideas of the proximal gradient method and the alternating direction method of multipliers (ADMM) to our settings. Furthermore, we develop non-asymptotic analyses for the proposed fused group Lasso and prove that the debiased test statistics admit chi-squared approximations even in the presence of high-dimensional variables. In simulations, the proposed test outperforms existing methods. The practical effectiveness of the proposed method is demonstrated by analyzing data from the Cancer Cell Line Encyclopedia (CCLE).

Three-dimensional magnetic resonance neurography aids in detection of brachial plexus nerve root signal and size alterations in patients with amyotrophic lateral sclerosis: a case-control study.

Since previous histopathological studies have shown a distal to proximal gradient of axonal damage in peripheral nerves of patients with amyotrophic lateral sclerosis (ALS), it would be worthwhile to evaluate consequence of such changes on magnetic resonance imaging (MRI). The aim of this study was to assess proximal-distal longitudinal signal and size alterations of brachial plexus nerve roots in ALS patients using 3-dimensional (3D) magnetic resonance neurography (MRN). A total of 21 ALS patients and 19 controls were evaluated. The diameters and signal-to-noise (SNR) ratio values of C5-C8 roots were measured at five points from proximal to distal sites. Student's t-test was performed to compare the differences at each point between two groups. Linear regression was performed for each nerve root, and the differences in linear regression slopes between two groups were analyzed. Receiver operating characteristic (ROC) analysis was performed for the diameter and SNR value ratio of the distal to the proximal points. Interobserver agreement was excellent [intraclass correlation coefficient (ICC): 0.802-0.913]. The diameters and SNR values of C5-C8 roots showed a significant decrease (P<0.05) from proximal to distal except SNR value of C5 root in controls. The slope values of diameters in ALS were -0.01924 for C5, -0.04404 for C6, -0.06228 for C7, and -0.06464 for C8. The slope values of SNR values in ALS were -10.14 for C5, -12.86 for C6, -15.99 for C7, and -19.06 for C8. The slope of nerve diameters and SNR values for ALS patients were more negatively sloped than controls (P<0.05) except SNR values of C5 and C7 roots. The ROC analysis confirmed that the diameter and SNR value ratio could differentiate ALS patients from controls with high accuracy. The cutoff values of diameter ratio were 0.7418 for C5, 0.6952 for C6, 0.6431 for C7, and 0.7147 for C8. The cutoff values of SNR value ratio were 0.5989 for C5, 0.6516 for C6, 0.6065 for C7, and 0.6758 for C8. Proximal-distal longitudinal diameters and SNR values decreased significantly for brachial plexus nerve roots in ALS patients with larger differences in slopes compared to controls. These results reflect pathophysiological changes of ALS and may be helpful in improving the diagnosis of ALS.

Trustworthy regularized huber regression for outlier detection

Machine learning has developed rapidly and involves many fields, such as signal detection and biological research. However, systems may exist outliers caused by the malicious violation of the attacker during the learning process, which makes machine learning instable and unreliable. Therefore, building a trustworthy model to meet robustness has become an urgent research topic. In this paper, we propose the trustworthy regularized Huber regression by Huber loss and decomposable regularizer under the mean shift model, which can identify and eliminate impacts on outliers and heavy-tailed errors. We establish the consistency and convergence rate of the regularized Huber estimators. The asymptotic property of outliers support recovery is explored in term of false and true discovery rates. Then we design an efficient algorithm by combining the alternating minimization and accelerated proximal gradient method to solve the trustworthy regularized Huber regression. Finally, extensive comparisons on simulation and real datasets demonstrate the superior performances.

Enhancing Label Correlations in multi-label classification through global-local label specific feature learning to Fill Missing labels

In multi-label classification, challenges arise from missing labels due to subjective analysis or label ambiguity. This makes it difficult to accurately capture label correlations and enhance classifier performance. Previous research has primarily focused on obtaining label correlations with sparsity assumptions from incomplete label matrices to fill in these missing labels. However, this approach leads to inaccurate label correlations since it relies on incomplete label matrix information. To overcome this issue and improve accuracy, this paper proposes a new method called LCFM, which effectively learns label correlations using a combination of global and local label-specific features to recover missing labels. Firstly, we rely on the sparsity assumption of Lasso regression coefficients to select label-specific features from the training samples. Secondly, we introduce l2-norm regularization to learn global label correlations through global label-specific features. simultaneously, we utilize similar label-specific features to describe local label correlations. Finally, we minimize the difference between predicted results and true labels using the binary cross-entropy loss function and employ the Accelerated Proximal Gradient (APG) algorithm for model optimization. Experimental results comparing LCFM with five benchmark indicators on nine datasets demonstrate that our proposed method outperforms other advanced methods, showing strong competitiveness.