Proximal Minimization Research Articles

Convergence of a class of algorithms with the level-set subdifferential error bound

In this paper, we study the convergence of algorithms for a class of nonconvex and nonsmooth optimization problems via an level-set subdifferential error bound. Many algorithms (cf. forward-backward splitting algorithm and alternating proximal minimization algorithm) satisfy sufficient descent and relative error conditions. We also find that these algorithms satisfy a quadratic error condition. Under these conditions and the level-set subdifferential error bound, we study the global convergence and the convergence rates of these algorithms without the Kurdyka-Łojasiewicz property which is stronger than the level-set subdifferential error bound. Two examples are given to illustrate the broader applicability of our findings across these algorithms.

A modified inertial proximal minimization algorithm for structured nonconvex and nonsmooth problem

We introduce an enhanced inertial proximal minimization algorithm tailored for a category of structured nonconvex and nonsmooth optimization problems. The objective function in question is an aggregation of a smooth function with an associated linear operator, a nonsmooth function dependent on an independent variable, and a mixed function involving two variables. Throughout the iterative procedure, parameters are selected employing a straightforward approach, and weak inertial terms are incorporated into two subproblems within the update sequence. Under a set of lenient conditions, we demonstrate that the sequence engendered by our algorithm is bounded. Furthermore, we establish the global and strong convergence of the algorithmic sequence, contingent upon the assumption that the principal function adheres to the Kurdyka–Łojasiewicz (KL) property. Ultimately, the numerical outcomes corroborate the algorithm’s feasibility and efficacy.

Communication-efficient ADMM using quantization-aware Gaussian process regression

In networks consisting of agents communicating with a central coordinator and working together to solve a global optimization problem in a distributed manner, the agents are often required to solve private proximal minimization subproblems. Such a setting often requires a decomposition method to solve the global distributed problem, resulting in extensive communication overhead. In networks where communication is expensive, it is crucial to reduce the communication overhead of the distributed optimization scheme. Gaussian processes (GPs) are effective at learning the agents' local proximal operators, thereby reducing the communication between the agents and the coordinator. We propose combining this learning method with adaptive uniform quantization for a hybrid approach that can achieve further communication reduction. In our approach, due to data quantization, the GP algorithm is modified to account for the introduced quantization noise statistics. We further improve our approach by introducing an orthogonalization process to the quantizer's input to address the inherent correlation of the input components. We also use dithering to ensure uncorrelation between the quantizer's introduced noise and its input. We propose multiple measures to quantify the trade-off between the communication cost reduction and the optimization solution's accuracy/optimality. Under such metrics, our proposed algorithms can achieve significant communication reduction for distributed optimization with acceptable accuracy, even at low quantization resolutions. This result is demonstrated by simulations of a distributed sharing problem with quadratic cost functions for the agents.

Principled analyses and design of first-order methods with inexact proximal operators

Proximal operations are among the most common primitives appearing in both practical and theoretical (or high-level) optimization methods. This basic operation typically consists in solving an intermediary (hopefully simpler) optimization problem. In this work, we survey notions of inaccuracies that can be used when solving those intermediary optimization problems. Then, we show that worst-case guarantees for algorithms relying on such inexact proximal operations can be systematically obtained through a generic procedure based on semidefinite programming. This methodology is primarily based on the approach introduced by Drori and Teboulle (Math Program 145(1–2):451–482, 2014) and on convex interpolation results, and allows producing non-improvable worst-case analyses. In other words, for a given algorithm, the methodology generates both worst-case certificates (i.e., proofs) and problem instances on which they are achieved. Relying on this methodology, we study numerical worst-case performances of a few basic methods relying on inexact proximal operations including accelerated variants, and design a variant with optimized worst-case behavior. We further illustrate how to extend the approach to support strongly convex objectives by studying a simple relatively inexact proximal minimization method.

Seismic data recovery using deep targeted denoising priors in an alternating optimization framework

The problem of recovering the complete seismic data from undersampled field-observed data is a long-term challenge. Many recent efforts to address this problem develop model-based recovery methods. However, current model-based methods cannot accurately capture inherent priors of seismic data to obtain optimal recovery results. For this issue, we have developed a novel model-based seismic data recovery method, which integrates a deep learning (DL)-based targeted denoiser into the seismic data recovery model and leverages the deep targeted denoising priors learned by this denoiser to implement high accuracy data recovery. Specifically, we first use operator-splitting technology to decouple the data-consistency term and prior constraint term in a seismic data recovery model, resulting in an alternating optimization scheme composed of a least-squares inversion subtask and a proximal minimization subtask. The DL-based targeted denoiser, a dual-channel deep denoising network (DcDDNet), is then plugged into the alternating optimization scheme as a modular component, according to the mathematical equivalence between the proximal operator and the targeted denoiser. This modular component replaces the formalized proximal minimization subtask and plays the role of an implicit prior in the recovery optimization problem. Finally, the least-squares inversion and deep denoising subtasks in the alternating optimization scheme are iteratively executed to achieve seismic data recovery. Due to the integration of the powerful DcDDNet, this method not only enjoys high flexibility and impressive generalization capability but also has high computational efficiency and excellent performance. Synthetic and field data with regular and irregular undersampling are used to evaluate the recovery capability of this method. The results demonstrate that, compared with the f- x adaptive interpolation and curvelet-based recovery methods, our method has advantages in visual and quality metrics. In addition, this method also is more practical in seismic applications because it works in the time-space domain and can handle spatial aliasing and spectral leakage.

Global convergence of model function based Bregman proximal minimization algorithms

Lipschitz continuity of the gradient mapping of a continuously differentiable function plays a crucial role in designing various optimization algorithms. However, many functions arising in practical applications such as low rank matrix factorization or deep neural network problems do not have a Lipschitz continuous gradient. This led to the development of a generalized notion known as the L-smad property, which is based on generalized proximity measures called Bregman distances. However, the L-smad property cannot handle nonsmooth functions, for example, simple nonsmooth functions like vert x^4-1 vert and also many practical composite problems are out of scope. We fix this issue by proposing the MAP property, which generalizes the L-smad property and is also valid for a large class of structured nonconvex nonsmooth composite problems. Based on the proposed MAP property, we propose a globally convergent algorithm called Model BPG, that unifies several existing algorithms. The convergence analysis is based on a new Lyapunov function. We also numerically illustrate the superior performance of Model BPG on standard phase retrieval problems and Poisson linear inverse problems, when compared to a state of the art optimization method that is valid for generic nonconvex nonsmooth optimization problems.

Distributed decision-coupled constrained optimization via Proximal-Tracking

In this paper we deal with decision-coupled problems involving multiple agents over a network. Each agent has its own local objective function and local constraints, and all agents aim at finding the value of a common decision vector that minimizes the sum of all agents’ cost functions and satisfies all local constraints. To this purpose, we introduce a Proximal-Tracking distributed optimization algorithm that integrates dynamic average consensus within the proximal minimization method. Convergence to an optimal consensus solution is guaranteed for any value of a constant penalty parameter, under a convexity assumption only, without requiring differentiability, Lipschitz continuity, or smoothness of the local objective functions. Numerical simulations show the effectiveness of the proposed scheme.

Proximal Regularization for the Saddle Point Gradient Dynamics

This article concerns the solution of a convex optimization problem through the saddle point gradient dynamics. Instead of using the standard Lagrangian as is classical in this method, we consider a regularized Lagrangian obtained through a proximal minimization step. We show that, without assumptions of smoothness or strict convexity in the original problem, the regularized Lagrangian is smooth and leads to globally convergent saddle point dynamics. The method is demonstrated through an application to resource allocation in cloud computing.

A Distributed Dual Proximal Minimization Algorithm for Constraint-Coupled Optimization Problems

We address constraint-coupled optimization for a system composed of multiple cooperative agents communicating over a time-varying network. We propose a distributed proximal minimization algorithm that is guaranteed to converge to an optimal solution of the optimization problem, under suitable convexity and connectivity assumptions. The performance of the introduced algorithm is shown on a numerical example of a charging scheduling problem for a fleet of plug-in electric vehicles.

Ptychographic phase retrieval by proximal algorithms

We derive a set of ptychography phase-retrieval iterative engines based on proximal algorithms originally developed in convex optimization theory, and discuss their connections with existing ones. The use of proximal operator creates a simple frame work that allows us to incorporate the effect of noise from a maximum-likelihood (ML) principle. We focus on three particular algorithms, namely proximal minimization, alternating direction method of multiplier and accelerated proximal gradient (APG). We benchmark their performance with numerical simulations, and discuss their optimal conditions for convergence and accuracy. An experimental dataset is used to demonstrate their effectiveness as well, in which case an array of cubic Au nanoparticles with a size of 50 nm is imaged. We show that with the presence of Poisson noise, a dataset with photon counts up to 104 at one detector pixel already requires ML-based methods to achieve a stable convergence. Among the three algorithms derived in this work, APG method is reported first time for its application in ptychographic reconstruction and shows superior performance in terms of both accuracy and convergence rate with a noisy dataset.

Robust sequential subspace clustering via ℓ1-norm temporal graph

Subspace clustering (SC) has been widely applied to segment data drawn from multiple subspaces. However, for sequential data, a main challenge in subspace clustering is to exploit temporal information. In this paper, we propose a novel robust sequential subspace clustering approach with a ℓ1-norm temporal graph. The ℓ1-norm temporal graph is designed to encode the temporal information underlying in sequential data. By using the ℓ2 norm, it can enforce well temporal similarity of neighboring frames with a sample-dependent weight, and mitigate the effect of noises and outliers on subspace clustering because large errors mixed in the real data can be suppressed. Under assumption of data self-expression, our clustering model is put forward by further integrating the classical Sparse Subspace Clustering and the ℓ1-norm Temporal Graph (SSC-L1TG). To solve the proposed model, we introduce a new efficient proximity algorithm. At each iteration, the sub-problem is solved by proximal minimization with closed-form solution. In contrast to the alternating direction method of multipliers (ADMM) employed in most existing clustering approaches without convergence guarantee, the proposed SSC-L1TG is guaranteed to converge to the desired optimal solution. Experimental results on both synthetic and real data demonstrate the efficacy of our method and its superior performance over the state-of-the-art methods.

Spatially regularized estimation of the tissue homogeneity model parameters in DCE-MRI using proximal minimization.

The Tofts and the extended Tofts models are the pharmacokinetic models commonly used in dynamic contrast-enhanced MRI (DCE-MRI) perfusion analysis, although they do not provide two important biological markers, namely, the plasma flow and the permeability-surface area product. Estimates of such markers are possible using advanced pharmacokinetic models describing the vascular distribution phase, such as the tissue homogeneity model. However, the disadvantage of the advanced models lies in biased and uncertain estimates, especially when the estimates are computed voxelwise. The goal of this work is to improve the reliability of the estimates by including information from neighboring voxels. Information from the neighboring voxels is incorporated in the estimation process through spatial regularization in the form of total variation. The spatial regularization is applied on five maps of perfusion parameters estimated using the tissue homogeneity model. Since the total variation is not differentiable, two proximal techniques of convex optimization are used to solve the problem numerically. The proposed algorithm helps to reduce noise in the estimated perfusion-parameter maps together with improving accuracy of the estimates. These conclusions are proved using a numerical phantom. In addition, experiments on real data show improved spatial consistency and readability of perfusion maps without considerable lowering of the quality of fit. The reliability of the DCE-MRI perfusion analysis using the tissue homogeneity model can be improved by employing spatial regularization. The proposed utilization of modern optimization techniques implies only slightly higher computational costs compared to the standard approach without spatial regularization.

Distributed Proximal Minimization Algorithm for Constrained Convex Optimization over Strongly Connected Networks

Distributed Proximal Minimization Algorithm for Constrained Convex Optimization over Strongly Connected Networks

求解一类非凸非光滑问题的惯性邻近交替极小化算法

求解一类非凸非光滑问题的惯性邻近交替极小化算法

A Proximal Minimization Algorithm for Structured Nonconvex and Nonsmooth Problems

We propose a proximal algorithm for minimizing objective functions consisting of three summands: the composition of a nonsmooth function with a linear operator, another nonsmooth function (with eac...

Optimal Resource Allocation for Green and Clustered Video Sensor Networks

Wireless video sensor networks (WVSNs) are opening the door for many applications, such as industrial surveillance, environmental tracking, border security, and infrastructure health monitoring. In WVSN, energy conservation is very essential because: 1) sensors are usually battery-operated and 2) each sensor node needs to compress the video prior to transmission, which consumes more power than conventional wireless sensor networks. In this paper, we study the problem of minimizing the total power consumption in a cluster-based WVSN, leveraging cross-layer design to optimize the encoding power, the transmission power, and the source rate at each sensor node. To realize this problem, we devise a resource optimization framework, which takes into account the video signal distortion due to compression, in addition to packet loss in the wireless channel while trying to allocate network resources among multiple video sensors. Leveraging duality theory, we design a proximal minimization algorithm that is capable of achieving the optimal allocation of network resources, source rates, and encoding and communication powers, while providing application-level quality of service represented by video distortion. The algorithm is extended for sensor nodes with hybrid energy sources, leveraging energy harvesting to minimize the aggregate power, while addressing the tradeoff between renewable versus grid energy sources.

Semantic scene completion with dense CRF from a single depth image

Abstract Scene understanding is a significant research topic in computer vision, especially for robots to understand their environment intelligently. Semantic scene segmentation can help robots to identify the objects that are present in their surroundings, while semantic scene completion can enhance the ability of the robot to infer the object shape, which is pivotal for several high-level tasks. With dense Conditional Random Field (CRF), one key issue is how to construct the long-range interactions between nodes with Gaussian pairwise potentials. Another issue is what effective and efficient inference algorithms can be adapted to resolve the optimization. In this paper, we focus on semantic scene segmentation and completion optimization technology simultaneously using dense CRF based on a single depth image only. Firstly, we convert the single depth image into different down-sampled Truncated Signed Distance Function (TSDF) or flipped TSDF voxel formats, and formulate the pairwise potentials terms with such a representation. Secondly, we use the output results of an end-to-end 3D convolutional neural network named SSCNet to obtain the unary potentials. Finally, we pursue the efficiency of different CRF inference algorithms (the mean-field inference, the negative semi-definite specific difference of convex relaxation, the proximal minimization of linear programming and its variants, etc.). The proposed dense CRF and inference algorithms are evaluated on three different datasets (SUNCG, NYU, and NYUCAD). Experimental results demonstrate that the voxel-level intersection over union (IoU) of predicted voxel’s semantic and completion can reach to state-of-the-art. Specifically, for voxel semantic segmentation, the highest IoU improvements are 2.6%, 1.3%, 3.1%, and for scene completion, the highest IoU improvements are 2.5%, 3.7%, 5.4%, respectively for SUNCG, NYU, and NYUCAD datasets.

Matrix completion with capped nuclear norm via majorized proximal minimization

Abstract We investigate the problem of matrix completion with capped nuclear norm regularization. Different from most existing regularizations that minimize all the singular values simultaneously, capped nuclear norm only penalties the singular values smaller than certain threshold. Due to its non-smoothness and non-convexity, by formulating with Majorization Minimization (MM) approach, we develop a fast Majorized Proximal Minimization Impute (MPM-Impute) algorithm. At each iteration, the sub-problem is relaxed to a surrogate (upper bound) function and solved via proximal minimization with closed form solution. Though it requires singular value decompositions (SVD) at each iteration, by incorporating with the randomized algorithm, we propose the Randomized Truncated Singular Value Thresholding (RTSVT) operator to lower the computational cost. In addition, in contrast with most MM approaches, our algorithm is guaranteed to converge to the stationary points. Experimental results on synthetic data, image inpainting show that the completion results exceed or achieve comparable performance than state-of-the-art, yet several times faster.

Distributed Constrained Optimization and Consensus in Uncertain Networks via Proximal Minimization

We provide a unifying framework for distributed convex optimization over time-varying networks, in the presence of constraints and uncertainty, features that are typically treated separately in the literature. We adopt a proximal minimization perspective and show that this set-up allows us to bypass the difficulties of existing algorithms while simplifying the underlying mathematical analysis. We develop an iterative algorithm and show the convergence of the resulting scheme to some optimizer of the centralized problem. To deal with the case where the agents’ constraint sets are affected by a possibly common uncertainty vector, we follow a scenario-based methodology and offer probabilistic guarantees regarding the feasibility properties of the resulting solution. To this end, we provide a distributed implementation of the scenario approach, allowing agents to use a different set of uncertainty scenarios in their local optimization programs. The efficacy of our algorithm is demonstrated by means of a numerical example related to a regression problem subject to regularization.

Local Convergence of the Heavy-Ball Method and iPiano for Non-convex Optimization

A local convergence result for an abstract descent method is proved. The sequence of iterates is attracted by a local (or global) minimum, stays in its neighborhood, and converges within this neighborhood. This result allows algorithms to exploit local properties of the objective function. In particular, the abstract theory in this paper applies to the inertial forward–backward splitting method: iPiano—a generalization of the Heavy-ball method. Moreover, it reveals an equivalence between iPiano and inertial averaged/alternating proximal minimization and projection methods. Key for this equivalence is the attraction to a local minimum within a neighborhood and the fact that, for a prox-regular function, the gradient of the Moreau envelope is locally Lipschitz continuous and expressible in terms of the proximal mapping. In a numerical feasibility problem, the inertial alternating projection method significantly outperforms its non-inertial variants.