Nonsmooth Nonconvex Optimization Problems Research Articles

Newton acceleration on manifolds identified by proximal gradient methods

Proximal methods are known to identify the underlying substructure of nonsmooth optimization problems. Even more, in many interesting situations, the output of a proximity operator comes with its structure at no additional cost, and convergence is improved once it matches the structure of a minimizer. However, it is impossible in general to know whether the current structure is final or not; such highly valuable information has to be exploited adaptively. To do so, we place ourselves in the case where a proximal gradient method can identify manifolds of differentiability of the nonsmooth objective. Leveraging this manifold identification, we show that Riemannian Newton-like methods can be intertwined with the proximal gradient steps to drastically boost the convergence. We prove the superlinear convergence of the algorithm when solving some nondegenerated nonsmooth nonconvex optimization problems. We provide numerical illustrations on optimization problems regularized by \(\ell _1\)-norm or trace-norm.

Proximal stochastic recursive momentum algorithm for nonsmooth nonconvex optimization problems

In this paper, we mainly consider a class of nonconvex nonsmooth optimization problems, whose objective function is the sum of a smooth function with a Lipschitz continuous gradient and a convex nonsmooth function. We first propose a proximal stochastic recursive momentum algorithm(ProxSTORM) with mini-batch for solving the optimization problems and consider its convergence behaviour. Then, based on the Polyak–Łojasiewicz inequality, we establish the global linear convergence rate of ProxSTORM. Finally, some numerical experiments have been conducted to illustrate the efficiency of our method.

Variance reduction on general adaptive stochastic mirror descent

In this work, we investigate the idea of variance reduction by studying its properties with general adaptive mirror descent algorithms in nonsmooth nonconvex finite-sum optimization problems. We propose a simple yet generalized framework for variance reduced adaptive mirror descent algorithms named SVRAMD and provide its convergence analysis in both the nonsmooth nonconvex problem and the P-L conditioned problem. We prove that variance reduction reduces the SFO complexity of adaptive mirror descent algorithms and thus accelerates their convergence. In particular, our general theory implies that variance reduction can be applied to algorithms using time-varying step sizes and self-adaptive algorithms such as AdaGrad and RMSProp. Moreover, the convergence rates of SVRAMD recover the best existing rates of non-adaptive variance reduced mirror descent algorithms without complicated algorithmic components. Extensive experiments in deep learning validate our theoretical findings.

A continuous-time neurodynamic algorithm for distributed nonconvex nonsmooth optimization problems with affine equality and nonsmooth convex inequality constraints

In this paper, a distributed nonsmooth nonconvex optimization (DNNO) problem with affine inequality and nonsmooth convex inequality constraints is studied. A continuous-time distributed neurodynamic algorithm is proposed to solve this problem. Under the assumed conditions, for any initial state, the solution of distributed neurodynamic algorithm is bounded and globally exists, and will converge to the critical point set of distributed problems in a finite time. Compared with other DNNO algorithms, distributed neurodynamic algorithm has a lower dimension and does not need to satisfy the assumption that the feasible region is bounded. Finally, a series of numerical examples are given to verify the effectiveness of the proposed algorithm.

Inertial alternating direction method of multipliers for non-convex non-smooth optimization

In this paper, we propose an algorithmic framework, dubbed inertial alternating direction methods of multipliers (iADMM), for solving a class of nonconvex nonsmooth multiblock composite optimization problems with linear constraints. Our framework employs the general minimization-majorization (MM) principle to update each block of variables so as to not only unify the convergence analysis of previous ADMM that use specific surrogate functions in the MM step, but also lead to new efficient ADMM schemes. To the best of our knowledge, in the nonconvex nonsmooth setting, ADMM used in combination with the MM principle to update each block of variables, and ADMM combined with \emph{inertial terms for the primal variables} have not been studied in the literature. Under standard assumptions, we prove the subsequential convergence and global convergence for the generated sequence of iterates. We illustrate the effectiveness of iADMM on a class of nonconvex low-rank representation problems.

Optimality conditions for invex nonsmooth optimization problems with fuzzy objective functions

In this paper, the definitions of Clarke generalized directional alpha -derivative and Clarke generalized gradient are introduced for a locally Lipschitz fuzzy function. Further, a nonconvex nonsmooth optimization problem with fuzzy objective function and both inequality and equality constraints is considered. The Karush-Kuhn-Tucker optimality conditions are established for such a nonsmooth extremum problem. For proving these conditions, the approach is used in which, for the considered nonsmooth fuzzy optimization problem, its associated bi-objective optimization problem is constructed. The bi-objective optimization problem is solved by its associated scalarized problem constructed in the weighting method. Then, under invexity hypotheses, (weakly) nondominated solutions in the considered nonsmooth fuzzy minimization problem are characterized through Pareto solutions in its associated bi-objective optimization problem and Karush-Kuhn-Tucker points of the weighting problem.

Block Bregman Majorization Minimization with Extrapolation

In this paper, we consider a class of nonsmooth nonconvex optimization problems whose objective is the sum of a block relative smooth function and a proper and lower semicontinuous block separable function. Although the analysis of block proximal gradient (BPG) methods for the class of block $L$-smooth functions have been successfully extended to Bregman BPG methods that deal with the class of block relative smooth functions, accelerated Bregman BPG methods are scarce and challenging to design. Taking our inspiration from Nesterov-type acceleration and the majorization-minimization scheme, we propose a block alternating Bregman Majorization-Minimization framework with Extrapolation (BMME). We prove subsequential convergence of BMME to a first-order stationary point under mild assumptions, and study its global convergence under stronger conditions. We illustrate the effectiveness of BMME on the penalized orthogonal nonnegative matrix factorization problem.

A new constraint qualification for optimality of nonconvex nonsmooth optimization problems

In this paper, we study the nonconvex nonsmooth optimization problem (P) of minimizing a tangentially convex function with inequality constraints where the constraint functions are tangentially convex. This is done by using the cone of tangential subdifferentials together with a new constraint qualification. Indeed, we present a new constraint qualification to guarantee that Karush-Kuhn-Tucker conditions are necessary and sufficient for optimality of the problem (P). Moreover, various nonsmooth (generalized) constraint qualifications that are a modification of the well known constraint qualifications are investigated. Several illustrative examples are presented to clarify the connection between nonsmooth constraint qualifications and new constraint qualification.

Global convergence of model function based Bregman proximal minimization algorithms

Lipschitz continuity of the gradient mapping of a continuously differentiable function plays a crucial role in designing various optimization algorithms. However, many functions arising in practical applications such as low rank matrix factorization or deep neural network problems do not have a Lipschitz continuous gradient. This led to the development of a generalized notion known as the L-smad property, which is based on generalized proximity measures called Bregman distances. However, the L-smad property cannot handle nonsmooth functions, for example, simple nonsmooth functions like vert x^4-1 vert and also many practical composite problems are out of scope. We fix this issue by proposing the MAP property, which generalizes the L-smad property and is also valid for a large class of structured nonconvex nonsmooth composite problems. Based on the proposed MAP property, we propose a globally convergent algorithm called Model BPG, that unifies several existing algorithms. The convergence analysis is based on a new Lyapunov function. We also numerically illustrate the superior performance of Model BPG on standard phase retrieval problems and Poisson linear inverse problems, when compared to a state of the art optimization method that is valid for generic nonconvex nonsmooth optimization problems.

Multi-block Nonconvex Nonsmooth Proximal ADMM: Convergence and Rates Under Kurdyka–Łojasiewicz Property

We study the convergence and convergence rates of a multi-block proximal alternating direction method of multipliers (PADMM) for solving linearly constrained separable nonconvex nonsmooth optimization problems. This algorithm is an important variant of the alternating direction method of multipliers (ADMM) which includes a proximal term in each subproblem, to cancel out complicated terms in applications where subproblems are not easy to solve or do not admit a simple closed form solution. We consider an over-relaxation step size in the dual update and provide a detailed proof of the convergence for any step size \(\beta \in (0,2)\). We prove the convergence of the sequence generated by the PADMM by showing that it has a finite length and it is Cauchy. Under the powerful Kurdyka–Łojasiewicz (KŁ) property, we establish the convergence rates for the values and the iterates, and we show that various values of KŁ-exponent associated with the objective function can raise PADMM with three different convergence rates. More precisely, we show that if the (KŁ) exponent \(\theta =0\), the sequence generated by PADMM converges in a finite numbers of iterations. If \(\theta \in (0,1/2]\), then the sequential rate of convergence is \(cQ^{k}\) where \(c>0\), \(Q\in (0,1)\), and \(k\in {\mathbb {N}}\) is the iteration number. If \(\theta \in (1/2,1]\), then \({\mathcal {O}}(1/k^{r})\) rate where \(r=(1-\theta )/(2\theta -1)\) is achieved.

An inertial Douglas–Rachford splitting algorithm for nonconvex and nonsmooth problems

SummaryIn the fields of wireless communication and data processing, there are varieties of mathematical optimization problems, especially nonconvex and nonsmooth problems. For these problems, one of the biggest difficulties is how to improve the speed of solution. To this end, here we mainly focused on a minimization optimization model that is nonconvex and nonsmooth. Firstly, an inertial Douglas–Rachford splitting (IDRS) algorithm was established, which incorporate the inertial technology into the framework of the Douglas–Rachford splitting algorithm. Then, we illustrated the iteration sequence generated by the proposed IDRS algorithm converges to a stationary point of the nonconvex nonsmooth optimization problem with the aid of the Kurdyka–Łojasiewicz property. Finally, a series of numerical experiments were carried out to prove the effectiveness of our proposed algorithm from the perspective of signal recovery. The results are implicit that the proposed IDRS algorithm outperforms another algorithm.

A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization

In this paper, a novel stochastic extra-step quasi-Newton method is developed to solve a class of nonsmooth nonconvex composite optimization problems. We assume that the gradient of the smooth part of the objective function can only be approximated by stochastic oracles. The proposed method combines general stochastic higher order steps derived from an underlying proximal type fixed-point equation with additional stochastic proximal gradient steps to guarantee convergence. Based on suitable bounds on the step sizes, we establish global convergence to stationary points in expectation and an extension of the approach using variance reduction techniques is discussed. Motivated by large-scale and big data applications, we investigate a stochastic coordinate-type quasi-Newton scheme that allows to generate cheap and tractable stochastic higher order directions. Finally, numerical results on large-scale logistic regression and deep learning problems show that our proposed algorithm compares favorably with other state-of-the-art methods.

Single image restoration through [formula omitted]-relaxed truncated [formula omitted] analysis-based sparse optimization in tight frames

Image restoration problems, i.e., recovery of an original high-quality image from the degraded observation, arise in various science and engineer areas. Over the past decades, the framelet-based methods are particularly investigated and adopted, owing to the excellent ability of sparse approximating the piecewise-smooth functions such as natural images. In this paper, we propose a novel tight frame-based ℓ2-relaxed truncated ℓ0 analysis-sparsity model that simultaneously exploiting the sparsity and support priors. The resulting nonconvex nonsmooth optimization problem is addressed by using the proposed proximal alternating adaptive hard thresholding (PAAHT) method. We also proved that the sequence generated by the proposed algorithm sublinearly converges. Numerical experiments on several typical image restoration problems demonstrate that the proposed method is more effective than the standard sparsity-inducing algorithms and outperforms several state-of-the-art methods in both objective and perceptual quality.

Learning Both Dynamic-Shared and Dynamic-Specific Patterns for Chaotic Time-Series Prediction.

In the real world, multivariate time series from the dynamical system are correlated with deterministic relationships. Analyzing them dividedly instead of utilizing the shared-pattern of the dynamical system is time consuming and cumbersome. Multitask learning (MTL) is an effective inductive bias method to utilize latent shared features and discover the structural relationships from related tasks. Base on this concept, we propose a novel MTL model for multivariate chaotic time-series prediction, which could learn both dynamic-shared and dynamic-specific patterns. We implement the dynamic analysis of multiple time series through a special network structure design. The model could disentangle the complex relationships among multivariate chaotic time series and derive the common evolutionary trend of the multivariate chaotic dynamical system by inductive bias. We also develop an efficient Crank-Nicolson-like curvilinear update algorithm based on the alternating direction method of multipliers (ADMM) for the nonconvex nonsmooth Stiefel optimization problem. Simulation results and analysis demonstrate the effectiveness on dynamic-shared pattern discovery and prediction performance.

Isotonicity of Proximity Operators in General Quasi-Lattices and Optimization Problems

Motivated by the recent works on proximity operators and isotone projection cones, in this paper, we discuss the isotonicity of the proximity operator in quasi-lattices, endowed with general cones. First, we show that Hilbert spaces, endowed with general cones, are quasi-lattices, in which the isotonicity of the proximity operator with respect to one order and two mutually dual orders is then, respectively, studied. Various sufficient conditions and examples are introduced. Moreover, we compare the proximity operator with the identity operator with respect to the orders. As applications, we study the solvability and approximation results for the nonconvex nonsmooth optimization problem by the order approaches. By establishing the increasing sequences, we, respectively, discuss the region of the solutions and the convergence rate, which vary with combinations of the mappings, and hence, one can choose the proper combination of the mappings under specific conditions. Compared to other approaches, the optimal solutions are obtained and inequality conditions hold only for comparable elements with respect to the orders.

Understanding Notions of Stationarity in Nonsmooth Optimization: A Guided Tour of Various Constructions of Subdifferential for Nonsmooth Functions

Many contemporary applications in signal processing and machine learning give rise to structured nonconvex nonsmooth optimization problems that can often be tackled by simple iterative methods quite effectively. One of the keys to understanding such a phenomenon-and, in fact, a very difficult conundrum even for experts-lies in the study of points of the problem in question. Unlike smooth optimization, for which the definition of a stationary point is rather standard, there are myriad definitions of stationarity in nonsmooth optimization. In this article, we provide an introduction to different stationarity concepts for several important classes of nonconvex nonsmooth functions, discuss the geometric interpretations of these concepts, and further clarify their relationships. We then demonstrate the relevance of these constructions in some representative applications and indicate how they could affect the performance of iterative methods for addressing these applications.

Robustness in Nonsmooth Nonconvex Optimization Problems

In this paper, the robust approach (the worst case approach) for nonsmooth nonconvex optimization problems with uncertainty data is studied. First various robust constraint qualifications are introduced based on the concept of tangential subdifferential. Further, robust necessary and sufficient optimality conditions are derived in the absence of the convexity of the uncertain sets and the concavity of the related functions with respect to the uncertain parameters. Finally, the results are applied to obtain the necessary and sufficient optimality conditions for robust weakly efficient solutions in multiobjective programming problems. In addition, several examples are provided to illustrate the advantages of the obtained outcomes.

Machine Intelligence at the Edge With Learning Centric Power Allocation

While machine-type communication (MTC) devices generate considerable amounts of data, they often cannot process the data due to limited energy and computational power. To empower MTC with intelligence, edge machine learning has been proposed. However, power allocation in this paradigm requires maximizing the learning performance instead of the communication throughput, for which the celebrated water-filling and max-min fairness algorithms become inefficient. To this end, this paper proposes learning centric power allocation (LCPA), which provides a new perspective on radio resource allocation in learning driven scenarios. By employing 1) an empirical classification error model that is supported by learning theory and 2) an uncertainty sampling method that accounts for different distributions at users, LCPA is formulated as a nonconvex nonsmooth optimization problem, and is solved using a majorization minimization (MM) framework. To get deeper insights into LCPA, asymptotic analysis shows that the transmit powers are inversely proportional to the channel gains, and scale exponentially with the learning parameters. This is in contrast to traditional power allocations where quality of wireless channels is the only consideration. Last but not least, a large-scale optimization algorithm termed mirror-prox LCPA is further proposed to enable LCPA in large-scale settings. Extensive numerical results demonstrate that the proposed LCPA algorithms outperform traditional power allocation algorithms, and the large-scale optimization algorithm reduces the computation time by orders of magnitude compared with MM-based LCPA but still achieves competing learning performance.

Primal–Dual Proximal Splitting and Generalized Conjugation in Non-smooth Non-convex Optimization

We demonstrate that difficult non-convex non-smooth optimization problems, such as Nash equilibrium problems and anisotropic as well as isotropic Potts segmentation model, can be written in terms of generalized conjugates of convex functionals. These, in turn, can be formulated as saddle-point problems involving convex non-smooth functionals and a general smooth but non-bilinear coupling term. We then show through detailed convergence analysis that a conceptually straightforward extension of the primal--dual proximal splitting method of Chambolle and Pock is applicable to the solution of such problems. Under sufficient local strong convexity assumptions of the functionals -- but still with a non-bilinear coupling term -- we even demonstrate local linear convergence of the method. We illustrate these theoretical results numerically on the aforementioned example problems.

Smoothing quadratic regularization method for hemivariational inequalities

Hemivariational inequalities arise in nonsmooth mechanics of solid involving nonmonotone and multi-valued mechanical relations. Typically, after the finite-element discretization, they lead to constrained nonsmooth nonconvex optimization problems with objective functions being the sum of quadratic functions and nonsmooth terms. In this paper, smoothing approximations are employed to solve the constrained nonsmooth nonconvex optimization problems. After properties of the smoothing functions are analysed, a smoothing quadratic regularization algorithm is presented and studied. The proposed algorithm can be implemented efficiently since the closed form solution is available at each iteration. Convergence of the algorithm is shown, and the worst-case complexity is investigated for reaching an ε-Clarke stationary point. A numerical example is reported to show the performance of the proposed method.