Karush Kuhn Tucker Point Research Articles

Distributed sparsity constrained optimization over the Stiefel manifold

Distributed optimization aims to effectively complete specified tasks through cooperation among multi-agent systems, which has achieved great success in large-scale optimization problems. However, it remains a challenging task to develop an effective distributed algorithm with theoretical guarantees, especially when dealing with nonconvex constraints. More importantly, high-dimensional data often exhibits inherent structures such as sparsity, which if exploited accurately, can significantly enhance the capture of its intrinsic characteristics. In this paper, we introduce a novel distributed sparsity constrained optimization framework over the Stiefel manifold, abbreviated as DREAM. DREAM innovatively integrates the ℓ2,0-norm constraint and Stiefel manifold constraint within a distributed optimization setting, which has not been investigated in existing literature. Unlike the existing distributed methods, the proposed DREAM not only can extract the similarity information among samples, but also more flexibly determine the number of features to be extracted. Then, we develop an efficient Newton augmented Lagrangian-based algorithm. In theory, we delve into the relationship between the minimizer, the Karush–Kuhn–Tucker point, and the stationary point, and rigorously demonstrate that the sequence generated by our algorithm converges to a stationary point. Extensive numerical experiments verify its superiority over state-of-the-art distributed methods.

The nearest graph Laplacian in Frobenius norm

AbstractWe address the problem of finding the nearest graph Laplacian to a given matrix, with the distance measured using the Frobenius norm. Specifically, for the directed graph Laplacian, we propose two novel algorithms by reformulating the problem as convex quadratic optimization problems with a special structure: one based on the active set method and the other on direct computation of Karush–Kuhn–Tucker points. The proposed algorithms can be applied to system identification and model reduction problems involving Laplacian dynamics. We demonstrate that these algorithms possess lower time complexities and the finite termination property, unlike the interior point method and V‐FISTA, the latter of which is an accelerated projected gradient method. Our numerical experiments confirm the effectiveness of the proposed algorithms.

A collaborative neurodynamic approach with two-timescale projection neural networks designed via majorization-minimization for global optimization and distributed global optimization

In this paper, two two-timescale projection neural networks are proposed based on the majorization-minimization principle for nonconvex optimization and distributed nonconvex optimization. They are proved to be globally convergent to Karush–Kuhn–Tucker points. A collaborative neurodynamic approach leverages multiple two-timescale projection neural networks repeatedly re-initialized using a meta-heuristic rule for global optimization and distributed global optimization. Two numerical examples are elaborated to demonstrate the efficacy of the proposed approaches.

Approximate weak optimality conditions in multiobjective generalized Nash equilibrium problems

A multiobjective generalized Nash equilibrium problem (MGNEP) is a Nash equilibrium problem with constraints that include multiobjective games. We focus in this paper on examining the approximate Karush–Kuhn–Tucker (KKT) conditions for MGNEPs and their impact on the global convergence of algorithms. To begin, we define standard approximate weak KKT (standard-AWKKT) conditions for MGNEPs. We demonstrate that every weak efficient solution meets the standard-AWKKT condition. As these optimality conditions are strong, we propose a new set of approximate weak KKT conditions, specifically tailored for MGNEPs, and show their convergence towards weak KKT points provided a cone-continuity property is satisfied. It is important to note that while these newly introduced approximate weak KKT conditions do not serve as optimality conditions for general MGNEPs, we illustrate that they hold true for the special case of a weak variational equilibrium point in a jointly convex MGNEP. Finally, we propose an enhanced Lagrangian-type algorithm for estimating an approximate weak KKT of an MGNEP and demonstrate its global convergence.

Extended convergence analysis of the Scholtes-type regularization for cardinality-constrained optimization problems

AbstractWe extend the convergence analysis of the Scholtes-type regularization method for cardinality-constrained optimization problems. Its behavior is clarified in the vicinity of saddle points, and not just of minimizers as it has been done in the literature before. This becomes possible by using as an intermediate step the recently introduced regularized continuous reformulation of a cardinality-constrained optimization problem. We show that the Scholtes-type regularization method is well-defined locally around a nondegenerate T-stationary point of this regularized continuous reformulation. Moreover, the nondegenerate Karush–Kuhn–Tucker points of the corresponding Scholtes-type regularization converge to a T-stationary point having the same index, i.e. its topological type persists. As consequence, we conclude that the global structure of the Scholtes-type regularization essentially coincides with that of CCOP.

On the weakest constraint qualification for sharp local minimizers

The sharp local minimality of feasible points of nonlinear optimization problems is known to possess a characterization by a strengthened version of the Karush–Kuhn–Tucker conditions, as long as the Mangasarian–Fromovitz constraint qualification holds. This strengthened condition is not easy to check algorithmically since it involves the topological interior of some set. In this article, we derive an algorithmically tractable version of this condition, called strong Karush–Kuhn–Tucker condition. We show that the Guignard constraint qualification is the weakest condition under which a feasible point is a strong Karush–Kuhn–Tucker point for every continuously differentiable objective function possessing the point as a sharp local minimizer. As an application, our results yield an algebraic characterization of strict local minimizers of linear programs with cardinality constraints.

Robust time-of-arrival localization via ADMM

This article considers the problem of source localization (SL) using possibly unreliable time-of-arrival (TOA) based range measurements. Adopting the strategy of statistical robustification, we formulate TOA SL as minimization of a versatile loss that possesses resistance against the occurrence of outliers. We then present an alternating direction method of multipliers (ADMM) to tackle the nonconvex optimization problem in a computationally attractive iterative manner. Moreover, we prove that the solution obtained by the proposed ADMM will correspond to a Karush–Kuhn–Tucker point of the formulation when the algorithm converges, and discuss reasonable assumptions about the robust loss function under which the approach can be theoretically guaranteed to be convergent. Numerical investigations demonstrate the superiority of our method over many existing TOA SL schemes in terms of positioning accuracy and computational simplicity. In particular, the proposed ADMM achieves estimation results with mean square error performance closer to the Cramér–Rao lower bound than its competitors in our simulations of impulsive noise environments.

Robust and Sparse Portfolio: Optimization Models and Algorithms

The robust and sparse portfolio selection problem is one of the most-popular and -frequently studied problems in the optimization and financial literature. By considering the uncertainty of the parameters, the goal is to construct a sparse portfolio with low volatility and decent returns, subject to other investment constraints. In this paper, we propose a new portfolio selection model, which considers the perturbation in the asset return matrix and the parameter uncertainty in the expected asset return. We define three types of stationary points of the penalty problem: the Karush–Kuhn–Tucker point, the strong Karush–Kuhn–Tucker point, and the partial minimizer. We analyze the relationship between these stationary points and the local/global minimizer of the penalty model under mild conditions. We design a penalty alternating-direction method to obtain the solutions. Compared with several existing portfolio models on seven real-world datasets, extensive numerical experiments demonstrate the robustness and effectiveness of our model in generating lower volatility.

An event-triggered collaborative neurodynamic approach to distributed global optimization

In this paper, we propose an event-triggered collaborative neurodynamic approach to distributed global optimization in the presence of nonconvexity. We design a projection neural network group consisting of multiple projection neural networks coupled via a communication network. We prove the convergence of the projection neural network group to Karush–Kuhn–Tucker points of a given global optimization problem. To reduce communication bandwidth consumption, we adopt an event-triggered mechanism to liaise with other neural networks in the group with the Zeno behavior being precluded. We employ multiple projection neural network groups for scattered searches and re-initialize their states using a meta-heuristic rule in the collaborative neurodynamic optimization framework. In addition, we apply the collaborative neurodynamic approach for distributed optimal chiller loading in a heating, ventilation, and air conditioning system.

A distributed approach for solving AC–DC multi-objective OPF problem

This paper first proposes an optimization model for Multi-Objective Optimal Power Flow (MO-OPF) problem in AC–DC networks. The considered objective functions are the fuel cost, active power losses, and environmental emission. The AC and DC power flow equations, the converter operating limits, the voltage magnitude limits, and line capacity limits are considered in the proposed AC–DC MO-OPF model. Then an equivalent Gradient Dynamical System (GDS) for the AC–DC MO-OPF model is derived. Each equilibrium point of this GDS is a Karush–Kuhn–Tucker (KKT) point of the original AC–DC MO-OPF model. In Theorem 1, we derive condition under which this KKT point is a saddle point of the original AC–DC MO-OPF problem. The equivalent GDS has the property that it can be solved using a distributed solution algorithm. Every bus in the AC–DC power grid is considered as a local agent in the proposed distributed GDS algorithm, which only obtains information only from its neighbors. In Theorem 2, we prove that proposed distributed approach is locally convergent to the saddle point of the original AC–DC MO-OPF problem. To show the performance of our distributed approach to solve our developed AC–DC MO-OPF model, the simulation results based on several test systems are provided.

Improved TV Image Denoising over Inverse Gradient

Noise in an image can affect one’s extraction of image information, therefore, image denoising is an important image pre-processing process. Many of the existing models have a large number of estimated parameters, which increases the time complexity of the model solution and the achieved denoising effect is less than ideal. As a result, in this paper, an improved image-denoising algorithm is proposed based on the TV model, which effectively solves the above problems. The L1 regularization term can make the solution generated by the model sparser, thus facilitating the recovery of high-quality images. Reducing the number of estimated parameters, while using the inverse gradient to estimate the regularization parameters, enables the parameters to achieve global adaption and improves the denoising effect of the model in combination with the TV regularization term. The split Bregman iteration method is used to decouple the model into several related subproblems, and the solutions of the coordinated subproblems are derived as optimal solutions. It is also shown that the solution of the model converges to a Karush–Kuhn–Tucker point. Experimental results show that the algorithm in this paper is more effective in both preserving image texture structure and suppressing image noise.

Properties of the Solution Set of Absolute Value Equations and the Related Matrix Classes

The absolute value equations (AVE) problem is an algebraic problem of solving . So far, most of the research has focused on methods for solving AVE, but we address the problem itself by analyzing properties of AVE and the corresponding solution set. In particular, we investigate topological properties of the solution set, such as convexity, boundedness, or connectedness, or whether it consists of finitely many solutions. Further, we address problems related to the nonnegativity of solutions such as solvability or unique solvability. AVE can be formulated by means of different optimization problems, and in this regard we are interested in how the solutions of AVE are related with optima, Karush–Kuhn–Tucker points, and feasible solutions of these optimization problems. We characterize the matrix classes associated with the above mentioned properties and inspect the computational complexity of the recognition problem; some of the classes are polynomially recognizable, but some others are proved to be NP-hard. For the intractable cases, we propose various sufficient conditions. We also post new challenging problems that were raised during the investigation of the problem.

The Augmented Lagrangian Method for Mathematical Programs with Vertical Complementarity Constraints Based on Inexact Scholtes Regularization

Mathematical program with vertical complementarity constraints (MPVCC) plays an important role in economics and engineering. Due to the vertical complementarity structure, most of the standard constraint qualifications fail to hold at a feasible point. Without constraint qualifications, the Karush–Kuhn–Tucker (KKT) conditions are not necessary optimality conditions. The classical methods for solving constrained optimization problems applied to MPVCC are likely to fail. It is necessary to establish efficient algorithms for solving MPVCC from both theoretical and numerical points of view. We present an algorithm to obtain stationarity of MPVCC by solving a sequence of Scholtes regularized problems. We consider the Scholtes regularization method with the sequence of approximate KKT points only. We prove that, under strictly weaker constraint qualifications, the accumulation point of the approximate KKT points is Clarke (C-) stationary point. In particular, we can get Mordukhovich (M-) or strongly (S-) stationary point under additional assumptions. From these results, we apply an augmented Lagrangian method to obtain a solution of MPVCC and give the convergence analysis. In particular, the accumulation point of the generated iterates is an S-stationary point if some boundedness conditions hold. The numerical results show that it is an effective way to solve MPVCC.

The Boosted DC Algorithm for Linearly Constrained DC Programming

The Boosted Difference of Convex functions Algorithm (BDCA) has been recently introduced to accelerate the performance of the classical Difference of Convex functions Algorithm (DCA). This acceleration is achieved thanks to an extrapolation step from the point computed by DCA via a line search procedure. In this work, we propose an extension of BDCA that can be applied to difference of convex functions programs with linear constraints, and prove that every cluster point of the sequence generated by this algorithm is a Karush–Kuhn–Tucker point of the problem if the feasible set has a Slater point. When the objective function is quadratic, we prove that any sequence generated by the algorithm is bounded and R-linearly (geometrically) convergent. Finally, we present some numerical experiments where we compare the performance of DCA and BDCA on some challenging problems: to test the copositivity of a given matrix, to solve one-norm and infinity-norm trust-region subproblems, and to solve piecewise quadratic problems with box constraints. Our numerical results demonstrate that this new extension of BDCA outperforms DCA.

Feasible Newton methods for symmetric tensor Z-eigenvalue problems

Finding a Z-eigenpair of a symmetric tensor is equivalent to finding a Karush–Kuhn–Tucker point of a sphere constrained minimization problem. Based on this equivalency, in this paper, we first propose a class of iterative methods to get a Z-eigenpair of a symmetric tensor. Each method can generate a sequence of feasible points such that the sequence of function evaluations is decreasing. These methods can be regarded as extensions of the descent methods for unconstrained optimization problems. We pay particular attention to the Newton method. We show that under appropriate conditions, the Newton method is globally and quadratically convergent. Moreover, after finitely many iterations, the unit steplength will always be accepted. We also propose a nonlinear equations-based Newton method and establish its global and quadratic convergence. In the end, we do several numerical experiments to test the proposed Newton methods. The results show that both Newton methods are very efficient.

On a solution method in indefinite quadratic programming under linear constraints

We establish some properties of the Proximal Difference-of-Convex functions decomposition algorithm in indefinite quadratic programming under linear constraints. The first property states that any iterative sequence generated by the algorithm is root linearly convergent to a Karush–Kuhn–Tucker point, provided that the problem has a solution. The second property says that iterative sequences generated by the algorithm converge to a locally unique solution of the problem if the initial points are taken from a suitably chosen neighbourhood of it. Through a series of numerical tests, we analyse the influence of the decomposition parameter on the rate of convergence of the iterative sequences and compare the performance of the Proximal Difference-of-Convex functions decomposition algorithm with that of the Projection Difference-of-Convex functions decomposition algorithm. In addition, the performances of the above algorithms and the Gurobi software in solving some randomly generated nonconvex quadratic programs are compared.

Linear programming with a feasible direction interior point technique for smooth optimization

We propose an adaptation of the Feasible Direction Interior Points Algorithm (FDIPA) of J. Herskovits, for solving large-scale linear programs. At each step, the solution of two linear systems with the same coefficient matrix is determined. This step involves a significant computational effort. Reducing the solution time of linear systems is, therefore, a way to improve the performance of the method. The linear systems to be solved are associated with definite positive symmetric matrices. Therefore, we use Split Preconditioned Conjugate Gradient (SPCG) method to solve them, together with an Incomplete Cholesky preconditioner using Matlab’s ICHOL function. We also propose to use the first iteration of the conjugate gradient, and to presolve before applying the algorithm, in order to reduce the computational cost. Following, we then provide mathematica proof that show that the iterations approach Karush–Kuhn–Tucker points of the problem under reasonable assumptions. Finally, numerical evidence show that the method not only works in theory but is also competitive with more advanced methods.

Multispectrally Constrained MIMO Radar Beampattern Design via Sequential Convex Approximation

This article deals with the multiple-input–multiple-output (MIMO) radar beampattern design in an effort to the coexistence with multiple communication systems. A waveform optimization model accounting for the minimization of the beampattern integrated sidelobe level (ISL) along with the mainlobe width, peak-to-average power ratio, and energy constraints, as well as multispectral requirements where the interference energy injected by the MIMO radar in each shared frequency band in a particular direction, is precisely controlled to ensure the desired quality of service at each communication system. Through an equivalent reformulation of the original nonconvex problem, a polynomial-time sequential convex approximation (SCA) procedure that involves the tackling of a series of constrained convex problems is proposed to monotonically decrease the ISL with the convergence guaranteed to a Karush–Kuhn–Tucker point. Herein, to speed up the convergence, a fast iterative algorithm based on the alternating-direction-method-of-multipliers framework is introduced to globally solve the convex problems during each iteration of the SCA procedure. Numerical results are provided to assess the proposed algorithm in terms of the computational complexity, the achieved beampattern, and spectral compatibility with some competitive counterparts available in the open literatures.

Continuation methods for nonnegative rank‐1 approximation of nonnegative tensors

AbstractIn this article, the rank‐1 approximation of a nonnegative tensor is considered. Mathematically, the approximation problem can be formulated as an optimization problem. The Karush–Kuhn–Tucker (KKT) point of the optimization problem can be obtained by computing the nonnegative Z‐eigenvector y of enlarged tensor . Therefore, we propose an iterative method with prediction and correction steps for computing nonnegative Z‐eigenvector y of enlarged tensor , called the continuation method. In the theoretical part, we show that the computation requires only flops for each iteration and the computed Z‐eigenvector y has nonzero component block, and hence, the KKT point can be obtained. In addition, we show that the KKT point is a local optimizer of the optimization problem. Numerical experiments are provided to support the theoretical results.

Distributed Nonconvex Optimization for Energy Efficiency in Mobile Ad Hoc Networks

This article proposes a distributed nonconvex optimization algorithm for energy efficiency (EE) in mobile ad hoc networks (MANETs). The proposed model for EE in MANETs can be classified into the category of nonconvex optimization problems. There are various distributed algorithms for solving nonconvex optimization problems. But, they are associated with some flaws such as complexity of implementation and low convergence rate. In this regard, in addition to the simplicity in implementation, the optimal point of the proposed model is a saddle point. Using augmented Lagrangian function with continuous-time optimization dynamics algorithm, a dynamic system will be established where its equilibrium points are Karush–Kuhn–Tucker points of the proposed model. Also, it will be shown that these points can be locally asymptotically stable under some conditions. Additionally, the proposed analytic method proves that the rate of convergence increases with an increasing penalty coefficient. To validate the performance of the presented algorithm, the problem of EE optimization for a sample mobile ad hoc network is simulated in MATLAB environment.