Non-convex Optimization Method Research Articles

Nonlinear convex and concave relaxations for the solutions of parametric ODEs

SUMMARYConvex and concave relaxations for the parametric solutions of ordinary differential equations (ODEs) are central to deterministic global optimization methods for nonconvex dynamic optimization and open‐loop optimal control problems with control parametrization. Given a general system of ODEs with parameter dependence in the initial conditions and right‐hand sides, this work derives sufficient conditions under which an auxiliary system of ODEs describes convex and concave relaxations of the parametric solutions, pointwise in the independent variable. Convergence results for these relaxations are also established. A fully automatable procedure for constructing an appropriate auxiliary system has been developed previously by the authors. Thus, the developments here lead to an efficient, automatic method for computing convex and concave relaxations for the parametric solutions of a very general class of nonlinear ODEs. The proposed method is presented in detail for a simple example problem. Copyright © 2012 John Wiley & Sons, Ltd.

Trust, But Verify: Fast and Accurate Signal Recovery From 1-Bit Compressive Measurements

The recently emerged compressive sensing (CS) framework aims to acquire signals at reduced sample rates compared to the classical Shannon-Nyquist rate. To date, the CS theory has assumed primarily real-valued measurements; it has recently been demonstrated that accurate and stable signal acquisition is still possible even when each measurement is quantized to just a single bit. This property enables the design of simplified CS acquisition hardware based around a simple sign comparator rather than a more complex analog-to-digital converter; moreover, it ensures robustness to gross nonlinearities applied to the measurements. In this paper we introduce a new algorithm - restricted-step shrinkage (RSS) - to recover sparse signals from 1-bit CS measurements. In contrast to previous algorithms for 1-bit CS, RSS has provable convergence guarantees, is about an order of magnitude faster, and achieves higher average recovery signal-to-noise ratio. RSS is similar in spirit to trust-region methods for nonconvex optimization on the unit sphere, which are relatively unexplored in signal processing and hence of independent interest.

Connection of some bilevel and nonlinear optimization problems

In this paper we reduce a quadratic-linear bilevel optimization problem with a guaranteed solution to a family of bilevel problems in the optimistic statement. Then we reduce the obtained bilevel problems to nonconvex one-level optimization problems and solve the latter by nonconvex optimization methods.

Outer Trust-Region Method for Constrained Optimization

Given an algorithm A for solving some mathematical problem based on the iterative solution of simpler subproblems, an outer trust-region (OTR) modification of A is the result of adding a trust-region constraint to each subproblem. The trust-region size is adaptively updated according to the behavior of crucial variables. The new subproblems should not be more complex than the original ones, and the convergence properties of the OTR algorithm should be the same as those of Algorithm A. In the present work, the OTR approach is exploited in connection with the greediness phenomenon of nonlinear programming. Convergence results for an OTR version of an augmented Lagrangian method for nonconvex constrained optimization are proved, and numerical experiments are presented.

Fixed-order strong H-infinity control of interconnected systems with time-delays

AbstractWe design fixed-order strong H-infinity controllers for general time-delay systems. The designer chooses the controller order and may introduce constant time-delays in the controller. We represent the closed-loop system of the plant and the controller as delay differential algebraic equations (DDAEs). This representation deals with any interconnection of systems with time-delays without any elimination techniques. We present a numerical algorithm to compute the strong H-infinity norm for DDAEs which is robust to arbitrarily small delay perturbations, unlike the standard H-infinity norm. We optimize the strong H-infinity norm of the closed-loop system based on non-smooth, non-convex optimization methods using this algorithm and the computation of the gradient of the strong H-infinity norm with respect to the controller parameters. We tune the controller parameters and design H-infinity controllers with a prescribed order or structure.

Fixed-Order H-Infinity Control for Interconnected Systems Using Delay Differential Algebraic Equations

We analyze and design H-infinity controllers for general time-delay systems with time-delays in systems' state, inputs, and outputs. We allow the designer to choose the order of the controller and to introduce constant time-delays into the controller. The closed-loop system of the plant and the controller is modeled by a system of delay differential algebraic equations (DDAEs). The advantage of the DDAE modeling framework is that any interconnection of systems and controllers prone to various types of delays can be dealt with in a systematic way, without using any elimination technique. We present a predictor-corrector algorithm for the H-infinity norm computation of systems described by DDAEs. Instrumental to this we analyze the properties of the H-infinity norm. In particular, we illustrate that it may be sensitive with respect to arbitrarily small delay perturbations. Due to this sensitivity, we introduce the strong H-infinity norm, which explicitly takes into account small delay perturbations, inevitable in any practical control application. We present a numerical algorithm to compute the strong H-infinity norm for DDAEs. Using this algorithm and the computation of the gradient of the strong H-infinity norm with respect to the controller parameters, we minimize the strong H-infinity norm of the closed-loop system based on nonsmooth, nonconvex optimization methods. By this approach, we tune the controller parameters and design H-infinity controllers with a prescribed order or structure.

Compressed sensing of color images

This work proposes a method for color imaging via compressive sampling. Random projections from each of the color channels are acquired separately. The problem is to reconstruct the original color image from the randomly projected (sub-sampled) data. Since each of the color channels are sparse in some domain (DCT, Wavelet, etc.) one way to approach the reconstruction problem is to apply sparse optimization algorithms. We note that the color channels are highly correlated and propose an alternative reconstruction method based on group sparse optimization. Two new non-convex group sparse optimization methods are proposed in this work. Experimental results show that incorporating group sparsity into the reconstruction problem produces significant improvement (more than 1 dB PSNR) over ordinary sparse algorithm.

On the Complexity of Steepest Descent, Newton's and Regularized Newton's Methods for Nonconvex Unconstrained Optimization Problems

It is shown that the steepest-descent and Newton's methods for unconstrained nonconvex optimization under standard assumptions may both require a number of iterations and function evaluations arbitrarily close to $O(\epsilon^{-2})$ to drive the norm of the gradient below $\epsilon$. This shows that the upper bound of $O(\epsilon^{-2})$ evaluations known for the steepest descent is tight and that Newton's method may be as slow as the steepest-descent method in the worst case. The improved evaluation complexity bound of $O(\epsilon^{-3/2})$ evaluations known for cubically regularized Newton's methods is also shown to be tight.

A Redistributed Proximal Bundle Method for Nonconvex Optimization

Proximal bundle methods have been shown to be highly successful optimization methods for unconstrained convex problems with discontinuous first derivatives. This naturally leads to the question of whether proximal variants of bundle methods can be extended to a nonconvex setting. This work proposes an approach based on generating cutting-planes models, not of the objective function as most bundle methods do but of a local convexification of the objective function. The corresponding convexification parameter is calculated “on the fly” in such a way that the algorithm can inform the user as to what proximal parameters are sufficiently large that the objective function is likely to have well-defined proximal points. This novel approach, shown to be sound from both the objective function and subdifferential modelling perspectives, opens the way to create workable nonconvex algorithms based on nonconvex $\mathcal{VU}$ theory. Both theoretical convergence analysis and some encouraging preliminary numerical experience are provided.

Robust optimization with simulated annealing

Complex systems can be optimized to improve the performance with respect to desired functionalities. An optimized solution, however, can become suboptimal or even infeasible, when errors in implementation or input data are encountered. We report on a robust simulated annealing algorithm that does not require any knowledge of the problems structure. This is necessary in many engineering applications where solutions are often not explicitly known and have to be obtained by numerical simulations. While this nonconvex and global optimization method improves the performance as well as the robustness, it also warrants for a global optimum which is robust against data and implementation uncertainties. We demonstrate it on a polynomial optimization problem and on a high-dimensional and complex nanophotonic engineering problem and show significant improvements in efficiency as well as in actual optimality.

Comments on “Dual methods for nonconvex spectrum optimization of multicarrier systems”

Yu and Liu’s strong duality theorem under the time-sharing property requires additionally the Slater regularity condition to hold for the considered general nonconvex problem, which is naturally satisfied for the specific application. We further extend the scope of the theorem under Ky Fan convexity, which is slightly weaker than Yu and Lui’s time-sharing property.

A new accelerating method for global non-convex quadratic optimization with non-convex quadratic constraints

In this paper, we combine the new global optimization method proposed by Qu et al. [S.J. Qu, K.C. Zhang, Y. Ji, A global optimization algorithm using parametric linearization relaxation, Appl. Math. Comput. 186 (2007) 763–771] with a suitable deleting technique to propose a new accelerating global optimization algorithm for solving the non-convex quadratic optimization problems with non-convex quadratic constraints (NQP). This technique offers a possibility to cut away a large part of the currently investigated region in which the global optimal solution of NQP does not exist, and can be seen as an accelerating device for the global optimization algorithm of the NQP problems. Compared with the method in Qu et al. (2007), numerical results show that the computational efficiency is obviously improved by using this new technique in the number of iterations, the required list length and the overall execution time of the algorithm.

On duality bound methods for nonconvex global optimization

A counter-example is given to several recently published results on duality bound methods for nonconvex global optimization.

Dual methods for nonconvex spectrum optimization of multicarrier systems

The design and optimization of multicarrier communications systems often involve a maximization of the total throughput subject to system resource constraints. The optimization problem is numerically difficult to solve when the problem does not have a convexity structure. This paper makes progress toward solving optimization problems of this type by showing that under a certain condition called the time-sharing condition, the duality gap of the optimization problem is always zero, regardless of the convexity of the objective function. Further, we show that the time-sharing condition is satisfied for practical multiuser spectrum optimization problems in multicarrier systems in the limit as the number of carriers goes to infinity. This result leads to efficient numerical algorithms that solve the nonconvex problem in the dual domain. We show that the recently proposed optimal spectrum balancing algorithm for digital subscriber lines can be interpreted as a dual algorithm. This new interpretation gives rise to more efficient dual update methods. It also suggests ways in which the dual objective may be evaluated approximately, further improving the numerical efficiency of the algorithm. We propose a low-complexity iterative spectrum balancing algorithm based on these ideas, and show that the new algorithm achieves near-optimal performance in many practical situations

Design of ODMA Digital Waveforms Using Non-Convex Optimization Methods

This paper presents a new approach for the design of a set of orthogonal digital filters which can be used as message-carrying waveforms in orthogonality division multiple access (ODMA) communication systems. The filter set design problem is formulated as a constrained L 2 space optimization problem. For bandwidth efficiency, all the digital filters in the set are constrained to have approximately the same desired spectral shapes in a prescribed passband; To minimize intersymbol interference and co-channel interference, all the digital filters in the set are constrained to have low values of auto-correlation at nonzero translates of multiple symbol interval and low values of cross-correlation at all translates of multiple symbol interval. Methods for solving the proposed non-convex optimization problem are outlined. Numerical results are presented to illustrate the usefulness of the proposed method.

Efficient Design of Orthonormal Wavelet Bases for Signal Representation

The efficient representation of a signal as a linear combination of elementary "atoms" or building blocks is central to much signal processing theory and many applications. Wavelets provide a powerful, flexible, and efficiently implementable class of such atoms. In this paper, we develop an efficient method for selecting an orthonormal wavelet that is matched to a given signal in the sense that the squared error between the signal and some finite resolution wavelet representation of it is minimized. Since the squared error is not an explicit function of the design parameters, some form of approximation of this objective is required if conventional optimization techniques are to be used. Previous approximations have resulted in nonconvex optimization problems, which require delicate management of local minima. In this paper, we employ an approximation that results in a design problem that can be transformed into a convex optimization problem and efficiently solved. Constraints on the smoothness of the wavelet can be efficiently incorporated into the design. We show that the error incurred in our approximation is bounded by a function that decays to zero as the number of vanishing moments of the wavelet grows. In our examples, we demonstrate that our method provides wavelet bases that yield substantially better performance than members of standard wavelet families and are competitive with those designed by more intricate nonconvex optimization methods.

Discretization and localization in successive convex relaxation methods for nonconvex quadratic optimization

Based on the authors’ previous work which established theoretical foundations of two, conceptual, successive convex relaxation methods, i.e., the SSDP (Successive Semidefinite Programming) Relaxation Method and the SSILP (Successive Semi-Infinite Linear Programming) Relaxation Method, this paper proposes their implementable variants for general quadratic optimization problems. These problems have a linear objective function c T x to be maximized over a nonconvex compact feasible region F described by a finite number of quadratic inequalities. We introduce two new techniques, “discretization” and “localization,” into the SSDP and SSILP Relaxation Methods. The discretization technique makes it possible to approximate an infinite number of semi-infinite SDPs (or semi-infinite LPs) which appeared at each iteration of the original methods by a finite number of standard SDPs (or standard LPs) with a finite number of linear inequality constraints. We establish:¶•Given any open convex set U containing F, there is an implementable discretization of the SSDP (or SSILP) Relaxation Method which generates a compact convex set C such that F⊆C⊆U in a finite number of iterations.¶The localization technique is for the cases where we are only interested in upper bounds on the optimal objective value (for a fixed objective function vector c) but not in a global approximation of the convex hull of F. This technique allows us to generate a convex relaxation of F that is accurate only in certain directions in a neighborhood of the objective direction c. This cuts off redundant work to make the convex relaxation accurate in unnecessary directions. We establish:¶•Given any positive number e, there is an implementable localization-discretization of the SSDP (or SSILP) Relaxation Method which generates an upper bound of the objective value within e of its maximum in a finite number of iterations.

A Heuristic Method for Nonconvex Optimization in Mechanics: Conceptual Idea, Theoretical Justification, Engineering Applications

Structures involving nonmonotone, possibly multivalued reaction-displacement or stress-strain laws cannot be effectively treated by the numerical methods for classical non-linearities. In this paper we make use of the fact that these problems have as a variational formulation a hemivariational inequality, leading to a noncovex optimization problem. A new method is proposed which approximates the nonmonotone problem by a series of monotone ones. The method constitutes an iterative scheme for the approximation of the solutions of the corresponding hemivariational inequality. A simple numerical example demonstrates the con-ceptual idea of the proposed numerical method. In the sequel the method is applied on an engineering problem concerning the ultimate strength analysis of an eccentric braced steel frame.

Comparing Nonsmooth Nonconvex Bundle Methods in Solving Hemivariational Inequalities

Hemivariational inequalities can be considered as a generalization of variational inequalities. Their origin is in nonsmooth mechanics of solid, especially in nonmonotone contact problems. The solution of a hemivariational inequality proves to be a substationary point of some functional, and thus can be found by the nonsmooth and nonconvex optimization methods. We consider two type of bundle methods in order to solve hemivariational inequalities numerically: proximal bundle and bundle-Newton methods. Proximal bundle method is based on first order polyhedral approximation of the locally Lipschitz continuous objective function. To obtain better convergence rate bundle-Newton method contains also some second order information of the objective function in the form of approximate Hessian. Since the optimization problem arising in the hemivariational inequalities has a dominated quadratic part the second order method should be a good choice. The main question in the functioning of the methods is how remarkable is the advantage of the possible better convergence rate of bundle-Newton method when compared to the increased calculation demand.

Comparative studies on nonconvex optimization methods for transmission network expansion planning

We have investigated and extensively tested three families of nonconvex optimization approaches for solving the transmission network expansion planning problem: simulated annealing (SA), genetic algorithms (GA), and tabu search algorithms (TS). The paper compares the main features of the three approaches and presents an integrated view of these methodologies. A hybrid approach is then proposed which presents performances which are far better than the ones obtained with any of these approaches individually. Results obtained in tests performed with large scale real-life networks are summarized.