General Nonconvex Research Articles

UAdam: Unified Adam-Type Algorithmic Framework for Nonconvex Optimization.

Adam-type algorithms have become a preferred choice for optimization in the deep learning setting; however, despite their success, their convergence is still not well understood. To this end, we introduce a unified framework for Adam-type algorithms, termed UAdam. It is equipped with a general form of the second-order moment, which makes it possible to include Adam and its existing and future variants as special cases, such as NAdam, AMSGrad, AdaBound, AdaFom, and Adan. The approach is supported by a rigorous convergence analysis of UAdam in the general nonconvex stochastic setting, showing that UAdam converges to the neighborhood of stationary points with a rate of O(1/T). Furthermore, the size of the neighborhood decreases as the parameter β1 increases. Importantly, our analysis only requires the first-order momentum factor to be close enough to 1, without any restrictions on the second-order momentum factor. Theoretical results also reveal the convergence conditions of vanilla Adam, together with the selection of appropriate hyperparameters. This provides a theoretical guarantee for the analysis, applications, and further developments of the whole general class of Adam-type algorithms. Finally, several numerical experiments are provided to support our theoretical findings.

Convex Hull Relaxation of Optimal Control Problems With General Nonconvex Control Constraints

Convex Hull Relaxation of Optimal Control Problems With General Nonconvex Control Constraints

Optimization under uncertainty of a hybrid waste tire and natural gas feedstock flexible polygeneration system using a decomposition algorithm

Market uncertainties motivate the development of flexible polygeneration systems that are able to adjust operating conditions to favor production of the most profitable product portfolio. However, this operational flexibility comes at the cost of higher capital expenditure. A scenario-based two-stage stochastic nonconvex Mixed-Integer Nonlinear Programming (MINLP) approach lends itself naturally to optimizing these trade-offs. This work studies the optimal design and operation under uncertainty of a hybrid feedstock flexible polygeneration system producing electricity, methanol, dimethyl ether, olefins or liquefied (synthetic) natural gas. A recently developed C++ based software framework (named GOSSIP) is used for modeling the optimization problem as well as its efficient solution using the Nonconvex Generalized Benders Decomposition (NGBD) algorithm. Two different cases are studied: The first uses estimates of the means and variances of the uncertain parameters from historical data, whereas the second assesses the impact of increased uncertain parameter volatility. The value of implementing flexible designs characterized by the value of the stochastic solution (VSS) is in the range of 260–405 M$ for a scale of approximately 893 MW of thermal input. Increased price volatility around the same mean results in higher expected net present value and VSS as operational flexibility allows for asymmetric exploitation of price peaks.

New Classes of Exponentially General Nonconvex Variational Inequalities

In this paper, some new classes of exponentially general nonconvex variational inequalities are introduced and investigated. Several special cases are discussed as applications of these nonconvex variational inequalities. Projection technique is used to establish the equivalence between the non covex variational inequalities and fixed point problem. This equivalent formulation is used to discuss the existence of the solution. Several inertial type methods are suggested and analyzed for solving exponentially general nonconvex variational inequalities. using the technique of the projection operator and dynamical systems. Convergence analysis of the iterative methods is analyzed under suitable and appropriate weak conditions. In this sense, our result can be viewed as improvement and refinement of the previously known results. Our methods of proof are very simple as compared with other techniques.

Sequential Convex Programming Methods for Real-Time Optimal Trajectory Planning in Autonomous Vehicle Racing

Optimization problems for trajectory planning in autonomous vehicle racing are characterized by their nonlinearity and nonconvexity. Instead of solving these optimization problems, usually a convex approximation is solved instead to achieve a high update rate. We present a real-time-capable model predictive control (MPC) trajectory planner based on a nonlinear single-track vehicle model and Pacejka’s magic tire formula for autonomous vehicle racing. After formulating the general nonconvex trajectory optimization problem, we form a convex approximation using sequential convex programming (SCP). The state of the art convexifies track constraints using sequential linearization (SL), which is a method of relaxing the constraints. Solutions to the relaxed optimization problem are not guaranteed to be feasible in the nonconvex optimization problem. We propose sequential convex restriction (SCR) as a method to convexify track constraints. SCR guarantees that resulting solutions are feasible in the nonconvex optimization problem. We show recursive feasibility of solutions to the restricted optimization problem. The MPC is evaluated on a scaled version of the Hockenheimring racing track in simulation. The results show that MPC using SCR yields faster lap times than MPC using SL, while still being real-time capable.

A two-level distributed algorithm for nonconvex constrained optimization

This paper aims to develop distributed algorithms for nonconvex optimization problems with complicated constraints associated with a network. The network can be a physical one, such as an electric power network, where the constraints are nonlinear power flow equations, or an abstract one that represents constraint couplings between decision variables of different agents. Despite the recent development of distributed algorithms for nonconvex programs, highly complicated constraints still pose a significant challenge in theory and practice. We first identify some difficulties with the existing algorithms based on the alternating direction method of multipliers (ADMM) for dealing with such problems. We then propose a reformulation that enables us to design a two-level algorithm, which embeds a specially structured three-block ADMM at the inner level in an augmented Lagrangian method framework. Furthermore, we prove the global and local convergence as well as iteration complexity of this new scheme for general nonconvex constrained programs, and show that our analysis can be extended to handle more complicated multi-block inner-level problems. Finally, we demonstrate with computation that the new scheme provides convergent and parallelizable algorithms for various nonconvex applications, and is able to complement the performance of the state-of-the-art distributed algorithms in practice by achieving either faster convergence in optimality gap or in feasibility or both.

Improving the Variational Quantum Eigensolver Using Variational Adiabatic Quantum Computing

The variational quantum eigensolver (VQE) is a hybrid quantum-classical algorithm for finding the minimum eigenvalue of a Hamiltonian that involves the optimization of a parameterized quantum circuit. Since the resulting optimization problem is in general nonconvex, the method can converge to suboptimal parameter values that do not yield the minimum eigenvalue. In this work, we address this shortcoming by adopting the concept of variational adiabatic quantum computing (VAQC) as a procedure to improve VQE. In VAQC, the ground state of a continuously parameterized Hamiltonian is approximated via a parameterized quantum circuit. We discuss some basic theory of VAQC to motivate the development of a hybrid quantum-classical homotopy continuation method. The proposed method has parallels with a predictor-corrector method for numerical integration of differential equations. While there are theoretical limitations to the procedure, we see in practice that VAQC can successfully find good initial circuit parameters to initialize VQE. We demonstrate this with two examples from quantum chemistry. Through these examples, we provide empirical evidence that VAQC, combined with other techniques (an adaptive termination criteria for the classical optimizer and a variance-based resampling method for the expectation evaluation), can provide more accurate solutions than “plain” VQE, for the same amount of effort.

Muscle-Effort-Minimization-Inspired Kinematic Redundancy Resolution for Replicating Natural Posture of Human Arm.

Replicating natural postures of human arms is essential to generate human-like behaviors in robotic applications for humans nearby. However, how to realize this requirement in interactive scenarios remains a challenge due to the kinematic redundancy and unknown postural control strategy of human arms. Inspired by the physiological characteristics that the musculoskeletal system is coordinated to minimize muscle effort in human behaviors, this paper aims to address the issue by solving a muscle effort minimization problem. It adopts a high-fidelity human arm musculoskeletal model (HAMM) and considers the implicit constraint (desired hand pose) and the inequality constraints (range of joint motion). The constrained minimization is in general nonconvex, consequently sensitive to initial guesses in iterative procedures. So, it is impracticable to solve it directly with existing gradient-based deterministic approaches or standard evolutionary algorithms. As the main contribution, a hybrid inverse kinematics algorithm was proposed for the HAMM with 7 independent and 13 mimic joints to obtain the feasible arm postures satisfying the minimization constraints. Using the arm swivel angle that parametrizes the kinematic redundancy of the HAMM, geometrically equidistant initial guess candidates can be generated over the 1-dimension feasible posture manifold. As another contribution, we present a two-phase global minimization algorithm to handle the nonconvexity of the constrained minimization. It consists of a local-search phase on the null-space of the geometric Jacobian matrix and a global-search phase with an initial guess resampling strategy. The proposed approach was validated by replicating the natural arm postures of 5 right-handed subjects in daily tasks.

Nonconvex flexible sparsity regularization: theory and monotone numerical schemes

Flexible sparsity regularization means stably approximating sparse solutions of operator equations by using coefficient-dependent penalizations. We propose and analyse a general nonconvex approach in this respect, from both theoretical and numerical perspectives. Namely, we show convergence of the regularization method and establish convergence properties of a couple of majorization approaches for the associated nonconvex problems. We also test a monotone algorithm for an academic example where the operator is an M matrix, and on a time-dependent optimal control problem, pointing out the advantages of employing variable penalties over a fixed penalty.

Opinion Dynamics Optimization by Varying Susceptibility to Persuasion via Non-Convex Local Search

A long line of work in social psychology has studied variations in people’s susceptibility to persuasion—the extent to which they are willing to modify their opinions on a topic. This body of literature suggests an interesting perspective on theoretical models of opinion formation by interacting parties in a network: in addition to considering interventions that directly modify people’s intrinsic opinions, it is also natural to consider interventions that modify people’s susceptibility to persuasion. In this work, motivated by this fact, we propose an influence optimization problem. Specifically, we adopt a popular model for social opinion dynamics, where each agent has some fixed innate opinion, and a resistance that measures the importance it places on its innate opinion; agents influence one another’s opinions through an iterative process. Under certain conditions, this iterative process converges to some equilibrium opinion vector. For the unbudgeted variant of the problem, the goal is to modify the resistance of any number of agents (within some given range) such that the sum of the equilibrium opinions is minimized; for the budgeted variant, in addition the algorithm is given upfront a restriction on the number of agents whose resistance may be modified. We prove that the objective function is in general non-convex. Hence, formulating the problem as a convex program as in an early version of this work (Abebe et al., KDD’18) might have potential correctness issues. We instead analyze the structure of the objective function, and show that any local optimum is also a global optimum, which is somehow surprising as the objective function might not be convex. Furthermore, we combine the iterative process and the local search paradigm to design very efficient algorithms that can solve the unbudgeted variant of the problem optimally on large-scale graphs containing millions of nodes. Finally, we propose and evaluate experimentally a family of heuristics for the budgeted variant of the problem.

Multi Set-Point Explicit Model Predictive Control for Nonlinear Process Systems

In this article, we introduce a novel framework for the design of multi set-point nonlinear explicit controllers for process systems engineering problems where the set-points are treated as uncertain parameters simultaneously with the initial state of the dynamical system at each sampling instance. To this end, an algorithm for a special class of multi-parametric nonlinear programming problems with uncertain parameters on the right-hand side of the constraints and the cost coefficients of the objective function is presented. The algorithm is based on computed algebra methods for symbolic manipulation that enable an analytical solution of the optimality conditions of the underlying multi-parametric nonlinear program. A notable property of the presented algorithm is the computation of exact, in general nonconvex, critical regions that results in potentially great computational savings through a reduction in the number of convex approximate critical regions.

On the Cheeger problem for rotationally invariant domains

We investigate the properties of the Cheeger sets of rotationally invariant, bounded domains $\Omega \subset \mathbb{R}^n$. For a rotationally invariant Cheeger set $C$, the free boundary $\partial C \cap \Omega$ consists of pieces of Delaunay surfaces, which are rotationally invariant surfaces of constant mean curvature. We show that if $\Omega$ is convex, then the free boundary of $C$ consists only of pieces of spheres and nodoids. This result remains valid for nonconvex domains when the generating curve of $C$ is closed, convex, and of class $\mathcal{C}^{1,1}$. Moreover, we provide numerical evidence of the fact that, for general nonconvex domains, pieces of unduloids or cylinders can also appear in the free boundary of $C$.

Investigating minimum income condition orders on European power exchanges: Controversial properties and enhancement proposals

One of the most difficult challenges of deregulated electricity markets is the proper management and operation planning of economically non-convex generators. The origin of economic non-convexity is twofold. Firstly, generators generally have non-zero minimum output power (MOP) levels, below which they cannot sustain operation. Secondly, they might have substantial fixed costs. The direct consideration of these characteristics leads to very hard non-convex mathematical problems with the increased probability of computational difficulties and long solution time for the market clearing process.Complex orders with explicit Minimum Income Conditions (MICs) have been invented to provide a method to represent non-convex generators on European day-ahead power exchanges. The rationale is that a MIC can be attached to a set of simple supply bids, which means that these bids can be accepted only if their acquired incomes surpass bidder-specified minimal values.The present research is conducted to investigate several controversial issues concerning the current use of MICs from the viewpoint of market participants. Firstly, no direct mechanism is provided to handle the MOP of generators. Secondly, simple bid price constraints and simultaneously attached income criteria can have contradictory implications that lead to sub-optimal clearing results. Thirdly, the current use of MICs contributes to cheating opportunities, because it entails distortions in the social welfare calculation. The paper defines new ways of MIC attachment to circumvent the revealed hindrances. For this purpose, several modification techniques are considered such as the discretization of bid quantities, the relaxation of elementary bid pricing rules and the revision of social welfare calculation. Two novel order types are proposed, supported by comparative numerical simulations using realistic market data.

Decision regions and decision boundaries of generalized K mean algorithm based on various norm criteria

This paper derives the decision regions and the decision boundaries of the generalized K mean algorithms based on the L1 norm criterion, the L2 norm criterion and the L∞ norm criterion. The decision boundaries of these three generalized K mean algorithms are all linear hyperplanes. However, the total numbers of the decision boundaries of the generalized K mean algorithms based on both the L1 norm criterion and the L∞ norm criterion are more than that based on the L2 norm criterion. On the other hand, the decision regions of the generalized K mean algorithm based on the L2 norm criterion are convex while that based on both the L1 norm criterion and the L∞ norm criterion are in general nonconvex. The computer numerical simulations on a toy example demonstrate the above phenomena. Besides, two examples are illustrated. The first example on the patent image retrieval system shows that the recognition accuracies of using the generalized K mean algorithms based on the L2 norm criterion, the L1 norm criterion and the L∞ norm criterion are 58.25%, 61% and 58.75%, respectively. The second example on the electromyogram based Parkinson’s disease detection system shows that the recognition accuracies of using the generalized K mean algorithms based on the L2 norm criterion, the L1 norm criterion and the L∞ norm criterion are 60%, 60% and 67%, respectively, if the signals are classified directly in the time domain. On the other hand, the recognition accuracies of the generalized K mean algorithms based on the L2 norm criterion, the L1 norm criterion and the L∞ norm criterion are 60%, 87% and 60%, respectively, if the signals are classified directly in the discrete cosine transform domain. The improvements are due to the nonconvexity of the decision regions of the generalized K mean algorithm based on the L1 norm criterion and the L∞ norm criterion.

Strategic Bidding in Extended Locational Marginal Price Scheme

The locational marginal price scheme, though widely adopted, fails to reflect the startup costs for many generators in the real-time market. To solve this issue, the extended locational marginal price (eLMP in short, a.k.a. convex hull price) has been proposed. The key idea is to convexify the non-convex generation cost functions due to the startup costs, and then design the uplift payment whenever necessary. While eLMP partially solves the incentive issue, there are still chances for generators to manipulate the market prices (as well as the uplift payments) by strategic bidding. In this letter, we first propose a profit decomposition method to evaluate different bidding strategies’ impacts on individual payoffs. This decomposition allows us to better identify the potential strategic generators and investigate their best strategies. We further propose to use the maximal markup as an index for eLMP scheme to quantify market power in the whole system. Numerical studies further highlight the existence of market power in practice.

An efficient characterization of submodular spanning tree games

Cooperative games form an important class of problems in game theory, where a key goal is to distribute a value among a set of players who are allowed to cooperate by forming coalitions. An outcome of the game is given by an allocation vector that assigns a value share to each player. A crucial aspect of such games is submodularity (or convexity). Indeed, convex instances of cooperative games exhibit several nice properties, e.g. regarding the existence and computation of allocations realizing some of the most important solution concepts proposed in the literature. For this reason, a relevant question is whether one can give a polynomial-time characterization of submodular instances, for prominent cooperative games that are in general non-convex. In this paper, we focus on a fundamental and widely studied cooperative game, namely the spanning tree game. An efficient recognition of submodular instances of this game was not known so far, and explicitly mentioned as an open question in the literature. We here settle this open problem by giving a polynomial-time characterization of submodular spanning tree games.

On the uniqueness of solutions to quadratic BSDEs with non-convex generators and unbounded terminal conditions

We prove a uniqueness result of the unbounded solution for a quadratic backward stochastic differential equation whose terminal condition is unbounded and whose generator $g$ may be non-Lipschitz continuous in the state variable $y$, non-convex (non-concave) in the state variable $z$, and instead satisfies a strictly quadratic condition and an additional assumption. The key observation is that if the generator is strictly quadratic, then the quadratic variation of the first component of the solution admits an exponential moment. Typically, a Lipschitz perturbation of some convex (concave) function satisfies the additional assumption mentioned above. This generalizes some results obtained in \cite{BriandHu2006PTRF} and \cite{BriandHu2008PTRF}.

Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions.

Stochastic gradient descent (SGD) is a popular and efficient method with wide applications in training deep neural nets and other nonconvex models. While the behavior of SGD is well understood in the convex learning setting, the existing theoretical results for SGD applied to nonconvex objective functions are far from mature. For example, existing results require to impose a nontrivial assumption on the uniform boundedness of gradients for all iterates encountered in the learning process, which is hard to verify in practical implementations. In this article, we establish a rigorous theoretical foundation for SGD in nonconvex learning by showing that this boundedness assumption can be removed without affecting convergence rates, and relaxing the standard smoothness assumption to Hölder continuity of gradients. In particular, we establish sufficient conditions for almost sure convergence as well as optimal convergence rates for SGD applied to both general nonconvex and gradient-dominated objective functions. A linear convergence is further derived in the case with zero variances.

Optimality of Double Intervals in Persuasion: A Convex Programming Framework

We study information design problems, where the payoff of the information designer is an increasing step function of the posterior mean of the state her signals induce. Settings where the designer employs public signaling mechanisms to persuade multiple receivers (who have (i) finitely many actions that exhibit strategic complementarities, and (ii) best responses that are increasing in the state), and where her payoff is increasing in the receivers’ actions, are special cases. We provide a novel and tractable convex optimization framework for obtaining an optimal signaling mechanism. This mechanism is deterministic and admits a double-interval structure: it pools some “good” and “bad” states, and with each signal realization it associates at most two subintervals of the set of states. Our framework is broadly applicable, and we illustrate this through two very different applications. In the first application, we focus on receivers in a network who take binary actions (that can represent, e.g., adoption of a product of unknown quality), and characterize the optimal mechanism in terms of the network’s cores. Interestingly, as a consequence of the double-interval structure, the set of states for which a receiver takes action 1 is in general nonconvex. We show that in large random networks the designer can obtain asymptotically optimal mechanisms even in the absence of detailed network information, by using only the degree distribution. In our second application, we focus on persuading a single receiver whose type is private. Using a slight modification of our framework, we establish that the optimal mechanism is based on a laminar interval structure – a generalization of the double-interval structure – where the number of subintervals associated with each signal realization increases with the number of types.

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

In machine learning research, many emerging applications can be (re)formulated as the composition optimization problem with nonsmooth regularization penalty. To solve this problem, traditional stochastic gradient descent (SGD) algorithm and its variants either have low convergence rate or are computationally expensive. Recently, several stochastic composition gradient algorithms have been proposed, however, these methods are still inefficient and not scalable to large-scale composition optimization problem instances. To address these challenges, we propose an asynchronous parallel algorithm, named Async-ProxSCVR, which effectively combines asynchronous parallel implementation and variance reduction method. We prove that the algorithm admits the fastest convergence rate for both strongly convex and general nonconvex cases. Furthermore, we analyze the query complexity of the proposed algorithm and prove that linear speedup is accessible when we increase the number of processors. Finally, we evaluate our algorithm Async-ProxSCVR on two representative composition optimization problems including value function evaluation in reinforcement learning and sparse mean-variance optimization problem. Experimental results show that the algorithm achieves significant speedups and is much faster than existing compared methods.