Large-scale Nonconvex Optimization Problems Research Articles

A Quasi-Newton Matrix Factorization-Based Model for Recommendation

Solving large-scale non-convex optimization problems is the fundamental challenge in the development of matrix factorization (MF)-based recommender systems. Unfortunately, employing conventional first-order optimization approaches proves to be an arduous endeavor since their curves are very complex. The exploration of second-order optimization methods holds great promise. They are more powerful because they consider the curvature of the optimization problem, which is captured by the second-order derivatives of the objective function. However, a significant obstacle arises when directly applying Hessian-based approaches: their computational demands are often prohibitively high. Therefore, the authors propose AdaGO, a novel quasi-Newton method-based optimizer to meet the specific requirements of large-scale non-convex optimization problems. AdaGO can strike a balance between computational efficiency and optimization performance. In the comparative studies with state-of-the-art MF-based models, AdaGO demonstrates its superiority by achieving higher prediction accuracy.

A hybrid robust-stochastic optimization approach for day-ahead scheduling of cascaded hydroelectric system in restructured electricity market

Uncertainties arising from complicated natural and market environments pose great challenges for the efficient operation of cascaded hydroelectric systems. To overcome these challenges, this paper studies the day-ahead scheduling of cascaded hydroelectric systems in a restructured electricity market with the presence of uncertainties in electricity price and natural water inflow. To properly model the uncertainty, we consider the unique characteristics of these two types of uncertainties and capture them via the uncertainty set and stochastic scenarios, respectively. A hybrid robust-stochastic optimization model is developed to simultaneously hedge against these two types of uncertainties, which is formulated as a large-scale non-convex optimization problem with mixed integer recourse. After introducing linearization of nonlinear terms, a tailored hybrid decomposition scheme combining Lagrangian relaxation and Dantzig-Wolfe decomposition is adopted to achieve efficient computation of the proposed model. Two real-world cases are conducted to demonstrate the capability and characteristics of the proposed model and algorithms.

A Stochastic Momentum Accelerated Quasi-Newton Method for Neural Networks (Student Abstract)

Incorporating curvature information in stochastic methods has been a challenging task. This paper proposes a momentum accelerated BFGS quasi-Newton method in both its full and limited memory forms, for solving stochastic large scale non-convex optimization problems in neural networks (NN).

DeepSplit: Scalable Verification of Deep Neural Networks via Operator Splitting

Analyzing the worst-case performance of deep neural networks against input perturbations amounts to solving a large-scale non-convex optimization problem, for which several past works have proposed convex relaxations as a promising alternative. However, even for reasonably-sized neural networks, these relaxations are not tractable, and so must be replaced by even weaker relaxations in practice. In this work, we propose a novel operator splitting method that can directly solve a convex relaxation of the problem to high accuracy, by splitting it into smaller sub-problems that often have analytical solutions. The method is modular, scales to very large problem instances, and compromises operations that are amenable to fast parallelization with GPU acceleration. We demonstrate our method in bounding the worst-case performance of large convolutional networks in image classification and reinforcement learning settings, and in reachability analysis of neural network dynamical systems.

MIMO-OFDM-Based Wireless-Powered Relaying Communication With an Energy Recycling Interface

This paper considers wireless-powered relaying multiple-input-multiple-output (MIMO) communication, where all four nodes (information source, energy source, relay, and destination) are equipped with multiple antennas. Orthogonal frequency division multiplexing (OFDM) is applied for information processing to compensate the frequency selectivity of communication channels between the information source and the relay and between the relay and the destination as these nodes are assumed to be located far apart from each. The relay is equipped with a full-duplexing interface for harvesting energy not only from the wireless transmission of the dedicated energy source but also from its own transmission while relaying the source information to the destination. The problem of designing the optimal power allocation over OFDM subcarriers and transmit antennas to maximize the overall spectral efficiency is addressed. Due to a very large number of subcarriers, this design problem poses a large-scale nonconvex optimization problem involving a few thousand variables of power allocation, which is very computationally challenging. A novel path-following algorithm is proposed for computation. Based on the developed closed-form calculation of linear computational complexity at each iteration, the proposed algorithm rapidly converges to an optimal solution. Compared to the best existing solvers, the computational complexity of the proposed algorithm is reduced at least 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">5</sup> times, making it very efficient and practical for online computation while existing solvers are ineffective. Numerical results for a practical simulation setting show promising results by achieving high spectral efficiency.

Introducing Spatial Regularization in SAR Tomography Reconstruction

The resolution achieved by current synthetic aperture radar (SAR) sensors provides a detailed visualization of urban areas. Spaceborne sensors such as TerraSAR-X can be used to analyze large areas at a very high resolution. In addition, repeated passes of the satellite give access to temporal and interferometric information on the scene. Because of the complex 3-D structure of urban surfaces, scatterers located at different heights (ground, building facade, and roof) produce radar echoes that often get mixed within the same radar cells. These echoes must be numerically unmixed in order to get a fine understanding of the radar images. This unmixing is at the core of SAR tomography. SAR tomography reconstruction is generally performed in two steps: 1) reconstruction of the so-called tomogram by vertical focusing, at each radar resolution cell, to extract the complex amplitudes (a 1-D processing) and 2) transformation from radar geometry to ground geometry and extraction of significant scatterers. We propose to perform the tomographic inversion directly in ground geometry in order to enforce spatial regularity in 3-D space. This inversion requires solving a large-scale nonconvex optimization problem. We describe an iterative method based on variable splitting and the augmented Lagrangian technique. Spatial regularizations can easily be included in this generic scheme. We illustrate, on simulated data and a TerraSAR-X tomographic data set, the potential of this approach to produce 3-D reconstructions of urban surfaces.

Integrating DVH criteria into a column generation algorithm for VMAT treatment planning

Volumetric-modulated arc therapy (VMAT) treatment planning is an efficient treatment technique with a high degree of flexibility in terms of dose rate, gantry speed, and aperture shapes during rotation around the patient. However, the dynamic nature of VMAT results in a large-scale nonconvex optimization problem. Determining the priority of the tissues and voxels to obtain clinically acceptable treatment plans poses additional challenges for VMAT optimization. The main purpose of this paper is to develop an automatic planning approach integrating dose-volume histogram (DVH) criteria in direct aperture optimization for VMAT, by adjusting the model parameters during the algorithm.The proposed algorithm is based on column generation, an optimization technique that sequentially generates the apertures and optimizes the corresponding intensities. We take the advantage of iterative procedure in this method to modify the weight vector of the penalty function based on the DVH criteria and decrease the use of trial-and-error in the search for clinically acceptable plans.We evaluate the efficiency of the algorithm and treatment quality using a clinical prostate case and a challenging head-and-neck case. In both cases, we generate 15 random initial weight vectors to assess the robustness of the algorithm. In the prostate case, our methodology obtained clinically acceptable plans in all instances with only a 10% increase in the computational time, while simple VMAT optimization found just three acceptable plans. To have an idea with respect to the existing software, we compared the obtained DVH to a commercial software. The quality of the diagrams of the proposed method, especially for the healthy tissues, is significantly better while the computational time is less. In the head-and-neck case, 93.3% of the clinically acceptable plans are obtained while no plan was acceptable in simple VMAT.In sum, the results demonstrate the ability of the proposed optimization algorithm to obtain clinically acceptable plans without human intervention and also its robustness to weight parameters. Moreover, our proposed weight adjustment procedure proves to reduce the symmetry in the solution space and the time required for the post-optimization phase.

A Distributed Primal Decomposition Scheme for Nonconvex Optimization

In this paper, we deal with large-scale nonconvex optimization problems, typically arising in distributed nonlinear optimal control, that must be solved by agents in a network. Each agent is equipped with a local cost function, depending only on a local variable. The variables must satisfy private nonconvex constraints and global coupling constraints. We propose a distributed algorithm for the fast computation of a feasible solution of the nonconvex problem in finite time, through a distributed primal decomposition framework. The method exploits the solution of a convexified version of the problem, with restricted coupling constraints, to compute a feasible solution of the original problem. Numerical computations corroborate the results.

Joint power allocation for MIMO-OFDM full-duplex relaying communications

In this paper, we address the problem of joint power allocation in a two-hop MIMO-OFDM network, where two full-duplex users communicate with each other via an amplify-and-forward relay. We consider a general model in which the full-duplex relay can forward the received message in either one-way or two-way mode. Our aim is to maximize the instantaneous end-to-end total throughput, subject to (i) the separate sum-power constraints at individual nodes or (ii) the joint sum-power constraint of the whole network. The formulated problems are large-scale nonconvex optimization problems, for which efficient and optimal solutions are currently not available. Using the successive convex approximation approach, we develop novel iterative algorithms of extremely low complexity which are especially suitable for large-scale computation. In each iteration, a simple closed-form solution is derived for the approximated convex program. The proposed algorithms guarantee to converge to at least a local optimum of the nonconvex problems. Numerical results verify that the devised solutions converge quickly, and that our optimal power allocation schemes significantly improve the throughput of MIMO-OFDM full-duplex one-way/two-way relaying over the conventional half-duplex relaying strategy.

Sequential Convex Programming for Computing Information-Theoretic Minimal Partitions: Nonconvex Nonsmooth Optimization

We consider an unsupervised image segmentation problem---from figure-ground separation to multiregion partitioning---that consists of maximal distribution separation (in terms of mutual information) with spatial regularity (total variation regularization), which is what we call information-theoretic minimal partitioning. Adopting the bounded variation framework, we provide an in-depth analysis of the problem which establishes theoretical foundations and investigates the structure of the associated energy from a variational perspective. In doing so, we show that the objective exhibits a form of difference of convex functionals, which leads us to a class of large-scale nonconvex optimization problems where convex optimization techniques can be successfully applied. In this regard, we propose sequential convex programming based on the philosophy of stochastic optimization and the Chambolle--Pock primal-dual algorithm. The key idea behind it is to construct a stochastic family of convex approximations of the original nonconvex function and sequentially minimize the associated subproblems. Indeed, its stochastic nature makes it possible to often escape from bad local minima toward near-optimal solutions, where such optimality can be justified in terms of the recent findings in statistical physics regarding a striking characteristic of the high-dimensional landscapes. We experimentally demonstrate such a favorable ability of the proposed algorithm as well as show the capacity of our approach in numerous experiments. The preliminary conference paper can be found in [Y. Kee, M. Souiai, D. Cremers, and J. Kim, Proceedings of the IEEE Conference on Computer Vision and Pattern, Recognition, 2014].

Convex Relaxation of Optimal Power Flow in Distribution Feeders with Embedded Solar Power

There is an increasing interest in using Distributed Energy Resources (DER) directly coupled to end user distribution feeders. This poses an array of challenges because most of today's distribution feeders are designed for unidirectional power flow. Therefore when installing DERs such as solar panels with uncontrolled inverters, the upper limit of installable capacity is quickly reached in many of today's distribution feeders. This problem can often be mitigated by optimally controlling the voltage angles of inverters. However, the optimal power flow problem in its standard form is a large scale non-convex optimization problem, and thus can’t be solved precisely and also is computationally heavy and intractable for large systems. This paper examines the use of a convex relaxation using Semi-definite programming to optimally control solar power inverters in a distribution grid in order to minimize the global line losses of the feeder. The mathematical model is presented in details. Further, case studies are completed with simulations involving a 15-bus radial distribution system. These simulations are run for 24 hour periods, with actual solar data and demand data.

Some Efficient Optimization Methods for Solving the Security-Constrained Optimal Power Flow Problem

The security-constrained optimal power flow problem considers both the normal state and contingency constraints, and it is formulated as a large-scale nonconvex optimization problem. We propose a global optimization algorithm based on Lagrangian duality to solve the nonconvex problem to optimality. As usual, the global approach is often time-consuming, thus, for practical uses when dealing with a large number of contingencies, we investigate two decomposition algorithms based on Benders cut and the alternating direction method of multipliers. These decomposition schemes often generate solutions with a smaller objective function values than those generated by the conventional approach and very close to the globally optimal points.

Bandgap optimization of two-dimensional photonic crystals using semidefinite programming and subspace methods

In this paper, we consider the optimal design of photonic crystal structures for two-dimensional square lattices. The mathematical formulation of the bandgap optimization problem leads to an infinite-dimensional Hermitian eigenvalue optimization problem parametrized by the dielectric material and the wave vector. To make the problem tractable, the original eigenvalue problem is discretized using the finite element method into a series of finite-dimensional eigenvalue problems for multiple values of the wave vector parameter. The resulting optimization problem is large-scale and non-convex, with low regularity and non-differentiable objective. By restricting to appropriate eigenspaces, we reduce the large-scale non-convex optimization problem via reparametrization to a sequence of small-scale convex semidefinite programs (SDPs) for which modern SDP solvers can be efficiently applied. Numerical results are presented for both transverse magnetic (TM) and transverse electric (TE) polarizations at several frequency bands. The optimized structures exhibit patterns which go far beyond typical physical intuition on periodic media design.

Global Optimization in Multiproduct and Multipurpose Batch Design under Uncertainty

This paper addresses the design of multiproduct and multipurpose batch plants with uncertainty in both product demands and processing parameters. The uncertain demands may be described by any continuous/discrete probability distribution. Uncertain processing parameters are handled in a scenario-based approach. Through the relaxation of the feasibility requirement, the design problem with a fixed number of pieces of equipment per stage is formulated as a single large-scale nonconvex optimization problem. This problem is solved using a branch and bound technique in which a convex relaxation of the original nonconvex problem is solved to provide a lower bound on the global solution. Several different expressions for the tight convex lower bounding functions are proposed. Using these expressions, a tight lower bound on the global optimum solution can be obtained at each iteration. The αBB algorithm is subsequently employed to refine the upper and lower bounds and converge to the global solution. The tight lower bounds and the efficiency of the proposed approach is demonstrated in several example problems. These case studies correspond to large-scale global optimization problems with nonconvex constraints ranging in number from 25 to 3750, variables ranging from 30 to 15 636 and nonconvex terms ranging from 50 to 15 000. It is shown that such large-scale multiproduct and multipurpose batch design problems can be solved to global optimality with reasonable computational effort.

Two-level primal-dual decomposition technique for large-scale nonconvex optimization problems with constraints

A two-level decomposition method for nonconvex separable optimization problems with additional local constraints of general inequality type is presented and thoroughly analyzed in the paper. The method is of primal-dual type, based on an augmentation of the Lagrange function. Previous methods of this type were in fact three-level, with adjustment of the Lagrange multipliers at one of the levels. This level is eliminated in the present approach by replacing the multipliers by a formula depending only on primal variables and Kuhn-Tucker multipliers for the local constraints. The primal variables and the Kuhn-Tucker multipliers are together the higher-level variables, which are updated simultaneously. Algorithms for this updating are proposed in the paper, together with their convergence analysis, which gives also indications on how to choose penalty coefficients of the augmented Lagrangian. Finally, numerical examples are presented.

New dual-type decomposition algorithm for nonconvex separable optimization problems

The aim of the paper is to present a new dual-type decomposition algorithm for large-scale nonconvex optimization problems of general separable structure. After reformulation of the augmented Lagrange function and introduction of auxiliary variables, called approximation points, separable local optimization problems are created. The lower level of the method consists of a few optimization runs of these problems for subsequently improved approximation points, whereas standard updating of Lagrange multipliers constitutes the highest level. Simple rules for adjusting the approximation points and the Lagrange multipliers are given and thoroughly analysed. Applicability conditions and example numerical results indicate that the presented algorithm eliminates, to a great extent, drawbacks of the previous approaches.

New Lagrangian function for nonconvex primal-dual decomposition

In this paper, a new Lagrangian function is reported which is particularly suited for large-scale nonconvex optimization problems with separable structure. Our modification convexifies the standard Lagrangian function without destroying its separable structure so that the primal-dual decomposition technique can be applied even to nonconvex optimization problems. Furthermore, the proposed Lagrangian results in two levels of iterative optimization as compared with the three levels needed for techniques recently proposed for nonconvex primal-dual decomposition.