Large-scale Nonconvex Optimization Research Articles

A New Hybrid Descent Algorithm for Large-Scale Nonconvex Optimization and Application to Some Image Restoration Problems

Conjugate gradient methods are widely used and attractive for large-scale unconstrained smooth optimization problems, with simple computation, low memory requirements, and interesting theoretical information on the features of curvature. Based on the strongly convergent property of the Dai–Yuan method and attractive numerical performance of the Hestenes–Stiefel method, a new hybrid descent conjugate gradient method is proposed in this paper. The proposed method satisfies the sufficient descent property independent of the accuracy of the line search strategies. Under the standard conditions, the trust region property and the global convergence are established, respectively. Numerical results of 61 problems with 9 large-scale dimensions and 46 ill-conditioned matrix problems reveal that the proposed method is more effective, robust, and reliable than the other methods. Additionally, the hybrid method also demonstrates reliable results for some image restoration problems.

A Quasi-Newton Matrix Factorization-Based Model for Recommendation

Solving large-scale non-convex optimization problems is the fundamental challenge in the development of matrix factorization (MF)-based recommender systems. Unfortunately, employing conventional first-order optimization approaches proves to be an arduous endeavor since their curves are very complex. The exploration of second-order optimization methods holds great promise. They are more powerful because they consider the curvature of the optimization problem, which is captured by the second-order derivatives of the objective function. However, a significant obstacle arises when directly applying Hessian-based approaches: their computational demands are often prohibitively high. Therefore, the authors propose AdaGO, a novel quasi-Newton method-based optimizer to meet the specific requirements of large-scale non-convex optimization problems. AdaGO can strike a balance between computational efficiency and optimization performance. In the comparative studies with state-of-the-art MF-based models, AdaGO demonstrates its superiority by achieving higher prediction accuracy.

Large-Scale Nonconvex Optimization: Randomization, Gap Estimation, and Numerical Resolution

Large-Scale Nonconvex Optimization: Randomization, Gap Estimation, and Numerical Resolution

Faster Riemannian Newton-type optimization by subsampling and cubic regularization

This work is on constrained large-scale non-convex optimization where the constraint set implies a manifold structure. Solving such problems is important in a multitude of fundamental machine learning tasks. Recent advances on Riemannian optimization have enabled the convenient recovery of solutions by adapting unconstrained optimization algorithms over manifolds. However, it remains challenging to scale up and meanwhile maintain stable convergence rates and handle saddle points. We propose a new second-order Riemannian optimization algorithm, aiming at improving convergence rate and reducing computational cost. It enhances the Riemannian trust-region algorithm that explores curvature information to escape saddle points through a mixture of subsampling and cubic regularization techniques. We conduct rigorous analysis to study the convergence behavior of the proposed algorithm. We also perform extensive experiments to evaluate it based on two general machine learning tasks using multiple datasets. The proposed algorithm exhibits improved computational speed, e.g., a speed improvement from 12% :text {to} :227%, and improved convergence behavior, e.g., an iteration number reduction from mathcal{O}left(maxleft(epsilon_g^{-2}epsilon_H^{-1},epsilon_H^{-3}right)right) ,text {to}: mathcal{O}left(maxleft(epsilon_g^{-2},epsilon_H^{-3}right)right), compared to a large set of state-of-the-art Riemannian optimization algorithms.

A Momentum-Accelerated Hessian-Vector-Based Latent Factor Analysis Model

Service-oriented applications commonly involve high-dimensional and sparse (HiDS) interactions among users and service-related entities, e.g., user-item interactions from a personalized recommendation services system. How to perform precise and efficient representation learning on such HiDS interactions data is a hot yet thorny issue. An efficient approach to it is latent factor analysis (LFA), which commonly depends on large-scale non-convex optimization. Hence, it is vital to implement an LFA model able to approximate second-order stationary points efficiently for enhancing its representation learning ability. However, existing second-order LFA models suffer from high computational cost, which significantly reduces its practicability. To address this issue, this paper presents a Momentum-accelerated Hessian-vector algorithm (MH) for precise and efficient LFA on HiDS data. Its main ideas are two-fold: a) adopting the principle of a Hessian-vector-product-based method to utilize the second-order information without manipulating a Hessian matrix directly, and b) incorporating a generalized momentum method into its parameter learning scheme for accelerating its convergence rate to a stationary point. Experimental results on nine industrial datasets demonstrate that compared with state-of-the-art LFA models, an MH-based LFA model achieves gains in both accuracy and convergence rate. These positive outcomes also indicate that a generalized momentum method is compatible with the algorithms, e.g., a second-order algorithm, which implicitly rely on gradients.

A new multipoint symmetric secant method with a dense initial matrix

In large-scale optimization, when either forming or storing Hessian matrices are prohibitively expensive, quasi-Newton methods are often used in lieu of Newton's method because they only require first-order information to approximate the true Hessian. Multipoint symmetric secant (MSS) methods can be thought of as generalizations of quasi-Newton methods in that they attempt to impose additional requirements on their approximation of the Hessian. Given an initial Hessian approximation, MSS methods generate a sequence of possibly-indefinite matrices using rank-2 updates to solve nonconvex unconstrained optimization problems. For practical reasons, up to now, the initialization has been a constant multiple of the identity matrix. In this paper, we propose a new limited-memory MSS method for large-scale nonconvex optimization that allows for dense initializations. Numerical results on the CUTEst test problems suggest that the MSS method using a dense initialization outperforms the standard initialization. Numerical results also suggest that this approach is competitive with both a basic L-SR1 trust-region method and an L-PSB method.

Decentralized nonconvex optimization with guaranteed privacy and accuracy

Privacy protection and nonconvexity are two challenging problems in decentralized optimization and learning involving sensitive data. Despite some recent advances addressing each of the two problems separately, no results have been reported that have theoretical guarantees on both privacy protection and saddle/maximum avoidance in decentralized nonconvex optimization. We propose a new algorithm for decentralized nonconvex optimization that can enable both rigorous differential privacy and saddle/maximum avoiding performance. The new algorithm allows the incorporation of persistent additive noise to enable rigorous differential privacy for data samples, gradients, and intermediate optimization variables without losing provable convergence, and thus circumventing the dilemma of trading accuracy for privacy in differential privacy design. More interestingly, the algorithm is theoretically proven to be able to efficiently guarantee accuracy by avoiding convergence to local maxima and saddle points, which has not been reported before in the literature on decentralized nonconvex optimization. The algorithm is efficient in both communication (it only shares one variable in each iteration) and computation (it is encryption-free), and hence is promising for large-scale nonconvex optimization and learning involving high-dimensional optimization parameters. Numerical experiments for both a decentralized estimation problem and an Independent Component Analysis (ICA) problem confirm the effectiveness of the proposed approach.

MINRES: From Negative Curvature Detection to Monotonicity Properties

The conjugate gradient method (CG) has long been the workhorse for inner-iterations of second-order algorithms for large-scale nonconvex optimization. Prominent examples include line-search based algorithms, e.g., Newton-CG, and those based on a trust-region framework, e.g., CG-Steihaug. This is mainly thanks to CG's several favorable properties, including certain monotonicity properties and its inherent ability to detect negative curvature directions, which can arise in nonconvex optimization. This is despite the fact that the iterative method-of-choice when it comes to real symmetric but potentially indefinite matrices is arguably the celebrated minimal residual (MINRES) method. However, limited understanding of similar properties implied by MINRES in such settings has restricted its applicability within nonconvex optimization algorithms. We establish several such nontrivial properties of MINRES, including certain useful monotonicity as well as an inherent ability to detect negative curvature directions. These properties allow MINRES to be considered as a potentially superior alternative to CG for all Newton-type nonconvex optimization algorithms that employ CG as their subproblem solver.

A hybrid robust-stochastic optimization approach for day-ahead scheduling of cascaded hydroelectric system in restructured electricity market

Uncertainties arising from complicated natural and market environments pose great challenges for the efficient operation of cascaded hydroelectric systems. To overcome these challenges, this paper studies the day-ahead scheduling of cascaded hydroelectric systems in a restructured electricity market with the presence of uncertainties in electricity price and natural water inflow. To properly model the uncertainty, we consider the unique characteristics of these two types of uncertainties and capture them via the uncertainty set and stochastic scenarios, respectively. A hybrid robust-stochastic optimization model is developed to simultaneously hedge against these two types of uncertainties, which is formulated as a large-scale non-convex optimization problem with mixed integer recourse. After introducing linearization of nonlinear terms, a tailored hybrid decomposition scheme combining Lagrangian relaxation and Dantzig-Wolfe decomposition is adopted to achieve efficient computation of the proposed model. Two real-world cases are conducted to demonstrate the capability and characteristics of the proposed model and algorithms.

A Stochastic Momentum Accelerated Quasi-Newton Method for Neural Networks (Student Abstract)

Incorporating curvature information in stochastic methods has been a challenging task. This paper proposes a momentum accelerated BFGS quasi-Newton method in both its full and limited memory forms, for solving stochastic large scale non-convex optimization problems in neural networks (NN).

DeepSplit: Scalable Verification of Deep Neural Networks via Operator Splitting

Analyzing the worst-case performance of deep neural networks against input perturbations amounts to solving a large-scale non-convex optimization problem, for which several past works have proposed convex relaxations as a promising alternative. However, even for reasonably-sized neural networks, these relaxations are not tractable, and so must be replaced by even weaker relaxations in practice. In this work, we propose a novel operator splitting method that can directly solve a convex relaxation of the problem to high accuracy, by splitting it into smaller sub-problems that often have analytical solutions. The method is modular, scales to very large problem instances, and compromises operations that are amenable to fast parallelization with GPU acceleration. We demonstrate our method in bounding the worst-case performance of large convolutional networks in image classification and reinforcement learning settings, and in reachability analysis of neural network dynamical systems.

MIMO-OFDM-Based Wireless-Powered Relaying Communication With an Energy Recycling Interface

This paper considers wireless-powered relaying multiple-input-multiple-output (MIMO) communication, where all four nodes (information source, energy source, relay, and destination) are equipped with multiple antennas. Orthogonal frequency division multiplexing (OFDM) is applied for information processing to compensate the frequency selectivity of communication channels between the information source and the relay and between the relay and the destination as these nodes are assumed to be located far apart from each. The relay is equipped with a full-duplexing interface for harvesting energy not only from the wireless transmission of the dedicated energy source but also from its own transmission while relaying the source information to the destination. The problem of designing the optimal power allocation over OFDM subcarriers and transmit antennas to maximize the overall spectral efficiency is addressed. Due to a very large number of subcarriers, this design problem poses a large-scale nonconvex optimization problem involving a few thousand variables of power allocation, which is very computationally challenging. A novel path-following algorithm is proposed for computation. Based on the developed closed-form calculation of linear computational complexity at each iteration, the proposed algorithm rapidly converges to an optimal solution. Compared to the best existing solvers, the computational complexity of the proposed algorithm is reduced at least 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">5</sup> times, making it very efficient and practical for online computation while existing solvers are ineffective. Numerical results for a practical simulation setting show promising results by achieving high spectral efficiency.

A Class of Approximate Inverse Preconditioners Based on Krylov-Subspace Methods for Large-Scale Nonconvex Optimization

We introduce a class of positive definite preconditioners for the solution of large symmetric indefinite linear systems or sequences of such systems, in optimization frameworks. The preconditioners are iteratively constructed by collecting information on a reduced eigenspace of the indefinite matrix by means of a Krylov-subspace solver. A spectral analysis of the preconditioned matrix shows the clustering of some eigenvalues and possibly the nonexpansion of its spectrum. Extensive numerical experimentation is carried out on standard difficult linear systems and by embedding the class of preconditioners within truncated Newton methods for large-scale unconstrained optimization (the issue of major interest). Although the Krylov-based method may provide modest information on matrix eigenspaces, the results obtained show that the proposed preconditioners lead to substantial improvements in terms of efficiency and robustness, particularly on very large nonconvex problems.

Introducing Spatial Regularization in SAR Tomography Reconstruction

The resolution achieved by current synthetic aperture radar (SAR) sensors provides a detailed visualization of urban areas. Spaceborne sensors such as TerraSAR-X can be used to analyze large areas at a very high resolution. In addition, repeated passes of the satellite give access to temporal and interferometric information on the scene. Because of the complex 3-D structure of urban surfaces, scatterers located at different heights (ground, building facade, and roof) produce radar echoes that often get mixed within the same radar cells. These echoes must be numerically unmixed in order to get a fine understanding of the radar images. This unmixing is at the core of SAR tomography. SAR tomography reconstruction is generally performed in two steps: 1) reconstruction of the so-called tomogram by vertical focusing, at each radar resolution cell, to extract the complex amplitudes (a 1-D processing) and 2) transformation from radar geometry to ground geometry and extraction of significant scatterers. We propose to perform the tomographic inversion directly in ground geometry in order to enforce spatial regularity in 3-D space. This inversion requires solving a large-scale nonconvex optimization problem. We describe an iterative method based on variable splitting and the augmented Lagrangian technique. Spatial regularizations can easily be included in this generic scheme. We illustrate, on simulated data and a TerraSAR-X tomographic data set, the potential of this approach to produce 3-D reconstructions of urban surfaces.

Integrating DVH criteria into a column generation algorithm for VMAT treatment planning

Volumetric-modulated arc therapy (VMAT) treatment planning is an efficient treatment technique with a high degree of flexibility in terms of dose rate, gantry speed, and aperture shapes during rotation around the patient. However, the dynamic nature of VMAT results in a large-scale nonconvex optimization problem. Determining the priority of the tissues and voxels to obtain clinically acceptable treatment plans poses additional challenges for VMAT optimization. The main purpose of this paper is to develop an automatic planning approach integrating dose-volume histogram (DVH) criteria in direct aperture optimization for VMAT, by adjusting the model parameters during the algorithm.The proposed algorithm is based on column generation, an optimization technique that sequentially generates the apertures and optimizes the corresponding intensities. We take the advantage of iterative procedure in this method to modify the weight vector of the penalty function based on the DVH criteria and decrease the use of trial-and-error in the search for clinically acceptable plans.We evaluate the efficiency of the algorithm and treatment quality using a clinical prostate case and a challenging head-and-neck case. In both cases, we generate 15 random initial weight vectors to assess the robustness of the algorithm. In the prostate case, our methodology obtained clinically acceptable plans in all instances with only a 10% increase in the computational time, while simple VMAT optimization found just three acceptable plans. To have an idea with respect to the existing software, we compared the obtained DVH to a commercial software. The quality of the diagrams of the proposed method, especially for the healthy tissues, is significantly better while the computational time is less. In the head-and-neck case, 93.3% of the clinically acceptable plans are obtained while no plan was acceptable in simple VMAT.In sum, the results demonstrate the ability of the proposed optimization algorithm to obtain clinically acceptable plans without human intervention and also its robustness to weight parameters. Moreover, our proposed weight adjustment procedure proves to reduce the symmetry in the solution space and the time required for the post-optimization phase.

A Distributed Primal Decomposition Scheme for Nonconvex Optimization

In this paper, we deal with large-scale nonconvex optimization problems, typically arising in distributed nonlinear optimal control, that must be solved by agents in a network. Each agent is equipped with a local cost function, depending only on a local variable. The variables must satisfy private nonconvex constraints and global coupling constraints. We propose a distributed algorithm for the fast computation of a feasible solution of the nonconvex problem in finite time, through a distributed primal decomposition framework. The method exploits the solution of a convexified version of the problem, with restricted coupling constraints, to compute a feasible solution of the original problem. Numerical computations corroborate the results.

An adaptive truncation criterion, for linesearch-based truncated Newton methods in large scale nonconvex optimization

Starting from the paper by Nash and Sofer (1990), we propose a heuristic adaptive truncation criterion for the inner iterations within linesearch-based truncated Newton methods. Our aim is to possibly avoid “over-solving” of the Newton equation, based on a comparison between the predicted reduction of the objective function and the actual reduction obtained. A numerical experience on unconstrained optimization problems highlights a satisfactory effectiveness and robustness of the adaptive criterion proposed, when a residual-based truncation criterion is selected.

Joint power allocation for MIMO-OFDM full-duplex relaying communications

In this paper, we address the problem of joint power allocation in a two-hop MIMO-OFDM network, where two full-duplex users communicate with each other via an amplify-and-forward relay. We consider a general model in which the full-duplex relay can forward the received message in either one-way or two-way mode. Our aim is to maximize the instantaneous end-to-end total throughput, subject to (i) the separate sum-power constraints at individual nodes or (ii) the joint sum-power constraint of the whole network. The formulated problems are large-scale nonconvex optimization problems, for which efficient and optimal solutions are currently not available. Using the successive convex approximation approach, we develop novel iterative algorithms of extremely low complexity which are especially suitable for large-scale computation. In each iteration, a simple closed-form solution is derived for the approximated convex program. The proposed algorithms guarantee to converge to at least a local optimum of the nonconvex problems. Numerical results verify that the devised solutions converge quickly, and that our optimal power allocation schemes significantly improve the throughput of MIMO-OFDM full-duplex one-way/two-way relaying over the conventional half-duplex relaying strategy.

Sequential Convex Programming for Computing Information-Theoretic Minimal Partitions: Nonconvex Nonsmooth Optimization

We consider an unsupervised image segmentation problem---from figure-ground separation to multiregion partitioning---that consists of maximal distribution separation (in terms of mutual information) with spatial regularity (total variation regularization), which is what we call information-theoretic minimal partitioning. Adopting the bounded variation framework, we provide an in-depth analysis of the problem which establishes theoretical foundations and investigates the structure of the associated energy from a variational perspective. In doing so, we show that the objective exhibits a form of difference of convex functionals, which leads us to a class of large-scale nonconvex optimization problems where convex optimization techniques can be successfully applied. In this regard, we propose sequential convex programming based on the philosophy of stochastic optimization and the Chambolle--Pock primal-dual algorithm. The key idea behind it is to construct a stochastic family of convex approximations of the original nonconvex function and sequentially minimize the associated subproblems. Indeed, its stochastic nature makes it possible to often escape from bad local minima toward near-optimal solutions, where such optimality can be justified in terms of the recent findings in statistical physics regarding a striking characteristic of the high-dimensional landscapes. We experimentally demonstrate such a favorable ability of the proposed algorithm as well as show the capacity of our approach in numerous experiments. The preliminary conference paper can be found in [Y. Kee, M. Souiai, D. Cremers, and J. Kim, Proceedings of the IEEE Conference on Computer Vision and Pattern, Recognition, 2014].

Convex Relaxation of Optimal Power Flow in Distribution Feeders with Embedded Solar Power

There is an increasing interest in using Distributed Energy Resources (DER) directly coupled to end user distribution feeders. This poses an array of challenges because most of today's distribution feeders are designed for unidirectional power flow. Therefore when installing DERs such as solar panels with uncontrolled inverters, the upper limit of installable capacity is quickly reached in many of today's distribution feeders. This problem can often be mitigated by optimally controlling the voltage angles of inverters. However, the optimal power flow problem in its standard form is a large scale non-convex optimization problem, and thus can’t be solved precisely and also is computationally heavy and intractable for large systems. This paper examines the use of a convex relaxation using Semi-definite programming to optimally control solar power inverters in a distribution grid in order to minimize the global line losses of the feeder. The mathematical model is presented in details. Further, case studies are completed with simulations involving a 15-bus radial distribution system. These simulations are run for 24 hour periods, with actual solar data and demand data.