Smooth Objective Function Research Articles

Distributed Anytime-Feasible Resource Allocation Subject to Heterogeneous Time-Varying Delays

This paper considers distributed allocation strategies, formulated as a distributed sum-preserving (fixed-sum) allocation of resources over a multi-agent network in the presence of heterogeneous arbitrary time-varying delays. We propose a double time-scale scenario for unknown delays and a faster single time-scale scenario for known delays. Further, the links among the nodes are considered subject to certain nonlinearities (e.g, quantization and saturation/clipping). We discuss different models for nonlinearities and how they may affect the convergence, sum-preserving feasibility constraint, and solution optimality over general weight-balanced uniformly strongly connected networks and, further, time-delayed undirected networks. Our proposed scheme works in a variety of applications with general non-quadratic strongly-convex smooth objective functions. The non-quadratic part, for example, can be due to additive convex penalty or barrier functions to address the local box constraints. The network can change over time, is not necessarily connected at all times, but is only assumed to be uniformly-connected. The novelty of this work is to address all-time feasible Laplacian gradient solutions in presence of nonlinearities, switching digraph topology (not necessarily all-time connected), and heterogeneous time-varying delays.

Про пришвидшення оптимізаційних методів для задачі синтезу багатошарових оптичних покриттів

Розглянуто п’ять способів для пришвидшення багатовимірного пошуку розв’язку задачі синтезу багатошарових оптичних покриттів за допомогою методів нульового та першого порядків. Перший спосіб — це використання аналітичної похідної для цільової функції якості багатошарового покриття. Він дозволяє точно (у межах комп’ютерної арифметики) обчислити значення градієнта гладкої цільової функції та узагальненого градієнта негладкої цільової функції. Перший спосіб потребує таку ж кількість арифметичних операцій, як і скінченно-різницеві способи обчислення градієнта та узагальненого градієнта. Другий спосіб — це використання пришвидшеного знаходження градієнта цільової функції за допомогою використання префікс- та суфікс-масивів у аналітичному способі обчислення градієнта. Цей прийом дозволяє зменшити кількість арифметичних операцій втричі для задач великої розмірності. Третій спосіб — це використання табуляції значень тригонометричних функцій для обчислення характеристичних матриць. Цей прийом зменшує час виконання операцій множення характеристичних матриць у десятки разів в залежності від характеристик комп’ютера. Для деяких архітектур комп’ютера ця перевага становить більше ніж 140 разів. Четвертий спосіб — це використання методу золотого перерізу для одновимірної оптимізації в задачах синтезу оптичних покриттів. Зокрема, при розв’язанні однієї часткової задачі показано, що метод тернарного пошуку потребує приблизно на 40 % більше часових затрат, ніж метод золотого перерізу. П’ятий спосіб — це використання ефективної реалізації множення двох матриць. Вона полягає у зміні порядку другого і третього циклів для загальновідомого методу множення двох матриць та фіксації у звичайній змінній значення елемента першої матриці. Це дозволяє суттєво прискорити виконання операції множення двох матриць. Для матриць розмірності 1000×1000 придшвидшення складає від 2 до 15 разів — залежно від характеристик комп’ютера.

Asymptotics for M-type smoothing splines with non-smooth objective functions

M-type smoothing splines are a broad class of spline estimators that include the popular least-squares smoothing spline but also spline estimators that are less susceptible to outlying observations and model misspecification. However, available asymptotic theory only covers smoothing spline estimators based on smooth objective functions and consequently leaves out frequently used resistant estimators such as quantile and Huber-type smoothing splines. We provide a general treatment in this paper and, assuming only the convexity of the objective function, show that the least-squares (super-)convergence rates can be extended to M-type estimators whose asymptotic properties have not been hitherto described. We further show that auxiliary scale estimates may be handled under significantly weaker assumptions than those found in the literature and we establish optimal rates of convergence for the derivatives, which have not been obtained outside the least-squares framework. A simulation study and a real-data example illustrate the competitive performance of non-smooth M-type splines in relation to the least-squares spline on regular data and their superior performance on data that contain anomalies.

Smoothing inertial neurodynamic approach for sparse signal reconstruction via [formula omitted]-norm minimization

In this paper, we propose a smoothing inertial neurodynamic approach (SINA) which is used to deal with Lp-norm minimization problem to reconstruct sparse signals. Note that the considered optimization problem is nonsmooth, nonconvex and non-Lipschitz. First, the problem is transformed into a smooth optimization problem based on smoothing approximation method, and the Lipschitz property of gradient of the smooth objective function is discussed. Then, SINA based on Karush–Kuhn–Tucker (KKT) condition, smoothing approximation and inertial dynamical approach, is designed to handle smooth optimization problem. The existence, uniqueness, global convergence and optimality of the solution of the SINA are discussed by the Cauchy–Lipschitz–Picard theorem, energy function and KKT condition. In addition, for p=1, the SINA has a mean sublinear convergence rate O1∕t under some mild conditions. Finally, some numerical examples on sparse signal reconstruction and image restoration are given to illustrate the theoretical results and the efficiency of SINA.

Extended Gauss-Newton and ADMM-Gauss-Newton algorithms for low-rank matrix optimization

In this paper, we develop a variant of the well-known Gauss-Newton (GN) method to solve a class of nonconvex optimization problems involving low-rank matrix variables. As opposed to the standard GN method, our algorithm allows one to handle general smooth convex objective function. We show, under mild conditions, that the proposed algorithm globally and locally converges to a stationary point of the original problem. We also show empirically that the GN algorithm achieves higher accurate solutions than the alternating minimization algorithm (AMA). Then, we specify our GN scheme to handle the symmetric case and prove its convergence, where AMA is not applicable. Next, we incorporate our GN scheme into the alternating direction method of multipliers (ADMM) to develop an ADMM-GN algorithm. We prove that, under mild conditions and a proper choice of the penalty parameter, our ADMM-GN globally converges to a stationary point of the original problem. Finally, we provide several numerical experiments to illustrate the proposed algorithms. Our results show that the new algorithms have encouraging performance compared to existing methods.

Anytime Minibatch With Delayed Gradients

Distributed optimization is widely deployed in practice to solve a broad range of problems. In a typical asynchronous scheme, workers calculate gradients with respect to out-of-date optimization parameters while the master uses stale (i.e., delayed) gradients to update the parameters. While using stale gradients can slow the convergence, asynchronous methods speed up the overall optimization with respect to wall clock time by allowing more frequent updates and reducing idling times. In this paper, we present a variable per-epoch minibatch scheme called Anytime Minibatch with Delayed Gradients (AMB-DG). In AMB-DG, workers compute gradients in epochs of a fixed time while the master uses stale gradients to update the optimization parameters. We analyze AMB-DG in terms of its regret bound and convergence rate. We prove that for convex smooth objective functions, AMB-DG achieves the optimal regret bound and convergence rate. We compare the performance of AMB-DG with that of Anytime Minibatch (AMB) which is similar to AMB-DG but does not use stale gradients. In AMB, workers stay idle after each gradient transmission to the master until they receive the updated parameters from the master while in AMB-DG workers never idle. We also extend AMB-DG to the fully distributed setting. We compare AMB-DG with AMB when the communication delay is long and observe that AMB-DG converges faster than AMB in wall clock time. We also compare the performance of AMB-DG with the state-of-the-art fixed minibatch approach that uses delayed gradients. We run our experiments on a real distributed system and observe that AMB-DG converges more than two times.

A Bfgs Method for the Problem of Building S-Shaped Curve

We consider the application of the BFGS method (Broyden, Fletcher, Goldfarb, Shanno) and its projective version L-BFGS-B for minimization of nonlinear function, which corresponds to finding solutions to a system of five nonlinear equations. Among them three equations are integral and depend on unknown parameters of integrands and unknown upper bounds for definite integrals. This system represents the problem of building an S-shaped curve in natural parameterization, which passes through the two given points with given tangents inclination angles and provides a given tangent inclination angle at an intermediate point with a given abscissa. The tangent inclination angle at the point with known abscissa is used to control the inflection point of the S-shaped curve, by which a fragment of the external contour of the Frankl nozzle is modeled.The material of the article is presented in three sections. Section 1 describes a system of five nonlinear equations and analyses its properties related to the existence of many solutions. In section 2 the optimization problem with a non-smooth objective function and linear constraints is presented, the global minima of which correspond to solutions of a system of nonlinear equations. There also is described the application of the modification of r-algorithm to find global minima of the optimization problem.Section 3 describes the application of BFGS and L-BFGS-B methods to find local minima of smooth functions corresponding to finding solutions of a system of nonlinear equations. We formulate two optimization problems with smooth objective functions and two-sided constraints on variables and describe the results of computational experiments for finding solution of the system of nonlinear equations, presented in Section 1, by S-shaped curve. It is shown that BFGS and L-BFGS-B methods are effective if the starting point is selected in such a neighborhood of the minimum point, where the minimized function is approximated fairly accurately by a convex quadratic function.Manuscript received 15.06.2020

A gradient‐type iterative method for impulse noise removal

AbstractImage denoising is a typical inverse problem and is hard to be solved. Fortunately, a powerful two‐phase method for restoring images corrupted by high‐level impulse noise has been proposed. The key point of the method is the computational efficiency of the second phase which requires the minimization of a smooth objective function defined on the terms of edge‐preserving potential function. In this paper, we propose an effective three‐term conjugate gradient method to restore the corrupted images in the second phase of two‐phase method. An attractive feature of the proposed method is that the search direction satisfies the sufficient descent property at each iteration without any line search. The global convergence is established for general smooth functions under the Armijo‐type line search. Preliminary numerical results are reported to indicate that the proposed method to be used for impulse noise removal is promising.

Semidefinite Programming and Nash Equilibria in Bimatrix Games

We explore the power of semidefinite programming (SDP) for finding additive ɛ-approximate Nash equilibria in bimatrix games. We introduce an SDP relaxation for a quadratic programming formulation of the Nash equilibrium problem and provide a number of valid inequalities to improve the quality of the relaxation. If a rank-1 solution to this SDP is found, then an exact Nash equilibrium can be recovered. We show that, for a strictly competitive game, our SDP is guaranteed to return a rank-1 solution. We propose two algorithms based on the iterative linearization of smooth nonconvex objective functions whose global minima by design coincide with rank-1 solutions. Empirically, we demonstrate that these algorithms often recover solutions of rank at most 2 and ɛ close to zero. Furthermore, we prove that if a rank-2 solution to our SDP is found, then a [Formula: see text]-Nash equilibrium can be recovered for any game, or a [Formula: see text]-Nash equilibrium for a symmetric game. We then show how our SDP approach can address two (NP-hard) problems of economic interest: finding the maximum welfare achievable under any Nash equilibrium, and testing whether there exists a Nash equilibrium where a particular set of strategies is not played. Finally, we show the connection between our SDP and the first level of the Lasserre/sum of squares hierarchy.

Approximate-Karush-Kuhn-Tucker Conditions and Interval Valued Vector Variational Inequalities

This Article deals with the Approximate Karush-Kuhn-Tucker (AKKT) optimality conditions for interval valued multiobjective function as a generalization of Karush-Kuhn-Tucker optimality conditions. Further, we establish relationship between vector variational inequality problems and multiobjective interval valued optimization problems under the assumption of LU−convex smooth and nonsmooth objective functions.

Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations

Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models. Recently, several approaches to mitigate this problem have been proposed, including different forms of gradient compression or computing local models and mixing them iteratively. In this paper, we propose Qsparse-local-SGD algorithm, which combines aggressive sparsification with quantization and local computation along with error compensation, by keeping track of the difference between the true and compressed gradients. We propose both synchronous and asynchronous implementations of Qsparse-local-SGD . We analyze convergence for Qsparse-local-SGD in the distributed setting for smooth non-convex and convex objective functions. We demonstrate that Qsparse-local-SGD converges at the same rate as vanilla distributed SGD for many important classes of sparsifiers and quantizers. We use Qsparse-local-SGD to train ResNet-50 on ImageNet and show that it results in significant savings over the state-of-the-art, in the number of bits transmitted to reach target accuracy.

A Stochastic Derivative-Free Optimization Method with Importance Sampling: Theory and Learning to Control

We consider the problem of unconstrained minimization of a smooth objective function in ℝn in a setting where only function evaluations are possible. While importance sampling is one of the most popular techniques used by machine learning practitioners to accelerate the convergence of their models when applicable, there is not much existing theory for this acceleration in the derivative-free setting. In this paper, we propose the first derivative free optimization method with importance sampling and derive new improved complexity results on non-convex, convex and strongly convex functions. We conduct extensive experiments on various synthetic and real LIBSVM datasets confirming our theoretical results. We test our method on a collection of continuous control tasks on MuJoCo environments with varying difficulty. Experiments show that our algorithm is practical for high dimensional continuous control problems where importance sampling results in a significant sample complexity improvement.

Wang-Landau algorithm as stochastic optimization and its acceleration.

We show that the Wang-Landau algorithm can be formulated as a stochastic gradient descent algorithm minimizing a smooth and convex objective function, of which the gradient is estimated using Markov chain Monte Carlo iterations. The optimization formulation provides us another way to establish the convergence rate of the Wang-Landau algorithm, by exploiting the fact that almost surely the density estimates (on the logarithmic scale) remain in a compact set, upon which the objective function is strongly convex. The optimization viewpoint motivates us to improve the efficiency of the Wang-Landau algorithm using popular tools including the momentum method and the adaptive learning rate method. We demonstrate the accelerated Wang-Landau algorithm on a two-dimensional Ising model and a two-dimensional ten-state Potts model.

Push–Pull Gradient Methods for Distributed Optimization in Networks

In this article, we focus on solving a distributed convex optimization problem in a network, where each agent has its own convex cost function and the goal is to minimize the sum of the agents' cost functions while obeying the network connectivity structure. In order to minimize the sum of the cost functions, we consider new distributed gradient-based methods where each node maintains two estimates, namely an estimate of the optimal decision variable and an estimate of the gradient for the average of the agents' objective functions. From the viewpoint of an agent, the information about the gradients is pushed to the neighbors, whereas the information about the decision variable is pulled from the neighbors, hence giving the name “push-pull gradient methods.” The methods utilize two different graphs for the information exchange among agents and, as such, unify the algorithms with different types of distributed architecture, including decentralized (peer to peer), centralized (master-slave), and semicentralized (leader-follower) architectures. We show that the proposed algorithms and their many variants converge linearly for strongly convex and smooth objective functions over a network (possibly with unidirectional data links) in both synchronous and asynchronous random-gossip settings. In particular, under the random-gossip setting, “push-pull” is the first class of algorithms for distributed optimization over directed graphs. Moreover, we numerically evaluate our proposed algorithms in both scenarios, and show that they outperform other existing linearly convergent schemes, especially for ill-conditioned problems and networks that are not well balanced.

VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

In this paper, we propose a simple variant of the original SVRG, called variance reduced stochastic gradient descent (VR-SGD). Unlike the choices of snapshot and starting points in SVRG and its proximal variant, Prox-SVRG, the two vectors of VR-SGD are set to the average and last iterate of the previous epoch, respectively. The settings allow us to use much larger learning rates, and also make our convergence analysis more challenging. We also design two different update rules for smooth and non-smooth objective functions, respectively, which means that VR-SGD can tackle non-smooth and/or non-strongly convex problems directly without any reduction techniques. Moreover, we analyze the convergence properties of VR-SGD for strongly convex problems, which show that VR-SGD attains linear convergence. Different from most algorithms that have no convergence guarantees for non-strongly convex problems, we also provide the convergence guarantees of VR-SGD for this case, and empirically verify that VR-SGD with varying learning rates achieves similar performance to its momentum accelerated variant that has the optimal convergence rate $\mathcal {O}(1/T^2)$ O ( 1 / T 2 ) . Finally, we apply VR-SGD to solve various machine learning problems, such as convex and non-convex empirical risk minimization, and leading eigenvalue computation. Experimental results show that VR-SGD converges significantly faster than SVRG and Prox-SVRG, and usually outperforms state-of-the-art accelerated methods, e.g., Katyusha.

A Computation-Efficient Decentralized Algorithm for Composite Constrained Optimization

This paper focuses on solving the problem of composite constrained convex optimization with a sum of smooth convex functions and non-smooth regularization terms (ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norm) subject to locally general constraints. Motivated by the modern large-scale information processing problems in machine learning (the samples of a training dataset are randomly decentralized across multiple computing nodes), each of the smooth objective functions is further considered as the average of several constituent functions. To address the problem in a decentralized fashion, we propose a novel computation-efficient decentralized stochastic gradient algorithm, which leverages the variance reduction technique and the decentralized stochastic gradient projection method with constant step-size. Theoretical analysis indicates that if the constant step-size is less than an explicitly estimated upper bound, the proposed algorithm can find the exact optimal solution in expectation when each constituent function (smooth) is strongly convex. Concerning the existing decentralized schemes, the proposed algorithm not only is suitable for solving the general constrained optimization problems but also possesses low computation cost in terms of the total number of local gradient evaluations. Furthermore, the proposed algorithm via differential privacy strategy can effectively mask the privacy of each constituent function, which is more practical in applications involving sensitive messages, such as military affairs or medical treatment. Finally, numerical evidence is provided to demonstrate the appealing performance of the proposed algorithm.

On the Use of Biased-Randomized Algorithms for Solving Non-Smooth Optimization Problems

Soft constraints are quite common in real-life applications. For example, in freight transportation, the fleet size can be enlarged by outsourcing part of the distribution service and some deliveries to customers can be postponed as well; in inventory management, it is possible to consider stock-outs generated by unexpected demands; and in manufacturing processes and project management, it is frequent that some deadlines cannot be met due to delays in critical steps of the supply chain. However, capacity-, size-, and time-related limitations are included in many optimization problems as hard constraints, while it would be usually more realistic to consider them as soft ones, i.e., they can be violated to some extent by incurring a penalty cost. Most of the times, this penalty cost will be nonlinear and even noncontinuous, which might transform the objective function into a non-smooth one. Despite its many practical applications, non-smooth optimization problems are quite challenging, especially when the underlying optimization problem is NP-hard in nature. In this paper, we propose the use of biased-randomized algorithms as an effective methodology to cope with NP-hard and non-smooth optimization problems in many practical applications. Biased-randomized algorithms extend constructive heuristics by introducing a nonuniform randomization pattern into them. Hence, they can be used to explore promising areas of the solution space without the limitations of gradient-based approaches, which assume the existence of smooth objective functions. Moreover, biased-randomized algorithms can be easily parallelized, thus employing short computing times while exploring a large number of promising regions. This paper discusses these concepts in detail, reviews existing work in different application areas, and highlights current trends and open research lines.

A consensus algorithm based on collective neurodynamic system for distributed optimization with linear and bound constraints

In this paper, an algorithm based on collective neurodynamic system is investigated for distributed constrained convex optimization, whose objective function is a sum of smooth convex functions and non-smooth L1-norm functions. Inspired by recent advances in distributed convex optimization, the continuous-time and discrete-time distributed optimization algorithms described by collective neurodynamic systems are proposed. In the systems, each of the smooth objective functions is allocated to each node as well as each of the L1-norm functions. However, the L1-norm functions are realized by projection operators. Meanwhile, each node satisfies the local linear and bound constraints. Then a connected network is constituted from all the nodes with consensus to find the optimal solutions.

Computation of Optimal Transport and Related Hedging Problems via Penalization and Neural Networks

This paper presents a widely applicable approach to solving (multi-marginal, martingale) optimal transport and related problems via neural networks. The core idea is to penalize the optimization problem in its dual formulation and reduce it to a finite dimensional one which corresponds to optimizing a neural network with smooth objective function. We present numerical examples from optimal transport, martingale optimal transport, portfolio optimization under uncertainty and generative adversarial networks that showcase the generality and effectiveness of the approach.

A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimization

We consider minimization of a smooth nonconvex objective function using an iterative algorithm based on Newton’s method and the linear conjugate gradient algorithm, with explicit detection and use of negative curvature directions for the Hessian of the objective function. The algorithm tracks Newton-conjugate gradient procedures developed in the 1980s closely, but includes enhancements that allow worst-case complexity results to be proved for convergence to points that satisfy approximate first-order and second-order optimality conditions. The complexity results match the best known results in the literature for second-order methods.