Smooth Convex Objects Research Articles

On Seven Fundamental Optimization Challenges in Machine Learning

Many recent successes of machine learning went hand in hand with advances in optimization. The exchange of ideas between these fields has worked both ways, with machine learning building on standard optimization procedures such as gradient descent, as well as with new directions in the optimization theory stemming from machine learning applications. In this thesis, we discuss new developments in optimization inspired by the needs and practice of machine learning, federated learning, and data science. In particular, we consider seven key challenges of mathematical optimization and develop a solution to each. Our first contribution is the resolution of a key open problem in Federated Learning: we establish the first theoretical guarantees for the famous Local SGD algorithm in the heterogeneous data regime. As the second challenge, we close the gap between the upper and lower bounds for the theory of two algorithms known as Random Reshuffling (RR) and Shuffle-Once that are widely used in practice, and set as the default data selection strategies for SGD in modern machine learning software. Our third contribution can be seen as a combination of our new theory for proximal RR and Local SGD yielding a new algorithm, which we call FedRR. Unlike Local SGD, FedRR can provably beat gradient descent in communication complexity in the heterogeneous data regime. The fourth challenge is related to the class of adaptive methods. In particular, we present the first parameter-free stepsize rule for gradient descent that provably works for any locally smooth convex objective. The fifth challenge is the development of an algorithm for distributed optimization with quantized updates that preserves linear convergence of gradient descent. Finally, in our sixth and seventh challenges, we develop new VR mechanisms applicable to the non-smooth setting based on proximal operators.

Quantization of Distributed Data for Learning

We consider machine learning applications that train a model by leveraging data distributed over a trusted network, where communication constraints can create a performance bottleneck. A number of recent approaches propose to overcome this bottleneck through compression of gradient updates. However, as models become larger, so does the size of the gradient updates. In this paper, we propose an alternate approach to learn from distributed data that quantizes data instead of gradients, and can support learning over applications where the size of gradient updates is prohibitive. Our approach leverages the dependency of the computed gradient on data samples, which lie in a much smaller space in order to perform the quantization in the smaller dimension data space. At the cost of an extra gradient computation, the gradient estimate can be refined by conveying the difference between the gradient at the quantized data point and the original gradient using a small number of bits. Lastly, in order to save communication, our approach adds a layer that decides whether to transmit a quantized data sample or not based on its importance for learning. We analyze the convergence of the proposed approach for smooth convex and non-convex objective functions and show that we can achieve order optimal convergence rates with communication that mostly depends on the data rather than the model (gradient) dimension. We use our proposed algorithm to train ResNet models on the CIFAR-10 and ImageNet datasets, and show that we can achieve an order of magnitude savings over gradient compression methods. These communication savings come at the cost of increasing computation at the learning agent, and thus our approach is beneficial in scenarios where communication load is the main problem.

Extended Gauss-Newton and ADMM-Gauss-Newton algorithms for low-rank matrix optimization

In this paper, we develop a variant of the well-known Gauss-Newton (GN) method to solve a class of nonconvex optimization problems involving low-rank matrix variables. As opposed to the standard GN method, our algorithm allows one to handle general smooth convex objective function. We show, under mild conditions, that the proposed algorithm globally and locally converges to a stationary point of the original problem. We also show empirically that the GN algorithm achieves higher accurate solutions than the alternating minimization algorithm (AMA). Then, we specify our GN scheme to handle the symmetric case and prove its convergence, where AMA is not applicable. Next, we incorporate our GN scheme into the alternating direction method of multipliers (ADMM) to develop an ADMM-GN algorithm. We prove that, under mild conditions and a proper choice of the penalty parameter, our ADMM-GN globally converges to a stationary point of the original problem. Finally, we provide several numerical experiments to illustrate the proposed algorithms. Our results show that the new algorithms have encouraging performance compared to existing methods.

Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations

Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models. Recently, several approaches to mitigate this problem have been proposed, including different forms of gradient compression or computing local models and mixing them iteratively. In this paper, we propose Qsparse-local-SGD algorithm, which combines aggressive sparsification with quantization and local computation along with error compensation, by keeping track of the difference between the true and compressed gradients. We propose both synchronous and asynchronous implementations of Qsparse-local-SGD . We analyze convergence for Qsparse-local-SGD in the distributed setting for smooth non-convex and convex objective functions. We demonstrate that Qsparse-local-SGD converges at the same rate as vanilla distributed SGD for many important classes of sparsifiers and quantizers. We use Qsparse-local-SGD to train ResNet-50 on ImageNet and show that it results in significant savings over the state-of-the-art, in the number of bits transmitted to reach target accuracy.

RLC Circuits-Based Distributed Mirror Descent Method

We consider distributed optimization with smooth convex objective functions defined on an undirected connected graph. Inspired by mirror descent mehod and RLC circuits, we propose a novel distributed mirror descent method. Compared with mirror-prox method, our algorithm achieves the same O (1/k) iteration complexity with only half the computation cost per iteration. We further extend our results to cases where a) gradients are corrupted by stochastic noise, and b) objective function is composed of both smooth and non-smooth terms. We demonstrate our theoretical results via numerical experiments.

The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization.

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this article, the convergence of individual iterates of projected subgradient (PSG) methods for nonsmooth convex optimization problems is theoretically studied based on Nesterov's extrapolation, which we name individual convergence. We prove that Nesterov's extrapolation has the strength to make the individual convergence of PSG optimal for nonsmooth problems. In light of this consideration, a direct modification of the subgradient evaluation suffices to achieve optimal individual convergence for strongly convex problems, which can be regarded as making an interesting step toward the open question about stochastic gradient descent (SGD) posed by Shamir. Furthermore, we give an extension of the derived algorithms to solve regularized learning tasks with nonsmooth losses in stochastic settings. Compared with other state-of-the-art nonsmooth methods, the derived algorithms can serve as an alternative to the basic SGD especially in coping with machine learning problems, where an individual output is needed to guarantee the regularization structure while keeping an optimal rate of convergence. Typically, our method is applicable as an efficient tool for solving large-scale l1 -regularized hinge-loss learning problems. Several comparison experiments demonstrate that our individual output not only achieves an optimal convergence rate but also guarantees better sparsity than the averaged solution.

Gradient-type penalty method with inertial effects for solving constrained convex optimization problems with smooth data

We consider the problem of minimizing a smooth convex objective function subject to the set of minima of another differentiable convex function. In order to solve this problem, we propose an algorithm which combines the gradient method with a penalization technique. Moreover, we insert in our algorithm an inertial term, which is able to take advantage of the history of the iterates. We show weak convergence of the generated sequence of iterates to an optimal solution of the optimization problem, provided a condition expressed via the Fenchel conjugate of the constraint function is fulfilled. We also prove convergence for the objective function values to the optimal objective value. The convergence analysis carried out in this paper relies on the celebrated Opial Lemma and generalized Fejér monotonicity techniques. We illustrate the functionality of the method via a numerical experiment addressing image classification via support vector machines.

Variable metric random pursuit

We consider unconstrained randomized optimization of smooth convex objective functions in the gradient-free setting. We analyze Random Pursuit (RP) algorithms with fixed (F-RP) and variable metric (V-RP). The algorithms only use zeroth-order information about the objective function and compute an approximate solution by repeated optimization over randomly chosen one-dimensional subspaces. The distribution of search directions is dictated by the chosen metric. Variable Metric RP uses novel variants of a randomized zeroth-order Hessian approximation scheme recently introduced by Leventhal and Lewis (Optimization 60(3):329---345, 2011. doi:10.1080/02331930903100141). We here present (1) a refined analysis of the expected single step progress of RP algorithms and their global convergence on (strictly) convex functions and (2) novel convergence bounds for V-RP on strongly convex functions. We also quantify how well the employed metric needs to match the local geometry of the function in order for the RP algorithms to converge with the best possible rate. Our theoretical results are accompanied by numerical experiments, comparing V-RP with the derivative-free schemes CMA-ES, Implicit Filtering, Nelder---Mead, NEWUOA, Pattern-Search and Nesterov's gradient-free algorithms.

Computer vision, camouflage breaking and countershading

Camouflage is frequently used in the animal kingdom in order to conceal oneself from visual detection or surveillance. Many camouflage techniques are based on masking the familiar contours and texture of the subject by superposition of multiple edges on top of it. This work presents an operator, D arg, for the detection of three-dimensional smooth convex (or, equivalently, concave) objects. It can be used to detect curved objects on a relatively flat background, regardless of image edges, contours and texture. We show that a typical camouflage found in some animal species seems to be a 'countermeasure' taken against detection that might be based on our method. Detection by D arg is shown to be very robust, from both theoretical considerations and practical examples of real-life images.

Exact and efficient evaluation of the InCircle predicate for parametric ellipses and smooth convex objects

We study the Voronoi diagram, under the Euclidean metric, of a set of ellipses, given in parametric representation. The article concentrates on the InCircle predicate, which is the hardest to compute, and describes an exact and complete solution. It consists of a customized subdivision-based method that achieves quadratic convergence, leading to a real-time implementation for non-degenerate inputs. Degenerate cases are handled using exact algebraic computation. We conclude with experiments showing that most instances run in less than 0.1 s, on a 2.6 GHz Pentium-4, whereas degenerate cases may take up to 13 s. Our approach readily generalizes to smooth convex objects.

A hybrid numerical-asymptotic boundary integral method for high-frequency acoustic scattering

We propose a new robust method for the computation of scattering of high-frequency acoustic plane waves by smooth convex objects in 2D. We formulate this problem by the direct boundary integral method, using the classical combined potential approach. By exploiting the known asymptotics of the solution, we devise particular expansions, valid in various zones of the boundary, which express the solution of the integral equation as a product of explicit oscillatory functions and more slowly varying unknown amplitudes. The amplitudes are approximated by polynomials (of minimum degree d) in each zone using a Galerkin scheme. We prove that the underlying bilinear form is continuous in L 2, with a continuity constant that grows mildly in the wavenumber k. We also show that the bilinear form is uniformly L 2-coercive, independent of k, for all k sufficiently large. (The latter result depends on rather delicate Fourier analysis and is restricted in 2D to circular domains, but it also applies to spheres in higher dimensions.) Using these results and the asymptotic expansion of the solution, we prove superalgebraic convergence of our numerical method as d → ∞ for fixed k. We also prove that, as k → ∞, d has to increase only very modestly to maintain a fixed error bound (d ∼ k 1/9 is a typical behaviour). Numerical experiments show that the method suffers minimal loss of accuracy as k →∞, for a fixed number of degrees of freedom. Numerical solutions with a relative error of about 10−5 are obtained on domains of size $$\mathcal{O}(1)$$ for k up to 800 using about 60 degrees of freedom.

Quantization of the rolling-body problem with applications to motion planning

The problem of manipulation by low-complexity robot hands is a key issue since many years. The performance of simplified hardware manipulators relies on the exploitation of nonholonomic effects that occur in rolling. Beside this issue, more recently, the attention of the scientific community has been devoted to the problems of finite capacity communication channels and of constraints on the complexity of computation. Quantization of controls proved to be efficient for dealing with such kinds of limitations. With this in mind, we consider the rolling of a pair of smooth convex objects, one on top of the other, under quantized control. The analysis of the reachable set is performed by exploiting the geometric nature of the system which helps in reducing to the case of a group acting on a manifold. The cases of a plane, a sphere and a body of revolution rolling on an arbitrary surface are treated in detail.

Dividing snake algorithm for multiple object segmentation

Active contour models, otherwise known as snakes, are extensively used in image processing and computer vision applications. However, although the approach is popular for detecting the contours of smooth convex objects, it is much more problematic in handling images containing an object with concave parts or sharp corners, or multiple objects. We further develop our segmented snake approach to contour detection and illustrate its flexibility by showing how it can be adapted to yield a dividing snake algorithm for use in multiple object segmentation. We also introduce a snake relaxation technique that can improve the convergence of the snake contour onto the object boundary.

The 3D visibility complex

Visibility problems are central to many computer graphics applications. The most common examples include hidden-part removal for view computation, shadow boundaries, mutual visibility of objects for lighting simulation. In this paper, we present a theoretical study of 3D visibility properties for scenes of smooth convex objects. We work in the space of light rays, or more precisely, ofmaximal free segments. We group segments that "see" the same object; this defines the3D visibility complex. The boundaries of these groups of segments correspond to thevisual eventsof the scene (limits of shadows, disappearance of an object when the viewpoint is moved, etc.). We provide a worst case analysis of the complexity of the visibility complex of 3D scenes, as well as a probabilistic study under a simple assumption for "normal" scenes. We extend the visibility complex to handle temporal visibility. We give an output-sensitive construction algorithm and present applications of our approach.

Convexity-Based Visual Camouflage Breaking

Camouflage is frequently used by animals and humans (usually for military purposes) in order to conceal objects from visual surveillance or inspection. Most camouflage methods are based on superposing multiple edges on the object that is supposed to be hidden, such that its familiar contours and texture are masked. In this work, we present an operator, Darg, that is applied directly to the intensity image in order to detect 3D smooth convex (or equivalently: concave) objects. The operator maximally responds to a local intensity configuration that corresponds to curved 3D objects, and thus, is used to detect curved objects on a relatively flat background, regardless of image edges, contours, and texture. In that regard, we show that a typical camouflage found in some animal species seems to be a “counter measure” taken against detection that might be based on our method. Detection by Darg is shown to be very robust, from both theoretic considerations and practical examples of real-life images. As a part of the camouflage breaking demonstration, Darg, which is non-edge-based, is compared with a representative edge-based operator. Better performance is maintained by Darg for both animal and military camouflage breaking.

A global planner for in-hand dextrous re-configuration of rigid objects

This paper describes a global motion planner for dextrous manipulation of three-dimensional objects by a multi-fingered robotic hand. We focus on the so-called re-configuration problem: find a feasible quasi-static trajectory (motions and contactforces) that moves a hand-object system from an initial grasp to a final desired configuration of the object. The planner is designed as a multi-level process: a global level that expands a tree of sub-goals in the configuration space of the object, and a local level that searches for feasible quasi-static trajectories of the entire manipulation system between adjacent sub-goals. A key feature of the planner is that it exploits the redundancy of the system by using, in a complementary way, different canonical manipulation modes and tackles the high dimensionality of the solution space (configuration and control spaces) by making use, instantaneously, of a random search over it. The planner is applied in simulation for achieving several non-trivial re-configuration tasks for (piecewise-) smooth convex objects demonstrating the promise of our approach.

Face detection by direct convexity estimation

We suggest a novel attentional mechanism for detection of smooth convex and concave objects based on direct processing of intensity values. The operator detects the regions of the eyes and hair in a facial image, and thus allows us to infer the face location and scale. Our operator is robust to variations in illumination, scale, and face orientation. Invariance to a large family of functions, serving for lighting improvement in images, is proved. An extensive comparison with edge-based methods is delineated.

Computation of penetration between smooth convex objects in three-dimensional space

There are many applications in robotics where collision detection, separation distance, and penetration distance between geometrical models of objects are required. Efficient numerical procedures for the computation of these proximal relations are important as they are frequently invoked. A new measure of penetration and separation called the growth distance has been introduced in the literature. It has been shown that the growth distance can be efficiently computed for convex polytopes. This article extends the computation of growth distance to smooth convex objects. Specifically, we introduce a formulation of the growth distance for smooth convex objects that is well suited for numerical computation. By modeling a convex object as union of convex subobjects, the growth distance of a wide family of objects can be computed. However, computation of growth distance for such object models may be expensive. A fast algorithm is introduced that reduces the computational time significantly. In the case where the objects undergo continuous relative motions and the growth distances must be evaluated for a large number of closely-spaced points along the motions, further reduction in computational effort is achieved. Numerical experiments with objects that are found in typical robot applications substantiate the claim. © 1996 John Wiley & Sons, Inc.

Resonance spectra of elongated elastic objects

The eigenfrequencies at which smooth convex objects resonate under the incidence of an acoustic wave correspond to the real parts of those complex frequency values at which circumferential waves generated by the incident signal phase-match after repeated circumnavigations around the object [H. Überall, L. R. Dragonette, and L. Flax, J. Acoust. Soc. Am. 61, 711 (1977)]. A resonance condition based on this principle is formulated, and applied to the case of elastic prolate spheroids and cylinders with hemispherical endcaps. Using then the known phase velocities of surface waves on elastic spheres, with a radius equal to the local radius of curvature along the surface path, the elastic resonance frequencies of these objects can be predicted. This was done for the Rayleigh wave on a prolate spheroid, where comparison with resonances in the scattering amplitude as obtained by a T-matrix calculation led to good agreement.

Resonance spectra of elongated elastic objects

It is well known [H. Überall, L. R. Dragonette, and L. Flax, J. Acoust. Soc. Am. 61, 711 (1977)] that the complex eigenfrequencies at which smooth convex objects resonate under the incidence of an acoustic wave are those at which circumferential waves generated by the incident signal phase match over a closed orbit. This principle was verified by us, by analytically obtaining the resonance frequencies of elastic cylinders and spheres [E. D. Breitenbach et al., J. Acoust. Soc. Am. 74, 1267 (1983)] and deriving phase and group velocities of the surface waves from this. In the present study, this approach is inverted, and applied to elastic prolate spheroids and to cylinders with hemispherical endcaps. From the known phase velocities and trajectories (geodesics) of the surface waves, we were able to predict the elastic resonance frequencies of these bodies. [Work supported by the David W. Taylor Naval Ship R & D Center, the Naval Research Laboratory, and the Office of Naval Research.]