Global Convergence Properties Research Articles

Time optimal control of triple integrator with input saturation and full state constraints

Multiple integrator systems with input saturation and state constraints ubiquitously exist in practical problems such as trajectory planning in CNC (computer numerical control) systems, robots, automotive systems and industrial processes. Without state constraints, the optimal control of the multiple integrator with only input saturation is well solved in classic optimal control theory. However, if state constraints are considered, it is still challenging to obtain a global time optimal analytical solution for multiple integrator systems with even higher than second order, e.g., a double integrator plant. In this paper, the global time optimal control law for triple integrator with input saturation and full state constraints is considered. The system has a serial structure of three integral elements, while the control input and system states are within box constraints. A bang-singular-bang time optimal control law is synthesized according to Pontryagin minimum principle. Then, costates and jump conditions are analyzed in detail. Based on Bellman’s principle of optimality, switching surfaces and curves in phase space are obtained through dynamic programming method. The time optimal control law is then obtained in analytical state feedback form, and its global convergence property is proved. Through the comparison with numerical solutions, the effectiveness of the proposed control strategy is verified. The proposed scheme will be useful for trajectory planning under input saturation and full state constraints in practical applications.

A Modified Dai–Liao Conjugate Gradient Method with a New Parameter for Solving Image Restoration Problems

One adaptive choice for the parameter of the Dai–Liao conjugate gradient method is suggested in this paper, which is obtained with modified quasi–Newton equation. So we get a modified Dai–Liao conjugate gradient method. Some interesting features of the proposed method are introduced: (i) The value of parameter t of the modified Dai–Liao conjugate gradient method takes both the gradient and function value information. (ii) We establish the global convergence property of the modified Dai–Liao conjugate gradient method under some suitable assumptions. (iii) Numerical results show that the modified DL method is effective in practical computation and the image restoration problems.

A method combining norm-relaxed QCQP subproblems with active set identification for inequality constrained optimization

ABSTRACT In this paper, an adaptation of the sequential quadratically constrained quadratic programming method is proposed to solve inequality constrained minimization. At each iteration, it solves a norm-relaxed quadratically constrained quadratic programming subproblem that uses an active set identification technique to reduce the scale and computational cost. By taking a valid line search, the iterates always get into the feasible set after a finite number of iterations, and followed by a suitable update rule for the penalty parameters. Under suitable conditions without the strict complementarity, our method has the global and superlinear convergence properties. In addition, numerical results are reported to demonstrate the efficiency of the proposed method.

Scaled three-term derivative-free methods for solving large-scale nonlinear monotone equations

In this paper, two effective derivative-free methods are proposed for solving large-scale nonlinear monotone equations, in which the search directions are sufficiently descent and independent of the line search. The methods are the extensions of the conjugate gradient methods proposed by Bojari and Eslahchi (Numer. Algorithms 83, pp. 901–933, 2020) combined with the hyperplane projection technique. Our approaches are low storage memory and derivative-free, which makes them suitable for large-scale nonsmooth monotone nonlinear equations. Under proper assumptions, we analyze the global convergence property of the proposed methods. Finally, numerical experiments show that the proposed methods outperform some existing ones.

Hybrid Riemannian conjugate gradient methods with global convergence properties

This paper presents Riemannian conjugate gradient methods and global convergence analyses under the strong Wolfe conditions. The main idea of the proposed methods is to combine the good global convergence properties of the Dai–Yuan method with the efficient numerical performance of the Hestenes–Stiefel method. One of the proposed algorithms is a generalization to Riemannian manifolds of the hybrid conjugate gradient method of the Dai and Yuan in Euclidean space. The proposed methods are compared well numerically with the existing methods for solving several Riemannian optimization problems. Python implementations of the methods used in the numerical experiments are available at https://github.com/iiduka-researches/202008-hybrid-rcg .

Conjugate Gradient ‎Method: A Developed Version to Resolve Unconstrained Optimization Problems

One of the important methods that are widely utilized to resolve ‎unconstrained ‎optimization problems is the Conjugate Gradient (CG) method. This paper aims to propose a new version of the CG method on the basis of Weak Wolfe-Powell (WWP) line search. The assumption is bounded below optimization problems with the Lipschitz continuous gradient. The new parameter obtains global convergence properties when the WWP line search is used. The descent condition is established without using any line search. The performance of the proposed CG method is tested by obtaining some unconstrained optimization problems from the CUTEst library. Testing results show that the proposed version of the CG method outperforms CG-DESCENT version 5.3 in terms of CPU time, function evaluations, gradient evaluations and number of iterations.

A modified Hestenes-Stiefel method for solving unconstrained optimization problems

The conjugate gradient methods are among the most efficient methods for solving optimization models. This is due to its simplicity, low memory requirement and the properties of its global convergent. Many researchers try to improve this technique. In this paper, we suggested a modification of the conjugate gradient parameter with global convergence properties via exact minimization rule. Preliminary experiment was conducted using some unconstrained optimization benchmark problems. Numerical outcome showed that the new algorithm is efficient and promising as it performs better than other classical methods both in terms of number of iteration and CPU time.

A primal-dual interior point trust-region method for nonlinear semidefinite programming

In this paper, we propose a primal-dual interior point trust-region method for solving nonlinear semidefinite programming problems. The method consists of the outer iteration (SDPIP) that finds a Karush–Kuhn–Tucker (KKT) point and the inner iteration (SDPTR) that calculates an approximate barrier KKT point. Algorithm SDPTR combines a commutative class of Newton-like directions with the steepest descent type direction within the framework of the trust-region strategy. We present a trust-region method in primal-dual space and prove the global convergence property of the proposed method. Some numerical experiments are given. In addition, we also present second-order approximations to the primal-dual merit function, and a trust-region method in primal space in Appendix.

A new spectral conjugate gradient method with descent condition and global convergence property for unconstrained optimization

The Spectral conjugate gradient method is an efficient method for solving large-scale unconstrained optimization problems. In this paper, we propose a new spectral conjugate gradient method in which performance is analyzed numerically. We establish the descent condition and global convergence property under some assumptions and the strong Wolfe line search. Numerical experiments to evaluate the method’s efficiency are conducted using 98 problems with various dimensions and initial points. The numerical results based on the number of iterations and central processing unit time show that the new method has a high performance computational.

Accelerating Federated Learning via Momentum Gradient Descent

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update which can potentially accelerate convergence. In this article, we consider momentum term which relates to the last iteration. The proposed momentum federated learning (MFL) uses momentum gradient descent (MGD) in the local update step of FL system. We establish global convergence properties of MFL and derive an upper bound on MFL convergence rate. Comparing the upper bounds on MFL and FL convergence rates, we provide conditions in which MFL accelerates the convergence. For different machine learning models, the convergence performance of MFL is evaluated based on experiments with MNIST and CIFAR-10 datasets. Simulation results confirm that MFL is globally convergent and further reveal significant convergence improvement over FL.

A new method based on the proximal bundle idea and gradient sampling technique for minimizing nonsmooth convex functions

In this paper, we combine the positive aspects of the gradient sampling (GS) and bundle methods, as the most efficient methods in nonsmooth optimization, to develop a robust method for solving unconstrained nonsmooth convex optimization problems. The main aim of the proposed method is to take advantage of both GS and bundle methods, meanwhile avoiding their drawbacks. At each iteration of this method, to find an efficient descent direction, the GS technique is utilized for constructing a local polyhedral model for the objective function. If necessary, via an iterative improvement process, this initial polyhedral model is improved by some techniques inspired by the bundle and GS methods. The convergence of the method is studied, which reveals that the global convergence property of our method is independent of the number of gradient evaluations required to establish and improve the initial polyhedral models. Thus, the presented method needs much fewer gradient evaluations in comparison to the original GS method. Furthermore, by means of numerical simulations, we show that the presented method provides promising results in comparison with GS methods, especially for large scale problems. Moreover, in contrast with some bundle methods, our method is not very sensitive to the accuracy of supplied gradients.

Self‐triggered stabilizing controllers for linear continuous‐time systems

SummarySelf‐triggered control is an improvement on event‐triggered control methods. Unlike the latter, self‐triggered control does not require monitoring the behavior of the system constantly. Instead, self‐triggered algorithms predict the events at which the control law has to be updated before they happen, relying on system model and past information. In this work, we present a self‐triggered version of an event‐triggered control method in which events are generated when a pseudo‐Lyapunov function (PLF) associated with the system increases up to a certain limit. This approach has been shown to considerably decrease the communications between the controller and the plant, while maintaining system stability. To predict the intersections between the PLF and the upper limit, we use a simple and fast root‐finding algorithm. The algorithm mixes the global convergence properties of the bisection and the fast convergence properties of the Newton‐Raphson method. Moreover, to ensure the convergence of the method, the initial iterate of the algorithm is found through a minimization algorithm.

Uniting Observers

We propose a framework for designing observers possessing global convergence properties and desired asymptotic behaviors for the state estimation of nonlinear systems. The proposed scheme consists in combining two given continuous-time observers: One, denoted as global, ensures (approximate) convergence of the estimation error for any initial condition ranging in some prescribed set, while the other, denoted as local, guarantees a desired local behavior. We make assumptions on the properties of these two observers, and not on their structures, and then explain how to unite them as a single scheme using hybrid techniques. Two case studies are provided to demonstrate the applicability of the framework. Finally, a numerical example is presented.

An efficient line search trust-region for systems of nonlinear equations

An improved derivative-free trust-region method to solve systems of nonlinear equations in several variables is presented, combined with the Wolfe conditions to update the trust-region radius. We believe that producing step-sizes by the Wolfe conditions can control the trust-region radius. The new algorithm for which strong global convergence properties are proved is robust and efficient enough to solve systems of nonlinear equations.

A new hyhbrid coefficient of conjugate gradient method

<p><span>Hybridization is one of the popular approaches in modifying the conjugate gradient method. In this paper, a new hybrid conjugate gradient is suggested and analyzed in which the parameter <!--[if gte mso 9]><xml>
 <o:OLEObject Type="Embed" ProgID="Equation.3" ShapeID="_x0000_i1025"
 DrawAspect="Content" ObjectID="_1640083713">
 </o:OLEObject>
 </xml><![endif]-->is evaluated as a convex combination of <!--[if gte mso 9]><xml>
 <o:OLEObject Type="Embed" ProgID="Equation.3" ShapeID="_x0000_i1026"
 DrawAspect="Content" ObjectID="_1640083714">
 </o:OLEObject>
 </xml><![endif]--> while using exact line search. The proposed method is shown to possess both sufficient descent and global convergence properties. Numerical performances show that the proposed method is promising and has overpowered other hybrid conjugate gradient methods in its number of iterations and central processing unit per time. </span></p>

Wei-Yao-Liu Conjugate Gradient Algorithm for Nonsmooth Convex Optimization Problems

This paper presents a Wei-Yao-Liu conjugate gradient algorithm for nonsmooth convex optimization problem. The proposed algorithm makes use of approximate function and gradient values of the Moreau-Yosida regularization function instead of the corresponding exact values. Under suitable conditions, the global convergence property could be established for the proposed conjugate gradient method. Finally, some numerical results are reported to show the efficiency of our algorithm.

Analysis of a two-layer neural network via displacement convexity

Fitting a function by using linear combinations of a large number $N$ of “simple” components is one of the most fruitful ideas in statistical learning. This idea lies at the core of a variety of methods, from two-layer neural networks to kernel regression, to boosting. In general, the resulting risk minimization problem is nonconvex and is solved by gradient descent or its variants. Unfortunately, little is known about global convergence properties of these approaches. Here, we consider the problem of learning a concave function $f$ on a compact convex domain $\Omega \subset {\mathbb{R}}^{d}$, using linear combinations of “bump-like” components (neurons). The parameters to be fitted are the centers of $N$ bumps, and the resulting empirical risk minimization problem is highly nonconvex. We prove that, in the limit in which the number of neurons diverges, the evolution of gradient descent converges to a Wasserstein gradient flow in the space of probability distributions over $\Omega $. Further, when the bump width $\delta $ tends to $0$, this gradient flow has a limit which is a viscous porous medium equation. Remarkably, the cost function optimized by this gradient flow exhibits a special property known as displacement convexity, which implies exponential convergence rates for $N\to \infty $, $\delta \to 0$. Surprisingly, this asymptotic theory appears to capture well the behavior for moderate values of $\delta $, $N$. Explaining this phenomenon, and understanding the dependence on $\delta $, $N$ in a quantitative manner remains an outstanding challenge.

Two Improved Conjugate Gradient Methods with Application in Compressive Sensing and Motion Control

To solve the monotone equations with convex constraints, a novel multiparameterized conjugate gradient method (MPCGM) is designed and analyzed. This kind of conjugate gradient method is derivative-free and can be viewed as a modified version of the famous Fletcher–Reeves (FR) conjugate gradient method. Under approximate conditions, we show that the proposed method has global convergence property. Furthermore, we generalize the MPCGM to solve unconstrained optimization problem and offer another novel conjugate gradient method (NCGM), which satisfies the sufficient descent property without any line search. Global convergence of the NCGM is also proved. Finally, we report some numerical results to show the efficiency of two novel methods. Specifically, their practical applications in compressive sensing and motion control of robot manipulator are also investigated.

Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction.

As an effective learning paradigm against insufficient training samples, multi-task learning (MTL) encourages knowledge sharing across multiple related tasks so as to improve the overall performance. In MTL, a major challenge springs from the phenomenon that sharing the knowledge with dissimilar and hard tasks, known as negative transfer, often results in a worsened performance. Though a substantial amount of studies have been carried out against the negative transfer, most of the existing methods only model the transfer relationship as task correlations, with the transfer across features and tasks left unconsidered. Different from the existing methods, our goal is to alleviate negative transfer collaboratively across features and tasks. To this end, we propose a novel multi-task learning method called task-feature collaborative learning (TFCL). Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks and suppressing inter-group knowledge sharing. We then propose an optimization method for the model. Extensive theoretical analysis shows that our proposed method has the following benefits: (a) it enjoys the global convergence property and (b) it provides a block-diagonal structure recovery guarantee. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks. We further apply it to the personalized attribute prediction problem with fine-grained modeling of user behaviors. Finally, experimental results on both simulated dataset and real-world datasets demonstrate the effectiveness of our proposed method.

Extension of the LP-Newton method to conic programming problems via semi-infinite representation

The LP-Newton method solves the linear programming problem (LP) by repeatedly projecting a current point onto a certain relevant polytope. In this paper, we extend the algorithmic framework of the LP-Newton method to the second-order cone programming problem (SOCP) via a linear semi-infinite programming (LSIP) reformulation of the given SOCP. In the extension, we produce a sequence by projection onto polyhedral cones constructed from LPs obtained by finitely relaxing the LSIP. We show the global convergence property of the proposed algorithm under mild assumptions, and investigate its efficiency through numerical experiments comparing the proposed approach with the primal-dual interior-point method for the SOCP.