Unknown Lipschitz Constant Research Articles

Non-fragile observers design for nonlinear systems with unknown Lipschitz constant

Non-fragile observers design for nonlinear systems with unknown Lipschitz constant

Diagonal Partitioning Strategy Using Bisection of Rectangles and a Novel Sampling Scheme

In this paper, we consider a global optimization problem where the objective function is assumed to be Lipschitz-continuous with an unknown Lipschitz constant. Building upon the recently introduced BIRECT (BIsection of RECTangles) algorithm, we propose a new diagonal partitioning and sampling scheme. Our framework, named BIRECT-V (V for vertices), combines bisection with the sampling of two points. In the initial hyper-rectangle, these points are located at 1/3 and 1 along the main diagonal. Unlike most DIRECT-type algorithms, where evaluating the objective function at vertices is not suitable for bisection, our strategy, when combined with bisection, provides more comprehensive information about the objective function. However, the creation of new sampling points may coincide with existing ones at shared vertices, resulting in additional evaluations of the objective function and increasing the number of function evaluations per iteration. To overcome this issue, we propose modifying the original optimization domain to obtain a good approximation of the global solution. Experimental investigations demonstrate that this modification positively impacts the performance of the BIRECT-V algorithm. Our proposal shows promise as a global optimization algorithm compared to the original BIRECT and two popular DIRECT-type algorithms on a set of test problems. It particularly excels at high-dimensional problems

Stopping Rules for Gradient Methods for Non-convex Problems with Additive Noise in Gradient

We study the gradient method under the assumption that an additively inexact gradient is available for, generally speaking, non-convex problems. The non-convexity of the objective function, as well as the use of an inexactness specified gradient at iterations, can lead to various problems. For example, the trajectory of the gradient method may be far enough away from the starting point. On the other hand, the unbounded removal of the trajectory of the gradient method in the presence of noise can lead to the removal of the trajectory of the method from the desired global solution. The results of investigating the behavior of the trajectory of the gradient method are obtained under the assumption of the inexactness of the gradient and the condition of gradient dominance. It is well known that such a condition is valid for many important non-convex problems. Moreover, it leads to good complexity guarantees for the gradient method. A rule of early stopping of the gradient method is proposed. Firstly, it guarantees achieving an acceptable quality of the exit point of the method in terms of the function. Secondly, the stopping rule ensures a fairly moderate distance of this point from the chosen initial position. In addition to the gradient method with a constant step, its variant with adaptive step size is also investigated in detail, which makes it possible to apply the developed technique in the case of an unknown Lipschitz constant for the gradient. Some computational experiments have been carried out which demonstrate effectiveness of the proposed stopping rule for the investigated gradient methods.

Experimental Study of Excessive Local Refinement Reduction Techniques for Global Optimization DIRECT-Type Algorithms

This article considers a box-constrained global optimization problem for Lipschitz continuous functions with an unknown Lipschitz constant. The well-known derivative-free global search algorithm DIRECT (DIvide RECTangle) is a promising approach for such problems. Several studies have shown that recent two-step (global and local) Pareto selection-based algorithms are very efficient among all DIRECT-type approaches. However, despite its encouraging performance, it was also observed that the candidate selection procedure has two possible shortcomings. First, there is no limit on how small the size of selected candidates can be. Secondly, a balancing strategy between global and local candidate selection is missing. Therefore, it may waste function evaluations by over-exploring the current local minimum and delaying finding the global one. This paper reviews and employs different strategies in a two-step Pareto selection framework (1-DTC-GL) to overcome these limitations. A detailed experimental study has revealed that existing strategies do not always improve and sometimes even worsen results. Since 1-DTC-GL is a DIRECT-type algorithm, the results of this paper provide general guidance for all DIRECT-type algorithms on how to deal with excessive local refinement more efficiently.

Субградиентные методы для задач негладкой оптимизации с некоторой релаксацией условия острого минимума

In this paper we propose a generalized condition for a sharp minimum, somewhat similar to the inexact oracle proposed recently by Devolder-Glineur-Nesterov. The proposed approach makes it possible to extend the class of applicability of subgradient methods with the Polyak step-size, to the situation of inexact information about the value of the minimum, as well as the unknown Lipschitz constant of the objective function. Moreover, the use of local analogs of the global characteristics of the objective function makes it possible to apply the results of this type to wider classes of problems. We show the possibility of applying the proposed approach to strongly convex non-smooth problems, also, we make an experimental comparison with the known optimal subgradient method for such a class of problems. Moreover, there were obtained some results connected to the applicability of the proposed technique to some types of problems with convexity relaxations: the recently proposed notion of weak $\beta$-quasi-convexity and ordinary quasi-convexity. Also in the paper, we study a generalization of the described technique to the situation with the assumption that the $\delta$-subgradient of the objective function is available instead of the usual subgradient. For one of the considered methods, conditions are found under which, in practice, it is possible to escape the projection of the considered iterative sequence onto the feasible set of the problem.

Granular sieving algorithm for selecting best n$$ n $$ parameters

A common type problem of optimization is to find simultaneously parameters that globally minimize an objective function of variables. Such problems are seen in signal and image processing and in various applications of mathematical analysis of several complex variables and Clifford algebras. Objective functions are usually assumed to be Lipschitzian with maybe unknown Lipschitz constants. A number of methods have been established to discard the sets called “bad sets” in a partition that is impossible to contain any optimal point, as well as to treat the unknown Lipschitz bound problem along with the algorithm. In the present paper, a simple criterion of eliminating bad sets is proposed for the first time. The elimination method leads to a concise and rigorous proof of convergence. The algorithm, on the range space side, converges to the global minimum with an exponential rate, while on the domain space side, converges with equal accuracy to the set of all the global minimizers. To treat the unknown Lipschitz constant dilemma, we propose a practical pseudo‐Lipshitz bound process. The methodology is of fundamental nature with straightforward mathematical formulation applicable to multivariate objective functions defined on any compactly connected manifolds in higher dimensions. The method is tested against an extensive number of benchmark functions in the literature. The experimental results exhibit considerable effectiveness and applicability of the algorithm.

Output-feedback distributed consensus for nonlinear multi-agent systems with quantization

In this paper, we investigate the distributed consensus problem for a class of nonlinear multi-agent systems. Firstly, the uniform quantizer with scaling function is considered in the communication channels to reduce the communication burden. Secondly, noting that the state’ information of each agent cannot be directly obtained, we design a suitable observer to rebuild the states of the nonlinear systems with the help of dynamic gain function. Since the information of communication quantization and state observation is coupled to each other, it is necessary to jointly design uniform quantizer with scaling function and the observer. Based on this, a distributed control protocol is proposed, which leads to a distributed asymptotic consensus of overall nonlinear systems with either known or unknown Lipschitz constant through ⌈log2(2R)⌉-bit information exchange between each pair of agents. Finally, to validity the feasibility of the control scheme, a simulation example is provided.

Global optimization method with dual Lipschitz constant estimates for problems with non-convex constraints

This paper considers the constrained global optimization problems, in which the functions are of the “black-box” type and satisfy the Lipschitz condition. The algorithms for solving the problems of this class require the use of adequate estimates of the a priori unknown Lipschitz constants for the problem functions. A novel approach presented in this paper is based on a simultaneous use of two estimates of the Lipschitz constant: an overestimated and an underestimated one. The upper estimate provides the global convergence, whereas the lower one reduces the number of trials necessary to find the global optimizer with the required accuracy. The considered algorithm for solving the constrained problems does not use the ideas of the penalty function method; each constraint of the problem is accounted for separately. The convergence conditions of the proposed algorithm are formulated in the corresponding theorem. The results of the numerical experiments on a series of multiextremal problems with non-convex constraints demonstrating the efficiency of the proposed scheme of dual Lipschitz constant estimates are presented.

Variance-Based Modified Backward-Forward Algorithm with Line Search for Stochastic Variational Inequality Problems and Its Applications

We propose a variance-based modified backward-forward algorithm with a stochastic approximation version of Armijo’s line search, which is robust with respect to an unknown Lipschitz constant, for solving a class of stochastic variational inequality problems. A salient feature of the proposed algorithm is to compute only one projection and two independent queries of a stochastic oracle at each iteration. We analyze the proposed algorithm for its asymptotic convergence, sublinear convergence rate in terms of the mean natural residual function, and optimal oracle complexity under moderate conditions. We also discuss the linear convergence rate with finite computational budget for the proposed algorithm without strong monotonicity. Preliminary numerical experiments indicate that the proposed algorithm is competitive with some existing algorithms. Furthermore, we consider an application in dealing with an equilibrium problem in stochastic natural gas trading market.

On the line-search gradient methods for stochastic optimization

We consider several line-search based gradient methods for stochastic optimization: a gradient and accelerated gradient methods for convex optimization and gradient method for non-convex optimization. The methods simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. The focus of this paper is to numerically compare such methods with state-of-the-art adaptive methods which are based on a different idea of taking norm of the stochastic gradient to define the stepsize, e.g., AdaGrad and Adam.

Nonlinear observer-based recurrent wavelet neuro-controller in disturbance rejection control of flexible structures

In this paper, a model-based output feedback recurrent wavelet neural network (RWNN) controller is proposed for a class of nonlinear MIMO systems with time-varying matched/mismatched uncertainties. The proposed RWNN emulator adaptively trains to follow an ideal state-feedback controller which is designed on the underlying linear model (ULM) of the plant. Simultaneously, the control system employs an adaptive neural network (NN) mechanism to estimate the mismatch between the RWNN controller and this ideal control law. As a result, the conservatism associated with the classical robust control methods where the controller is synthesized based on worst-case bounds is addressed. Moreover, in order to generalize the subjected class of the investigatable plants, the echo-state feature of adaptive RWNN is used to contribute to the performance of nonminimum phase systems. Accordingly, in the context of flexible smart structures with non-collocated sensor/actuator configuration, a delayed feedback is added in the network which brings about a better match between the model output and the measured output. As a result, even for systems with an unknown Lipschitz constant of lumped uncertainty, the controller can be trained online to compensate with an additional revision of the control law following some Lyapunov-based adaptive stabilizing rules. Additionally, the current approach is proposed as an alternative to the hot topic of nonlinear system identification-based control synthesis where the exact structure of the nonlinearity is required.

Improved scheme for selection of potentially optimal hyper-rectangles in DIRECT

We consider a box-constrained global optimization problem with a Lipschitz-continuous objective function and an unknown Lipschitz constant. The well known derivative-free global-search DIRECT (DIvide a hyper-RECTangle) algorithm performs well solving such problems. However, the efficiency of the DIRECT algorithm deteriorates on problems with many local optima and when the solution with high accuracy is required. To overcome these difficulties different regimes of global and local search are introduced or the algorithm is combined with local optimization. In this paper we investigate a different direction of improvement of the DIRECT algorithm and propose a new strategy for the selection of potentially optimal rectangles, what does not require any additional parameters or local search subroutines. An extensive experimental investigation reveals the effectiveness of the proposed enhancements.

GOSH: derivative-free global optimization using multi-dimensional space-filling curves

Global optimization is a field of mathematical programming dealing with finding global (absolute) minima of multi-dimensional multiextremal functions. Problems of this kind where the objective function is non-differentiable, satisfies the Lipschitz condition with an unknown Lipschitz constant, and is given as a “black-box” are very often encountered in engineering optimization applications. Due to the presence of multiple local minima and the absence of differentiability, traditional optimization techniques using gradients and working with problems having only one minimum cannot be applied in this case. These real-life applied problems are attacked here by employing one of the mostly abstract mathematical objects—space-filling curves. A practical derivative-free deterministic method reducing the dimensionality of the problem by using space-filling curves and working simultaneously with all possible estimates of Lipschitz and Holder constants is proposed. A smart adaptive balancing of local and global information collected during the search is performed at each iteration. Conditions ensuring convergence of the new method to the global minima are established. Results of numerical experiments on 1000 randomly generated test functions show a clear superiority of the new method w.r.t. the popular method DIRECT and other competitors.

Global optimization based on bisection of rectangles, function values at diagonals, and a set of Lipschitz constants

We consider a global optimization problem for Lipschitz-continuous functions with an unknown Lipschitz constant. Our approach is based on the well-known DIRECT (DIviding RECTangles) algorithm and motivated by the diagonal partitioning strategy. One of the main advantages of the diagonal partitioning scheme is that the objective function is evaluated at two points at each hyper-rectangle and, therefore, more comprehensive information about the objective function is considered than using the central sampling strategy used in most DIRECT-type algorithms. In this paper, we introduce a new DIRECT-type algorithm, which we call BIRECT (BIsecting RECTangles). In this algorithm, a bisection is used instead of a trisection which is typical for diagonal-based and DIRECT-type algorithms. The bisection is preferable to the trisection because of the shapes of hyper-rectangles, but usual evaluation of the objective function at the center or at the endpoints of the diagonal are not favorable for bisection. In the proposed algorithm the objective function is evaluated at two points on the diagonal equidistant between themselves and the endpoints of a diagonal. This sampling strategy enables reuse of the sampling points in descendant hyper-rectangles. The developed algorithm gives very competitive numerical results compared to the DIRECT algorithm and its well know modifications.

Globalios optimizacijos algoritmas, naudojantis lokalų Lipšico konstantos įvertį

Šiame darbe pasiūlytas naujas deterministinis globalios optimizacijos algoritmas, skirtas juodos dėžės funkcijoms, kurioms galioja Lipšico sąlyga, bet Lipšico konstanta nežinoma. Algoritmo pradinėje stadijoje leistinoji sritis yra padalinama simpleksais. Kiekvienoje algoritmo iteracijoje visiems simpleksams randami Lipšico konstantos įverčiai ir galimos mažiausios funkcijos reikšmės simplekse įverčiai; perspektyviausi simpleksai yra atrenkami ir padalinami. Galimai mažiausiai funkcijos reikšmei simplekse rasti sprendžiamas vidinis optimizavimo uždavinys, norint kurį išspręsti buvo pasiūlytas vidinis algoritmas. Eksperimentai atlikti su dviejų ir trijų dimensijų optimizavimo uždaviniais, panaudojant 400 testinių funkcijų, sugeneruotų su GKLS funkcijų generatoriumi. Rezultatai parodė, kad sudėtingi uždaviniai su pasiūlytuoju algoritmu išsprendžiami su mažesniu funkcijos įvertinimų skaičiumi negu su kitais alternatyviais algoritmais.

A novel adaptive-impulsive synchronization of fractional-order chaotic systems**Project supported by the National Natural Science Foundations of China (Grant Nos. 11161027 and 11262009), the Key Natural Science Foundation of Gansu Province, China (Grant No. 1104WCGA195), the Specialized Research Fund for the Doctoral Program of Higher Education of China (Grant No. 20136204110001).

A novel adaptive–impulsive scheme is proposed for synchronizing fractional-order chaotic systems without the necessity of knowing the attractors’ bounds in priori. The nonlinear functions in these systems are supposed to satisfy local Lipschitz conditions but which are estimated with adaptive laws. The novelty is that the combination of adaptive control and impulsive control offers a control strategy gathering the advantages of both. In order to guarantee the convergence is no less than an expected exponential rate, a combined feedback strength design is created such that the symmetric axis can shift freely according to the updated transient feedback strength. All of the unknown Lipschitz constants are also updated exponentially in the meantime of achieving synchronization. Two different fractional-order chaotic systems are employed to demonstrate the effectiveness of the novel adaptive–impulsive control scheme.

Local Tuning in Nested Scheme of Global Optimization

Numerical methods for global optimization of the multidimensional multiextremal functions in the framework of the approach oriented at dimensionality reduction by means of the nested optimization scheme are considered. This scheme reduces initial multidimensional problem to a set of univariate subproblems connected recursively. That enables to apply efficient univariate algorithms for solving the multidimensional problems. The nested optimization scheme served as the source of many methods for optimization of Lipschitzian function. However, in all of them there is the problem of estimating the Lipschitz constant as the parameter of the function optimized and, as a consequence, of tuning to it the optimization method. In the methods proposed earlier, as a rule, a global estimate (related to whole search domain) is used whereas local Lipschitz constants in some subdomains can differ significantly from the global constant. It can slow down the optimization process considerably. To overcome this drawback in the article the finer estimates of a priori unknown Lipschitz constants taking into account local properties of the objective function are considered and used in the nested optimization scheme. The results of numerical experiments presented demonstrate the advantages of methods with mixed (local and global) estimates of Lipschitz constants in comparison with the use of the global ones only.

Deterministic global optimization using space-filling curves and multiple estimates of Lipschitz and Hölder constants

In this paper, the global optimization problem miny∈SF(y) with S being a hyperinterval in RN and F(y) satisfying the Lipschitz condition with an unknown Lipschitz constant is considered. It is supposed that the function F(y) can be multiextremal, non-differentiable, and given as a ‘black-box’. To attack the problem, a new global optimization algorithm based on the following two ideas is proposed and studied both theoretically and numerically. First, the new algorithm uses numerical approximations to space-filling curves to reduce the original Lipschitz multi-dimensional problem to a univariate one satisfying the Hölder condition. Second, the algorithm at each iteration applies a new geometric technique working with a number of possible Hölder constants chosen from a set of values varying from zero to infinity showing so that ideas introduced in a popular DIRECT method can be used in the Hölder global optimization. Convergence conditions of the resulting deterministic global optimization method are established. Numerical experiments carried out on several hundreds of test functions show quite a promising performance of the new algorithm in comparison with its direct competitors.

Globally-biased Disimpl algorithm for expensive global optimization

Direct-type global optimization algorithms often spend an excessive number of function evaluations on problems with many local optima exploring suboptimal local minima, thereby delaying discovery of the global minimum. In this paper, a globally-biased simplicial partition Disimpl algorithm for global optimization of expensive Lipschitz continuous functions with an unknown Lipschitz constant is proposed. A scheme for an adaptive balancing of local and global information during the search is introduced, implemented, experimentally investigated, and compared with the well-known Direct and Direct l methods. Extensive numerical experiments executed on 800 multidimensional multiextremal test functions show a promising performance of the new acceleration technique with respect to competitors.

Adaptive backstepping synchronization between chaotic systems with unknown Lipschitz constant

The error between two nonlinear terms is a key point of many synchronization problems, however, the Lipschitz constant of the nonlinear term is not always easy to calculate for the stability analysis of the controlled error system, thus the nonlinear systems with unknown parameters and unknown Lipschitz constant is considered in this paper. Their scalar synchronous controller is proposed based on the thought of backstepping design. Without the need to evaluate the invariant set and calculate the Lipschitz constant, an assistant adaptive estimator is designed for the Lipschitz constant. What’s more, as a problem solving skill, two different estimators are used on the same unknown parameter. Finally, the synchronization control for both chaotic autonomous Van der Pol–Duffing (ADVP) systems and chaotic Genesio systems with unknown parameters are given as examples to verify the control effect.