Deterministic Convergence Research Articles

Fully complex conjugate gradient-based neural networks using Wirtinger calculus framework: Deterministic convergence and its application

Conjugate gradient method has been verified to be one effective strategy for training neural networks due to its low memory requirements and fast convergence. In this paper, we propose an efficient conjugate gradient method to train fully complex-valued network models in terms of Wirtinger differential operator. Two ways are adopted to enhance the training performance. One is to construct a sufficient descent direction during training by designing a fine tuning conjugate coefficient. Another technique is to pursue the optimal learning rate instead of a fixed constant in each iteration which is determined by employing a generalized Armijo search. In addition, we rigorously prove its weak and strong convergence results, i.e., the gradient norms of objective function with respect to weights approach zero along with the increasing iterations and the weight sequence tends to the optimal point. To verify the effectiveness and rationality of the proposed method, four illustrated simulations have been performed on both typical regression and classification problems.

A convex-programming-based guidance algorithm to capture a tumbling object on orbit using a spacecraft equipped with a robotic manipulator

An algorithm to guide the capture of a tumbling resident space object by a spacecraft equipped with a robotic manipulator is presented. A solution to the guidance problem is found by solving a collection of convex programming problems. As convex programming offers deterministic convergence properties, this algorithm is suitable for onboard implementation and real-time use. A set of hardware-in-the-loop experiments substantiates this claim. To cast the guidance problem as a collection of convex programming problems, the capture maneuver is divided into two simultaneously occurring sub-maneuvers: a system-wide translation and an internal re-configuration. These two sub-maneuvers are optimized in two consecutive steps. A sequential convex programming procedure, overcoming the presence of non-convex constraints and nonlinear dynamics, is used on both optimization steps. A proof of convergence is offered for the system-wide translation, while a set of structured heuristics—trust regions—is used for the optimization of the internal re-configuration sub-maneuver. Videos of the numerically simulated and experimentally demonstrated maneuvers are included as supplementary material.

A Split-Complex Valued Gradient-Based Descent Neuro-Fuzzy Algorithm for TS System and Its Convergence

In order to broaden the study of the most popular and general Takagi–Sugeno (TS) system, we propose a complex-valued neuro-fuzzy inference system which realises the zero-order TS system in the complex-valued network architecture and develop it. In the complex domain, boundedness and analyticity cannot be achieved together. The splitting strategy is given by computing the gradients of the real-valued error function with respect to the real and the imaginary parts of the weight parameters independently. Specifically, this system has four layers: in the Gaussian layer, the L-dimensional complex-valued input features are mapped to a Q-dimensional real-valued space, and in the output layer, complex-valued weights are employed to project it back to the complex domain. Hence, split-complex valued gradients of the real-valued error function are obtained, forming the split-complex valued neuro-fuzzy (split-CVNF) learning algorithm based on gradient descent. Another contribution of this paper is that the deterministic convergence of the split-CVNF algorithm is analysed. It is proved that the error function is monotone during the training iteration process, and the sum of gradient norms tends to zero. By adding a moderate condition, the weight sequence itself is also proved to be convergent.

Is there deterministic, stochastic, and/or club convergence in ecological footprint indicator among G20 countries?

Ecological footprint has been widely accepted as an indicator of environmental performance in recent years since it considers carbon dioxide emissions, the collapse of fisheries, the change in land use, and deforestation. This paper investigates, if exists, the convergence in per capita ecological footprint among G20 countries by employing the annual data for the period 1961 to 2014. A bootstrap-based panel KPSS test with structural breaks and club convergence test are carried out. Eventually, this paper is expected to contribute to the literature of natural resources and ecology/environment by (1) monitoring the panel variable of ecological footprint, (2) launching stochastic and deterministic convergence analyses, and (3) estimating the club convergence parameters. In conclusion, the confirmative results in favor of environmental convergence are obtained by exhibiting the stochastic and deterministic convergences and deriving the output of merging clubs.

Deterministic Convergence and Strong Regularity

Abstract Bayesians since Savage ([1972]) have appealed to asymptotic results to counter charges of excessive subjectivity. Their claim is that objectionable differences in prior probability judgements will vanish as agents learn from evidence, and individual agents will converge to the truth. Glymour ([1980]), Earman ([1992]) and others have voiced the complaint that the theorems used to support these claims tell us, not how probabilities updated on evidence will actually behave in the limit, but merely how Bayesian agents believe they will behave, suggesting that the theorems are too weak to underwrite notions of scientific objectivity and intersubjective agreement. I investigate, in a very general framework, the conditions under which updated probabilities actually converge to a settled opinion and the conditions under which the updated probabilities of two agents actually converge to the same settled opinion. I call this mode of convergence ‘deterministic’, and derive results that extend theorems in (Huttegger [2015b]). The results here lead to a simple characterization of deterministic convergence for Bayesian learners and give rise to an interesting argument for what I call ‘strong regularity’, the view that probabilities of non-empty events should be bounded away from zero. 1Introduction2Preliminaries3Deterministic Convergence4Consensus in the Limit5Strong Regularity6ConclusionAppendix

A Polak-Ribière-Polyak Conjugate Gradient-Based Neuro-Fuzzy Network and its Convergence

Conjugate gradient methods have advantages, such as fast convergence and low memory requirement, which are important for many real-life applications. For zero-order Takagi-Sugeno inference systems, a Polak-Ribiere-Polyak conjugate gradient-based algorithm is proposed to train a neuro-fuzzy network. Compared with the existing gradient-based training algorithm, this scheme efficiently enhances the learning performance. Two deterministic convergence results are proved in detail, which indicate that the gradient of the objective function approaches zero (weak convergence) and the sequence of system parameters tends to a fixed point (strong convergence), respectively. To demonstrate the effectiveness of the proposed algorithm, as well as, to validate the theoretical results, numerical simulations are provided for both function approximation and classification type systems.

A multilevel Monte Carlo finite difference method for random scalar degenerate convection–diffusion equations

This paper proposes a finite difference multilevel Monte Carlo algorithm for degenerate parabolic convection–diffusion equations where the convective and diffusive fluxes are allowed to be random. We establish a notion of stochastic entropy solutions to these equations. Our chief goal is to efficiently compute approximations to statistical moments of these stochastic entropy solutions. To this end, we design a multilevel Monte Carlo method based on a finite volume scheme for each sample. We present a novel convergence rate analysis of the combined multilevel Monte Carlo finite volume method, allowing in particular for low [Formula: see text]-integrability of the random solution with [Formula: see text], and low deterministic convergence rates (here, the theoretical rate is [Formula: see text]). We analyze the design and error versus work of the multilevel estimators. We obtain that the maximal rate (based on optimizing possibly the pessimistic upper bounds on the discretization error) is obtained for [Formula: see text], for finite volume convergence rate of [Formula: see text]. We conclude with numerical experiments.

Deterministic Convergence for Learning Control Systems Over Iteration-Dependent Tracking Intervals.

This brief addresses the iterative learning control (ILC) problems for discrete-time systems subject to iteration-dependent tracking time intervals. A modified class of P-type ILC algorithms is proposed by properly defining an available modified output, for which robust convergence analysis is performed with an inductive approach. It is shown that if a persistent full-learning property is ensured, then a necessary and sufficient convergence condition of ILC can be derived to reach the perfect output tracking objective though the tracking time interval is iteration-dependent. That is, the tracking of ILC for iteration-dependent time intervals can be guaranteed in the same deterministic (not stochastic) convergence way as that of traditional ILC over a fixed time interval. Furthermore, the developed tracking results can be extended to admit iteration-dependent uncertainties in initial state and external disturbances. Simulation tests are also included to demonstrate the effectiveness of the modified P-type ILC.

Constrained Trajectory Optimization for Planetary Entry via Sequential Convex Programming

In this paper, the highly nonlinear planetary-entry optimal control problem is formulated as a sequence of convex problems to facilitate rapid solution. The nonconvex control constraint is avoided by introducing a new state variable to the original three-dimensional equations of motion. The nonconvex objective function and path constraints are convexified by first-order Taylor-series expansions, and the nonconvex terms in the dynamics are approximated by successive linearizations. A successive solution procedure is developed to find an approximated solution to the original problem, and its convergence is discussed. In each iteration, a convex optimization problem is solved by the state-of-the-art interior-point method with deterministic convergence properties. Finally, the proposed method is verified and compared to a general-purpose optimal control solver by numerical solutions of minimum terminal-velocity and minimum heat-load entry problems. The sequential method converges to accurate solutions with faster speed than the general-purpose solver using MATLAB on a desktop computer with a 64-bit operating system and an Intel Xeon E3-1225 V2 3.2 GHz processor, which demonstrates its potential real-time application for computational guidance.

Autonomous trajectory planning for space vehicles with a Newton–Kantorovich/convex programming approach

Space maneuverings of the space vehicle require the capability of onboard trajectory planning. Convex programming based optimization strategy gets much attention in the design of trajectory planning methods with deterministic convergence properties. Due to the nonlinear dynamics, space-trajectory planning problems are always non-convex and difficult to be solved by the convex programming approach directly. This paper presents a Newton–Kantorovich/convex programming (N–K/CP) approach, based on the combination of the convex programming and the Newton–Kantorovich (N–K) method, to solve the nonlinear and non-convex space-trajectory planning problem. This trajectory planning problem is formulated as a nonlinear optimal control problem. By linearization and relaxation techniques, the nonlinear optimal control problem is convexified as a convex programming problem, which can be solved efficiently with convex programming solvers. For the linearized convex optimization problem, N–K method is introduced to design an iterative solving algorithm, the solution of which approximates the original trajectory planning problem with high accuracy. The convergence of the proposed N–K/CP approach is proved, and the effectiveness is demonstrated by numerical experiments and comparisons with other state-of-the-art methods.

On the lifting of deterministic convergence rates for inverse problems with stochastic noise

Both for the theoretical and practical treatment of Inverse Problems, the modeling of the noise is crucial. One either models the measurement via a deterministic worst-case error assumption or assumes a certain stochastic behavior of the noise. Although some connections between both models are known, the communities develop rather independently. In this paper we seek to bridge the gap between the deterministic and the stochastic approach and show convergence and convergence rates for Inverse Problems with stochastic noise by lifting the theory established in the deterministic setting into the stochastic one. This opens the wide field of deterministic regularization methods for stochastic problems without having to do an individual stochastic analysis for each problem.

Deterministic Convergence of Wirtinger-Gradient Methods for Complex-Valued Neural Networks

In this paper, we establish a deterministic convergence of Wirtinger-gradient methods for a class of complex-valued neural networks on the premise of a limited number of training samples. It is different from the probabilistic convergence results under the assumption that a large number of training samples are available. Weak and strong convergence results for Wirtinger-gradient methods are proved, indicating that Wirtinger-gradient of the error function goes to zero and the weight sequence goes to a fixed value. An upper bound of the learning rate is also provided to guarantee the deterministic convergence of Wirtinger-gradient methods. Simulations are provided to support the theoretical findings.

An examination of convergence hypothesis for EU-15 countries

In this paper we investigate whether EU-15 countries experience convergence in per capita GDP levels with respect to the EU-15 average over the period 1950–2015. Nonlinear and nonlinear-asymmetric unit root tests as well as structural break Lagrange Multiplier (LM) unit root tests are employed. When nonlinear and nonlinear-asymmetric unit root tests are employed, five countries exhibit long-run or deterministic convergence with the EU-15 average. However, when endogenous structural break LM unit root tests are employed, nine countries exhibit stochastic convergence. Test results indicate that real per capita income levels of 11 EU countries converge towards the EU-15 average.

African stock markets convergence: Regional and global analysis

This letter examines the convergence of Africa's equity markets within the income convergence thesis. The results indicate partial deterministic convergence of Africa's equity markets both globally and regionally. The findings contribute to extant literature by addressing the puzzle on whether or not emerging equity markets should still be considered as a separate asset class or a unit bucket of assets with those in the developed world.

Structural breaks and labor market disparities in the Canadian provinces

Purpose– The purpose of this paper is to use quarterly time series data from Canada and the Canadian provinces to determine if the unemployment rates in the Canadian provinces are converging to the national rate of unemployment.Design/methodology/approach– First, the authors check for existence of stochastic convergence using recent unit root statistics, see Perron and Rodríguez (2003) and Rodríguez (2007). Second, the authors verify existence of convergence using methods proposed by Volgelsang (1998) and Bai and Perron (1998, 2003). All these methods allows for structural break(s) in the data.Findings– Results from different unit root tests, without and with structural breaks, confirm that stochastic convergence exists in all provinces. The other results show strong evidence that deterministic convergence exists and the unemployment rates of the Canadian provinces are converging to the unemployment rate of Canada. This conclusion is stronger when multiple breaks are allowed in the trend function using the approach of Bai and Perron (1998, 2003).Practical implications– Since the authors have verified the existence of stochastic convergence, any intervention in the labor markets of the Canadian provinces to control the provincial unemployment rate would have a temporary effect and these policies will not have a permanent influence on the unemployment rates. However, existence ofβ-convergence in the Canadian provinces shows that general policies toward lowering the national unemployment rate would decrease the provincial unemployment rates as well.Originality/value– To the best of the knowledge, the paper attempts to study the unemployment rate convergence in the Canadian provinces using the above-mentioned approaches. These approaches allow the authors to take into consideration the possibility of structural breaks in order to get results that are more accurate.

Deterministic convergence of chaos injection-based gradient method for training feedforward neural networks.

It has been shown that, by adding a chaotic sequence to the weight update during the training of neural networks, the chaos injection-based gradient method (CIBGM) is superior to the standard backpropagation algorithm. This paper presents the theoretical convergence analysis of CIBGM for training feedforward neural networks. We consider both the case of batch learning as well as the case of online learning. Under mild conditions, we prove the weak convergence, i.e., the training error tends to a constant and the gradient of the error function tends to zero. Moreover, the strong convergence of CIBGM is also obtained with the help of an extra condition. The theoretical results are substantiated by a simulation example.

Relaxed conditions for convergence of batch BPAP for feedforward neural networks

In this paper, the deterministic convergence of the batch back-propagation algorithm with penalty (BPAP) is proved under certain relaxed conditions for the activation function, the learning rate and the stationary point set of the error function. Both weak and strong convergence results are established. The boundedness of the weights in the training procedure is also proved in a simple and clear way. As a result, the usual requirements on the boundedness of the weights to guarantee the convergence are removed. Simulation results for an approximation problem are presented to support our theoretical findings.

Convergence of Gradient Descent Algorithm with Penalty Term For Recurrent Neural Networks

This paper investigates a gradient descent algorithm with penalty for a recurrent neural network. The penalty we considered here is a term proportional to the norm of the weights. Its primary roles in the methods are to control the magnitude of the weights. After proving that all of the weights are automatically bounded during the iteration process, we also present some deterministic convergence results for this learning methods, indicating that the gradient of the error function goes to zero(weak convergence) and the weight sequence goes to a fixed point(strong convergence), respectively. A numerical example is provided to support the theoretical analysis.

A stochastic convergence analysis for Tikhonov regularization with sparsity constraints

In this paper we investigate convergence properties of Tikhonov regularization for linear ill-posed problems under a stochastic error model. Namely, we assume that we are given a finite amount of measurements, each contaminated by Gaussian noise with zero mean and known finite variance. Using Besov-space penalty terms to promote sparse solutions with respect to a preassigned wavelet basis, the Ky Fan metric allows us to lift deterministic convergence results into the stochastic setting. In particular, we formulate a general convergence theorem and propose a formula to directly calculate a suitable regularization parameter. This immediately leads to convergence rates. Numerical examples are presented to verify the theoretical results.

Spiral bacterial foraging optimization method: Algorithm, evaluation and convergence analysis

A biologically-inspired algorithm called Spiral Bacterial Foraging Optimization (SBFO) is investigated in this article. SBFO, previously proposed by the same authors, is a multi-agent, gradient-based algorithm that minimizes both the main objective function (local cost) and the distance between each agent and a temporary central point (global cost). A random jump is included normal to the connecting line of each agent to the central point, which produces a vortex around the temporary central point. This random jump is also suitable to cope with premature convergence, which is a feature of swarm-based optimization methods. The most important advantages of this algorithm are as follows: First, this algorithm involves a stochastic type of search with a deterministic convergence. Second, as gradient-based methods are employed, faster convergence is demonstrated over GA, DE, BFO, etc. Third, the algorithm can be implemented in a parallel fashion in order to decentralize large-scale computation. Fourth, the algorithm has a limited number of tunable parameters, and finally SBFO has a strong certainty of convergence which is rare in existing global optimization algorithms. A detailed convergence analysis of SBFO for continuously differentiable objective functions has also been investigated in this article.