Step Size Selection Research Articles

An Interior-Point Algorithm for Linear Programming with Optimal Selection of Centering Parameter and Step Size

For interior-point algorithms in linear programming, it is well-known that the selection of the centering parameter is crucial for proving polynomility in theory and for efficiency in practice. However, the selection of the centering parameter is usually by heuristics and separate from the selection of the line-search step size. The heuristics are quite different while developing practically efficient algorithms, such as MPC, and theoretically efficient algorithms, such as short-step path-following algorithm. This introduces a dilemma that some algorithms with the best-known polynomial bound are least efficient in practice, and some most efficient algorithms may not be polynomial. In this paper, we propose a systematic way to optimally select the centering parameter and line-search step size at the same time, and we show that the algorithm based on this strategy has the best-known polynomial bound and may be very efficient in computation for real problems.

Distributed multi‐agent optimisation via coordination with second‐order nearest neighbours

The authors examine the distributed multi-agent optimisation problem allowing each agent to explicitly utilise the information of its first-order and second-order nearest neighbours for cooperative decision-making. Specifically, at each time instance, each agent in the network employs the states of its second-order nearest neighbours at the last time instance, which can be transmitted from its first-order nearest neighbours in practice. Under the proposed framework, they propose a distributed subgradient algorithm asymptotically leading all agents to an agreement on the optimal solution of the multi-agent optimisation problem under an appropriate selection of stepsize. Furthermore, they show that the proposed algorithm outperforms the existing algorithms built on the assumption that only the states of first-order neighbours are available in terms of the convergence rate.

Optimal Operation of Energy Hubs Considering Uncertainties and Different Time Resolutions

This article presents a robust chance-constrained optimization framework for the optimal operation management of an energy hub (EH) in the presence of electrical, heating, and cooling demands, and renewable power generation. The proposed strategy can be used for optimal decision making of operators of EHs or energy providers. The electrical energy storage device in the studied EH can handle the fluctuations in operating points raised by such uncertainties. In order to model the hourly demands and renewable power generation uncertainties, a robust chance-constrained close-to-real-time model is adopted in this article. The considered EH in this study follows a centralized framework and the EH operator is responsible for the optimal operation of the hub assets based on the day-ahead scheduling. A thorough analysis of energy flows with different carriers is presented. In addition, a numerical stability test regarding the selection of the time step size is performed to guarantee the solution's time resolution independence, occurring in previous studies.

Vector-based discrete element method for solid elastic materials

This paper presents the vector-based discrete element method applied to two-dimensional elastic bodies, including details of formulation, its implementation on graphics processing units (GPUs) for accelerating simulations and validation cases. Simulation of elastic bodies has traditionally been realised through continuum-mechanics based methods such as finite elements while using discrete element methods have been restricted to small spatial and temporal scales due to the relatively high computational cost. The vector-based discrete element method, or V-model, overcomes the limitations of both traditional continuum-based mechanics and discrete element approaches to enable the possibility to model additional physics such as cracking, recombination and rupture in future studies. In this study we develop and compare CPU and GPU implementations for elastic bodies under deformation only with both static and dynamic validation studies to assess the performance of the method. Results demonstrate the ability of the method to model linear deformation within 1% of the analytical solution and provide qualitative representation of non-linear deformation. The spatial rate of convergence with decreasing particle size is demonstrated to be approximately first order with a methodology to clarify selection of time step size. This paper presents the first implementation of the V-model on GPUs to model elastic bodies demonstrating a 20x speed-up over the CPU implementation and is applied to the stochastic modelling of material properties in a deforming beam.

Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms

SummaryWe present Nesterov‐type acceleration techniques for alternating least squares (ALS) methods applied to canonical tensor decomposition. While Nesterov acceleration turns gradient descent into an optimal first‐order method for convex problems by adding a momentum term with a specific weight sequence, a direct application of this method and weight sequence to ALS results in erratic convergence behavior. This is so because ALS is accelerated instead of gradient descent for our nonconvex problem. Instead, we consider various restart mechanisms and suitable choices of momentum weights that enable effective acceleration. Our extensive empirical results show that the Nesterov‐accelerated ALS methods with restart can be dramatically more efficient than the stand‐alone ALS or Nesterov's accelerated gradient methods, when problems are ill‐conditioned or accurate solutions are desired. The resulting methods perform competitively with or superior to existing acceleration methods for ALS, including ALS acceleration by nonlinear conjugate gradient, nonlinear generalized minimal residual method, or limited‐memory Broyden‐Fletcher‐Goldfarb‐Shanno, and additionally enjoy the benefit of being much easier to implement. We also compare with Nesterov‐type updates where the momentum weight is determined by a line search (LS), which are equivalent or closely related to existing LS methods for ALS. On a large and ill‐conditioned 71×1,000×900 tensor consisting of readings from chemical sensors to track hazardous gases, the restarted Nesterov‐ALS method shows desirable robustness properties and outperforms any of the existing methods we compare with by a large factor. There is clear potential for extending our Nesterov‐type acceleration approach to accelerating other optimization algorithms than ALS applied to other nonconvex problems, such as Tucker tensor decomposition.

Analysis of biased stochastic gradient descent using sequential semidefinite programs

We present a convergence rate analysis for biased stochastic gradient descent (SGD), where individual gradient updates are corrupted by computation errors. We develop stochastic quadratic constraints to formulate a small linear matrix inequality (LMI) whose feasible points lead to convergence bounds of biased SGD. Based on this LMI condition, we develop a sequential minimization approach to analyze the intricate trade-offs that couple stepsize selection, convergence rate, optimization accuracy, and robustness to gradient inaccuracy. We also provide feasible points for this LMI and obtain theoretical formulas that quantify the convergence properties of biased SGD under various assumptions on the loss functions.

A self-adaptive descent LQP alternating direction method for the structured variational inequalities

In this paper, by combining the logarithmic-quadratic proximal (LQP) method and alternating direction method, we proposed an LQP alternating direction method for solving structured variational inequalities. The new iterate is generated by searching the optimal step size along a descent direction with a new step size αk. The choice of the descent direction and the step size selection strategies are important for the algorithm’s efficiency. The O(1/t) convergence rate of the proposed method is studied, and its efficiency is also verified by some numerical experiments.

Variable Step Exponentially Fitted Explicit Sixth-Order Hybrid Method with Four Stages for Spring-Mass and Other Oscillatory Problems

For the numerical integration of differential equations with oscillatory solutions an exponentially fitted explicit sixth-order hybrid method with four stages is presented. This method is implemented using variable step-size while its derivation is accomplished by imposing each stage of the formula to integrate exactly { 1 , t , t 2 , … , t k , exp ( ± μ t ) } where the frequency μ is imaginary. The local error that is employed in the step-size selection procedure is approximated using an exponentially fitted explicit fourth-order hybrid method. Numerical comparisons of the new and existing hybrid methods for the spring-mass and other oscillatory problems are tabulated and discussed. The results show that the variable step exponentially fitted explicit sixth-order hybrid method outperforms the existing hybrid methods with variable coefficients for solving several problems with oscillatory solutions.

Geometric numerical integration of the assignment flow

The assignment flow is a smooth dynamical system that evolves on an elementary statistical manifold and performs contextual data labeling on a graph. We derive and introduce the linear assignment flow that evolves nonlinearly on the manifold, but is governed by a linear ODE on the tangent space. Various numerical schemes adapted to the mathematical structure of these two models are designed and studied, for the geometric numerical integration of both flows: embedded Runge–Kutta–Munthe–Kaas schemes for the nonlinear flow, adaptive Runge–Kutta schemes and exponential integrators for the linear flow. All algorithms are parameter free, except for setting a tolerance value that specifies adaptive step size selection by monitoring the local integration error, or fixing the dimension of the Krylov subspace approximation. These algorithms provide a basis for applying the assignment flow to machine learning scenarios beyond supervised labeling, including unsupervised labeling and learning from controlled assignment flows.

Effect of eigenvalue spread in noise cancellation of two sensory systems using adaptive algorithms

The least mean square (LMS) algorithm is the basic generic search algorithm and simple to implement, also it has stable and has robust performance. Therefore, numerous adaptive algorithms have been derived from LMS. But maximum research is limited to convergence rate maximization in the algorithms of LMS family. Eigenvalue spread changes the convergence of any adaptive algorithms as it also affects the selection of step size. When spectral power of input signal is non-stationary, the residual error is more in all the cases. The performance analysis of adaptive algorithms for eigenvalue spread is still ignored by researchers. In this paper the effect of eigenvalue spread on different popular algorithm of LMS family has been studied. For this, many noises of different eigenvalue spread were mixed with a voice signal i.e. pronunciation of the word “This activity is truly unproductive” to make primary signal. Then simulation has done to analyze the effect of eigenvalue spread in MATLAB for different algorithms. Eigenvalue spread is calculated for all contaminated primary signals then steady state error or residual errors have also been calculated for LMS, VSSLMS, and QLMS. Their performance is compared by changing the surrounding noisy environment as car noise, water raining noise, thonder noise, and aeroplane noise. To analyses the effect of eigenvalue on adaptive algorithms, a table is prepared on the basis of residual error and eigenvalue spread for chosen algorithms. This analysis will help to select the algorithm for devices, which need to work in changing surrounding environment.

A self-adaptive regularized alternating least squares method for tensor decomposition problems

Though the alternating least squares algorithm (ALS), as a classic and easily implemented algorithm, has been widely applied to tensor decomposition and approximation problems, it has some drawbacks: the convergence of ALS is not guaranteed, and the swamp phenomenon appears in some cases, causing the convergence rate to slow down dramatically. To overcome these shortcomings, the regularized-ALS algorithm (RALS) was proposed in the literature. By employing the optimal step-size selection rule, we develop a self-adaptive regularized alternating least squares method (SA-RALS) to accelerate RALS in this paper. Theoretically, we show that the step-size is always larger than unity, and can be larger than [Formula: see text], which is quite different from several optimization algorithms. Furthermore, under mild assumptions, we prove that the whole sequence generated by SA-RALS converges to a stationary point of the objective function. Numerical results verify that the SA-RALS performs better than RALS in terms of the number of iterations and the CPU time.

Efficient gradient‐based dictionary optimisation method for time‐varying array incorporating with structure characteristic

To guarantee good sparsity reconstruction quality, a suitable dictionary should be as orthogonal as possible. Also, to reduce the coherence of a dictionary, a sensing matrix optimisation model should be formulated. In this model, a high-dimension O (N) problem has to be considered because the sparse representation base is overcompleted, thus leading to a heavy computational load. This study proposes an efficient gradient-based method to address the dictionary optimisation problem of the time-varying arrays, whose elements relatively move in an arbitrary but known way. The dictionary optimisation models associated with different array geometries are formulated with distinct structure characteristic, and the Toeplitz and circulant properties are incorporated into the corresponding models to reduce the complexity by dimension reduction from O (N) to O (2) and O (1), respectively. An alternating minimisation approach for sensing matrix design (SMD) is derived by the gradient descent method, and an adaptive stepsize selection method is derived to further reduce complexity. Numerous simulations are conducted, and simulation results demonstrate that the presented methods have lower computational complexity but similar sparse recovery performance as the normalisation-constrained SMD method. Also, the effectiveness of the proposed structured optimisation framework to design a sensing matrix of a time-varying array is verified.

Stepsize selection in the rigorous defect control of Taylor series methods

Modern numerical methods for initial-value problems (IVPs) in ordinary differential equations (ODE) produce a (piecewise) differentiable numerical solution. The defect or residual evaluated at it induces a perturbed ODE that is satisfied exactly by this solution. Defect control methods try to ensure that a norm of the defect is bounded by a user-specified tolerance. This paper justifies and implements a simple and effective stepsize selection strategy and an overall method for both validated and non-validated defect control of an explicit Taylor series method for IVP ODEs. In validated mode, this method guarantees that the defect is bounded by a user-specified tolerance.

Adaptive step-size selection for state-space probabilistic differential equation solvers

When models are defined implicitly by systems of differential equations with no closed-form solution, small local errors in finite-dimensional solution approximations can propagate into deviations from the true underlying model trajectory. Some recent perspectives in quantifying this uncertainty are based on Bayesian probability modeling: a prior is defined over the unknown solution and updated by conditioning on interrogations of the forward model. Improvement in accuracy via grid refinement must be considered in order for such Bayesian numerical methods to compete with state-of-the-art numerical techniques. We review the principles of Bayesian statistical design and apply these to develop an adaptive probabilistic method to sequentially select time-steps for state-space probabilistic ODE solvers. We investigate the behavior of local error under the adaptive scheme which underlies numerical variable step-size methods. Numerical experiments are used to illustrate the performance of such an adaptive scheme, showing improved accuracy over uniform designs in terms of local error.

Computable upper error bounds for Krylov approximations to matrix exponentials and associated {varvec{varphi }}-functions

An a posteriori estimate for the error of a standard Krylov approximation to the matrix exponential is derived. The estimate is based on the defect (residual) of the Krylov approximation and is proven to constitute a rigorous upper bound on the error, in contrast to existing asymptotical approximations. It can be computed economically in the underlying Krylov space. In view of time-stepping applications, assuming that the given matrix is scaled by a time step, it is shown that the bound is asymptotically correct (with an order related to the dimension of the Krylov space) for the time step tending to zero. This means that the deviation of the error estimate from the true error tends to zero faster than the error itself. Furthermore, this result is extended to Krylov approximations of varphi -functions and to improved versions of such approximations. The accuracy of the derived bounds is demonstrated by examples and compared with different variants known from the literature, which are also investigated more closely. Alternative error bounds are tested on examples, in particular a version based on the concept of effective order. For the case where the matrix exponential is used in time integration algorithms, a step size selection strategy is proposed and illustrated by experiments.

Multi-Implicit Methods with Automatic Error Control in Applications with Chemical Reactions

Multi-implicit methods with a second derivative for stiff systems of ordinary differential equations are described. An algorithm for automatic step size control and selection based on multi-implicit methods of the eighth and sixth orders of accuracy is proposed. The efficiency of the variable step size methods is demonstrated as applied to nonequilibrium kinetics of chemical reactions describing an explosion of a hydrogen–oxygen mixture consisting of six species (H2, O2, H, O, OH, H2O).

Efficient Direction-of-Arrival Estimation Method Based on Variable-Step-Size LMS Algorithm

Conventional direction-of-arrival (DOA) estimation methods based on subspace decomposition are computationally complex and difficult to implement in practical applications. Although a fixed-step-size least mean square (FSS-LMS) algorithm for DOA estimation has overcome the complexity problem; however, the manual selection of suitable step size is difficult. In this letter, a variable-step-size LMS algorithm is proposed for DOA estimation. In the proposed method, the step size is updated by using the estimated signal power and the cumulative sum of squared instantaneous error, and the spatial spectrum is obtained by the reciprocal of array pattern, where the peak values indicate the estimated DOAs of signals. The proposed method can choose step size adaptively and provide better performance with low computational complexity. The performance of proposed method is verified by the numerical simulations and compared with some existing methods.

Low complex direction of arrival estimation method based on adaptive filtering algorithm

Conventional subspace decomposition-based direction of arrival (DOA) estimation methods require eigenvalue decomposition of spatial covariance matrix, therefore these methods are computationally intensive, and their implementation is difficult in real-time applications. However, a DOA estimation method based on fixed step size least mean square (LMS) algorithm has overcome this deficiency but the suitable selection of step size is very difficult. A DOA estimation method based on the variable step size LMS algorithm is proposed in this article. In the proposed method, the step size is updated by using the estimated error signal. The spatial spectrum is obtained by the reciprocal of array pattern, where the peak values indicate the estimated DOAs of signals. The proposed method can provide better performance with low computational cost. The performance of proposed method is verified by the simulations and compared with some existing methods.

Implementation and evaluation distributed mixed pixels analysis algorithm for hyperspectral image based on constraint non-negative matrix factorization

As an effective blind source separation method, non-negative matrix factorization has been widely adopted to analyze mixed data in hyperspectral images. To avoid trapping in local optimum, appropriate constraints are added to the objective function of NMF, whose reflections of image essential attribute determine the performance finally. In this paper, a new NMF-based mixed data analysis algorithm is presented, with maximum overall coverage constraint introduced in traditional NMF. The new constraint was proposed using data geometrical properties in the feature space to maximizes the number of pixels contained in the simplex constructed by endmembers compulsorily and introduced in objective function of NMF, named maximum overall coverage constraint NMF (MOCC-NMF), to analyze mixed data in highly mixed hyperspectral data without pure pixels. For implementing easily, multiplicative update rules are applied to avoid step size selection problem occurred in traditional gradient-based optimization algorithm frequently. Furthermore, in order to handle huge computation involved, parallelism implementation of the proposed algorithm using MapReduce is described and the new partitioning strategy to obtain matrix multiplication and determinant value is discussed in detail. In the numerical experiments conducted on real hyperspectral and synthetic datasets of different sizes, the efficiency and scalability of the proposed algorithm are confirmed.

Explicit methods for integrating stiff Cauchy problems

An explicit method for solving stiff Cauchy problems is proposed. The method relies on explicit schemes and a step size selection algorithm based on the curvature of an integral curve. Closed-form formulas are derived for finding the curvature. For Runge-Kutta schemes with up to four stages, the corresponding sets of coefficients are given. The method is validated on a test problem with a given exact solution. It is shown that the method is as accurate and robust as implicit methods, but is substantially superior to them in efficiency. A numerical example involving chemical kinetics computations with 9 components and 50 reactions is given.