Convergence Rate Of Algorithm Research Articles

Gossip algorithm with nonuniform clock distribution: Optimization over classical and quantum networks

SummaryDistributed gossip algorithm has been studied for practical implementation of fundamental algorithms employed for collaborative processing. This paper focuses on optimizing the convergence rate of the gossip algorithm for both classical and quantum networks. By modifying the rate of the Poisson process, a new model of the gossip algorithm with nonuniform clock distribution is proposed which can reach the optimal convergence rate of the continuous‐time consensus algorithm. For quantum gossip algorithm, the evolution equation of the quantum gossip algorithm is transformed to the state update equation of the classical gossip algorithm. Defining the quantum gossip operator, the original optimization problem is formulated as a convex optimization problem, where the analytical answer is provided for a series of topologies. It is shown that the optimal results obtained for uniform clock distribution are suboptimal compared to those of the nonuniform one and for nonuniform distribution the optimal answer is not unique. Based on the optimal continuous‐time consensus algorithm and the detailed balance property, an effective method of obtaining one of these optimal answers is proposed.

On the convergence rate of the elitist genetic algorithm based on mutation probability

The Markov chain approach to the elitist genetic algorithm enables not only to prove its convergence to an equilibrium distribution, but also to establish its convergence rate. These convergence rates are based on the transition matrix of the Markov chain which models the algorithm. This paper improves existing estimates of the convergence rates of the elitist genetic algorithm and presents new ones based only on the mutation probability. Experimental results illustrate that, for a fixed mutation probability, the algorithm’s mean convergence time tends to remain unchanged as the crossover probability varies. On the other hand, the reciprocal is not observed.

Newton-type multilevel optimization method

Inspired by multigrid methods for linear systems of equations, multilevel optimization methods have been proposed to solve structured optimization problems. Multilevel methods make more assumptions regarding the structure of the optimization model, and as a result, they outperform single-level methods, especially for large-scale models. The impressive performance of multilevel optimization methods is an empirical observation, and no theoretical explanation has so far been proposed. In order to address this issue, we study the convergence properties of a multilevel method that is motivated by second-order methods. We take the first step toward establishing how the structure of an optimization problem is related to the convergence rate of multilevel algorithms.

Investigation on an Adaptive Method for Tonal Vibration Suppression

Abstract An adaptive method for suppressing mechanical vibration of multiple frequencies is investigated. The adaptive controller is reinforced with saturation alleviator to improve the convergence rate and performance of the adaptive algorithm. Tracking filters are used to extract harmonics of fluctuating frequencies and the anti-saturation unit works in series with the tracking filters to give constrained harmonic output. As a result, the controller is insensitive to abnormally large input that would otherwise induce saturation in actuators. A dynamic model is built for vibration suppression simulation and the numerical results indicate that the adaptive algorithm is effective in cases of multiple fluctuating frequencies and output saturation. Experiments were also conducted to test the performance of the adaptive method. Excitation with oscillating frequencies was applied, and the results have demonstrated that the harmonics can be suppressed effectively with the adaptive method.

An Adaptive Upper Threshold Based Gain Function for the ZA-PNLMS Algorithm

The recently proposed Zero-Attracting Proportionate Normalized Least Mean Square (ZA-PNLMS) algorithm improves the performance of the PNLMS algorithm for identifying sparse systems. In particular, it keeps the fast initial convergence rate of the PNLMS algorithm, and improves its transient performance by arresting the fall in the convergence rate at the later stage of the adaptation process, and it also improves the steady-state mean square error (MSE). However, the improvement in the steady-state performance is marginal. In this brief, we propose a novel gain function for the ZA-PNLMS algorithm by introducing an adaptive upper threshold parameter. It has two fold implications. First, the proposed threshold parameter truncates the proportional gains of highly active taps when they approach to their steady-states and helps the lesser active taps to converge faster by providing them more gains, and thereby the overall transient performance is further improved. Secondly, it also improves steady-state MSE significantly by reducing the fluctuation of the active taps at the steady-state. Extensive simulation studies have been carried out to justify the improvement obtained by the proposed algorithm.

Convergence rate of a relaxed inertial proximal algorithm for convex minimization

ABSTRACTIn a Hilbert space setting, the authors recently introduced a general class of relaxed inertial proximal algorithms that aim to solve monotone inclusions. In this paper, we specialize this study in the case of non-smooth convex minimization problems. We obtain convergence rates for values which have similarities with the results based on the Nesterov accelerated gradient method. The joint adjustment of inertia, relaxation and proximal terms plays a central role. In doing so, we highlight inertial proximal algorithms that converge for general monotone inclusions, and which, in the case of convex minimization, give fast convergence rates of values in the worst case.

Adaptive Tracking Control of FIR Systems Under Binary-Valued Observations and Recursive Projection Identification

In this article, adaptive tracking control of finite impulse response (FIR) systems is studied with binary-valued measurements. An adaptive control strategy is proposed based on an online identification algorithm. First, the designed control inputs are proved to be bounded and satisfy a persistent excitation (PE) condition under the assumption of periodic and PE target signals, which ensures the convergence of the identification algorithm. Second, the convergence rate of the identification algorithm is proved to be O(1/t) and it depends on the true parameter instead of a priori information of the parameter, which is more intuitive. Due to the convergence and the convergence rate of the identification algorithm, we finally prove that the adaptive tracking control is asymptotically optimal and the tracking speed is faster than the previous control algorithm. The simulations are given to validate the developed results in this article.

A novel hybrid whale–Nelder–Mead algorithm for optimization of design and manufacturing problems

This paper introduces a new hybrid optimization algorithm (HWOANM) based on the Nelder–Mead local search algorithm (NM) and whale optimization algorithm (WOA). The aim of hybridization is to accelerate global convergence speed of the whale algorithm for solving manufacturing optimization problems. The main objective of our study on hybridization is to accelerate the global convergence rate of the whale algorithm to solve production optimization problems. This paper is the first research study of both the whale algorithm and HWOANM for the optimization of processing parameters in manufacturing processes. The HWOANM is evaluated using the well-known benchmark problems such as cantilever beam problem, welded beam problem, and three-bar truss problem. Finally, a grinding manufacturing optimization problem is solved to investigate the performance of the HWOANM. The results of the HWOANM for both the design and manufacturing problems solved in this paper are compared with other optimization algorithms presented in the literature such as the ant colony algorithm, genetic algorithm, scatter search algorithm, differential evolution algorithm, particle swarm optimization algorithm, simulated annealing algorithm, artificial bee colony algorithm, improved differential evolution algorithm, harmony search algorithm, hybrid particle swarm algorithm, teaching-learning–based optimization algorithm, cuckoo search algorithm, grasshopper optimization algorithm, salp swarm optimization algorithm, mine blast algorithm, gravitational search algorithm, ant lion optimizer, multi-verse optimizer, whale optimization algorithm, and the Harris hawks optimization algorithm. The results show that the HWOANM provides better exploration and exploitation properties, and can be considered as a promising new algorithm for optimizing both design and manufacturing optimization problems.

Spatial interpolation methods for spectral-spatial remote sensing image super-resolution algorithm based on gradient descent approach

The paper presents an investigation of the influence of spatial interpolation methods on the quality of the image obtained as a result of the spectral-spatial remote sensing image super-resolution reconstruction based on the gradient descent approach. As an example of the super-resolution method, we applied our earlier developed super-resolution image reconstruction algorithm. The algorithm provides the minimization of error of the observation model that connects the input low-resolution images with the target high-resolution image. The iterations of the gradient descent method are performed in the high-resolution spectral and spatial coordinates grid. For this reason, the spatial interpolation operator is added in the observation model. It is evident that spatial interpolation affects both the quality of the reconstructed image and the algorithm convergence rate. The objective of our research was to define the most appropriate spatial interpolation method. The paper presents the results of the spectral-spatial super-resolution image reconstruction using the following spatial interpolation methods: bilinear, bicubic, sinc, and nearest neighbour interpolation. We compare these implementations in terms of such image quality indicators as the root mean square error of the estimated high-resolution image, the algorithm convergence rate, and the presence of textural and colour artefacts as well.

Local Coordinate Exhaustive Search Hybridization Enhancing Big Bang-Big Crunch and Particle Swarm Optimization Algorithms

In general, all of the hybridized evolutionary optimization algorithms use “first diversification and then intensification” routine approach. In other words, these hybridized methods all begin with a global search mode using a highly random initial search population and then switch to intense local search mode at some stage. The population initialization is still a crucial point in the hybridized evolutionary optimization algorithms since it can affect the speed of convergence and the quality of the final solution. In this study, we introduce a new approach by creating a paradigm shift that reverses the “diversification” and then “intensification” routines. Here, instead of starting from a random initial population, we firstly find a unique starting point by conducting an initial exhaustive search based on the coordinate exhaustive search local optimization algorithm only for single step iteration in order to collect a rough but some meaningful knowledge about the nature of the problem. Thus, our main assertion is that this approach will ameliorate convergence rate of any evolutionary optimization algorithms. In this study, we illustrate how one can use this unique starting point in the initialization of two evolutionary optimization algorithms, including but not limited to Big Bang-Big Crunch optimization and Particle Swarm Optimization. Experiments on a commonly used benchmark test suite, which consist of mainly rotated and shifted functions, show that the proposed initialization procedure leads to great improvement for the above-mentioned two evolutionary optimization algorithms.

Adaptive Control of a Piezo-Actuated Steering Mirror to Restrain Laser-Beam Jitter

Tracking, pointing, and imaging performances of precision laser systems are restricted by laser-beam jitter, which is time varying or uncertain. This paper proposes an adaptive control that uses a piezo-actuated steering mirror to actively resist the jitter and achieve precise pointing. First, in order to realize the single-valued linear input–output relationship, which can simplify the adaptive-control design and improve the stability and convergence rate of the adaptive filter algorithm, the hysteresis non-linearity of the piezo-actuators is feedforward compensated. Then, an adaptive controller is built based on an adaptive lattice recursive least-square filter that identifies the disturbance statistics in real-time by utilizing sensor data from a position sensing detector. The developed controllers are verified on a jitter-control system. Experimental results demonstrate that the adaptive-control method significantly extends the disturbance-rejection bandwidth, and that the control performance surpasses the conventional proportion-integral-differential (PID) control method.

A Decentralized Proximal-Gradient Method With Network Independent Step-Sizes and Separated Convergence Rates

This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a previous algorithm, PG-EXTRA \cite{shi2015proximal}, but has a few advantages. First of all, agents use uncoordinated step-sizes, and the stable upper bounds on step-sizes are independent of network topologies. The step-sizes depend on local objective functions, and they can be as large as those of the gradient descent. Secondly, for the special case without non-smooth terms, linear convergence can be achieved under the strong convexity assumption. The dependence of the convergence rate on the objective functions and the network are separated, and the convergence rate of the new algorithm is as good as one of the two convergence rates that match the typical rates for the general gradient descent and the consensus averaging. We provide numerical experiments to demonstrate the efficacy of the introduced algorithm and validate our theoretical discoveries.

Benders decomposition for the inventory vehicle routing problem with perishable products and environmental costs

We consider the problem of inventory routing in the context of perishable products and find near-optimal replenishment scheduling and vehicle routes when the objective is to maximize the supplier’s profit and minimize the costs due to fuel consumption, inventory holding, and greenhouse gas emissions. Greenhouse gas emissions are calculated as a function of fuel consumed, and fuel consumption levels are accurately calculated as a function of vehicle speed, load and traveled distance. To solve the problem efficiently, we develop an exact method based on Benders decomposition to find high-quality solutions in reasonable time. To enhance the convergence rate of the Benders decomposition algorithm, we present several acceleration strategies, such as addition of valid inequalities to the master problem and warm-up start. The warm-up start acceleration strategy itself is a meta-heuristic based on the greedy random adaptive search procedure (GRASP) and mathematical programming formulations. We present computational results which illustrate the superior performance of the proposed solution methodology in solving large-scale instances with 60 customers and 6 planning periods with 4 vehicles using Benders decomposition. Additionally, we show that utilizing a more comprehensive model to calculate the fuel cost results in fuel savings of 2 to 11% on the tested instances compared to traditional models that assume that the fuel cost is solely a function of the distance traveled during delivery.

Two Low Computational Complexity Improved Multiband Structured Subband Adaptive Filter Algorithms

The improved multiband-structured subband adaptive filter (IMSAF) applies the input regressors at each subband to speed up the convergence rate of MSAF. When the projection order is increased, the convergence rate of the IMSAF algorithm improves at the cost of increased complexity. The present research introduces two new IMSAF algorithms with low computational complexity feature. In the first algorithm, the selective partial update approach (SPU) is extended to IMSAF algorithms and SPU-IMSAF is established. In SPU-IMSAF, the filter coefficients are partially updated at each subband for every adaptation. In the second algorithm, the set-membership (SM) strategy is utilized in IMSAF and SM-IMSAF is established. The SM-IMSAF has fast convergence rate, low steady-state error and low computational complexity features at the same time. Also, by combining SM and SPU methods, the SM-SPU-IMSAF is introduced. Simulation results demonstrate the good performance of the proposed algorithms.

Shot-record extended model domain preconditioners for least-squares migration

We have studied the preconditioned conjugate gradient (CG) algorithm in the context of shot-record extended model domain least-squares migration. The CG algorithm is a powerful iterative technique that can solve the least-squares migration problem efficiently; however, to see the merits of least-squares migration, one needs to apply the algorithm for several iterations. Generally speaking, the convergence rate of the CG algorithm depends on the condition number of the operator. Preconditioners are a family of operators that are easy to build and invert. Proper preconditioners can cluster the eigenvalues of the original operator; hence, they reduce the condition number of the operator that one wishes to invert. Accordingly, preconditioning the operator can, in theory, improve the convergence rate of the algorithm. In least-squares migration, the diagonal scaling of the Hessian and the approximated inverse of the Hessian are proven to work well as a preconditioner. We develop and apply two types of preconditioners for the shot-record extended model domain least-squares migration problem. The first preconditioner belongs to the diagonal scaling category, and a second preconditioner is a filter-based approach, which approximates the partial Hessian operators by local convolutional filters. The goal is to increase the convergence rate of the shot-record extended model domain least-squares migration using the reformulated cost function with a preconditioned operator. Experiments with a synthetic Sigsbee model and a real data example from the Gulf of Mexico, Mississippi Canyon data set, indicate that preconditioning the linear system of the equations improves the convergence rate of the algorithm.

Correction to: convergence rates for Kaczmarz-type algorithms

We will refer to the paper “Convergence rates for Kaczmarz-type algorithms”, published in Numerical Algorithms, 79(1)(2018), 1-17. It was observed by Dr. Mokhtar Abbasi (Department of Mathematics, University of Qom, Iran) that in the proof of Theorem 7, related to the linear convergence rate of the MREK algorithm it exists an inequality which is not always true.

Benefits of Using a Spatially-Variant Penalty Strength With Anatomical Priors in PET Reconstruction.

In this study, we explore the use of a spatially-variant penalty strength in penalized image reconstruction using anatomical priors to reduce the dependence of lesion contrast on surrounding activity and lesion location. This work builds on a previous method to make the local perturbation response (LPR) approximately spatially invariant. While the dependence of lesion contrast on the local properties introduced by the anatomical penalty is intentional, the method aims to reduce the influence from surroundings lying along the lines of response (LORs) but not in the penalty neighborhood structure. The method is evaluated using simulated data, assuming that the anatomical information is absent or well-aligned with the corresponding activity images. Since the parallel level sets (PLS) penalty is convex and has shown promising results in the literature, it is chosen as the representative anatomical penalty and incorporated into the previously proposed preconditioned algorithm (L-BFGS-B-PC) for achieving good image quality and fast convergence rate. A 2D disc phantom with a feature at the center and a 3D XCAT thorax phantom with lesions inserted in different slices are used to study how surrounding activity and lesion location affect the visual appearance and quantitative consistency. A bias and noise analysis is also performed with the 2D disc phantom. The consistency of the algorithm convergence rate with respect to different data noise and background levels is also investigated using the XCAT phantom. Finally, an example of reconstruction for a patient dataset with inserted pseudo lesions is used as a demonstration in a clinical context. We show that applying the spatially-variant penalization with PLS can reduce the dependence of the lesion contrast on the surrounding activity and lesion location. It does not affect the bias and noise trade-off curves for matched local resolution. Moreover, when using the proposed penalization, significant improvement in algorithm convergence rate and convergence consistency is observed.

A fundamental proof of convergence of alternating direction method of multipliers for weakly convex optimization

The convergence of the alternating direction method of multipliers (ADMMs) algorithm to convex/nonconvex combinational optimization has been well established in the literature. Due to the extensive applications of a weakly convex function in signal processing and machine learning, in this paper, we investigate the convergence of an ADMM algorithm to the strongly and weakly convex combinational optimization (SWCCO) problem. Specifically, we firstly show the convergence of the iterative sequences of the SWCCO-ADMM under a mild regularity condition; then we establish the o(1/k) sublinear convergence rate of the SWCCO-ADMM algorithm using the same conditions and the linear convergence rate by imposing the gradient Lipschitz continuity condition on the objective function. The techniques used for the convergence analysis in this paper are fundamental, and we accomplish the global convergence without using the Kurdyka–Łojasiewicz (KL) inequality, which is common but complex in the proof of nonconvex ADMM.

Development of a Novel Hybrid Optimization Algorithm for Minimizing Irrigation Deficiencies

One of the most important issues in the field of water resource management is the optimal utilization of dam reservoirs. In the current study, the optimal utilization of the Aydoghmoush Dam Reservoir is examined based on a hybrid of the bat algorithm (BA) and particle swarm optimization algorithm (PSOA) by increasing the convergence rate of the new hybrid algorithm (HA) without being trapped in the local optima. The main goal of the study was to reduce irrigation deficiencies downstream of this reservoir. The results showed that the HA reduced the computational time and increased the convergence rate. The average downstream irrigation demand over a 10-year period (1991–2000) was 25.12 × 106 m3, while the amount of water release based on the HA was 24.48 × 106 m3. Therefore, the HA was able to meet the irrigation demands better than some other evolutionary algorithms. Moreover, lower indices of root mean square error (RMSE) and mean absolute error (MAE) were obtained for the HA. In addition, a multicriteria decision-making model based on the vulnerability, reliability, and reversibility indices and the objective function performed better with the new HA than with the BA, PSOA, genetic algorithm (GA), and shark algorithm (SA) in terms of providing for downstream irrigation demands.

On relaxation of some customized proximal point algorithms for convex minimization: from variational inequality perspective

The proximal point algorithm (PPA) is a fundamental method for convex programming. When applying the PPA to solve linearly constrained convex problems, we may prefer to choose an appropriate metric matrix to define the proximal regularization, so that the computational burden of the resulted PPA can be reduced, and sometimes even admit closed form or efficient solutions. This idea results in the so-called customized PPA (also known as preconditioned PPA), and it covers the linearized ALM, the primal-dual hybrid gradient algorithm, ADMM as special cases. Since each customized PPA owes special structures and has popular applications, it is interesting to ask wether we can design a simple relaxation strategy for these algorithms. In this paper we treat these customized PPA algorithms uniformly by a mixed variational inequality approach, and propose a new relaxation strategy for these customized PPA algorithms. Our idea is based on correcting the dual variables individually and does not rely on relaxing the primal variables. This is very different from previous works. From variational inequality perspective, we prove the global convergence and establish a worst-case convergence rate for these relaxed PPA algorithms. Finally, we demonstrate the performance improvements by some numerical results.