Optimal Step Size Research Articles

Gradient methods exploiting spectral properties

We propose a new stepsize for the gradient method. It is shown that this new stepsize will converge to the reciprocal of the largest eigenvalue of the Hessian, when Dai-Yang's asymptotic optimal gradient method (Computational Optimization and Applications, 2006, 33(1): 73–88) is applied for minimizing quadratic objective functions. Based on this spectral property, we develop a monotone gradient method that takes a certain number of steps using the asymptotically optimal stepsize by Dai and Yang, and then follows by some short steps associated with this new stepsize. By employing one step retard of the asymptotic optimal stepsize, a nonmonotone variant of this method is also proposed. Under mild conditions, R-linear convergence of the proposed methods is established for minimizing quadratic functions. In addition, by combining gradient projection techniques and adaptive nonmonotone line search, we further extend those methods for general bound constrained optimization. Two variants of gradient projection methods combining with the Barzilai-Borwein stepsizes are also proposed. Our numerical experiments on both quadratic and bound constrained optimization indicate that the new proposed strategies and methods are very effective.

Improved utility harmonic impedance measurement based on robust independent component analysis and bootstrap check

In this study, a new method for utility harmonic impedance measurement based on robust independent component analysis (RICA) and bootstrap check is proposed. The RICA approach which adopts exact line search to seek the optimal step-size in iteration has better performance for short data records. At the same time, a bootstrap check technique to remove the singular solutions of this sort of independent component analyses methods is presented as well. This verification technique is performed through judging the dispersion degree of the estimates of the bootstrap samples which is quantified by the coefficient of variation. Based on the results of computer simulation and filed test, it is proved that the combination of the RICA approach with the bootstrap check technique can reduce the adverse impact of the background harmonic fluctuation and the singular solutions on impedance estimation effectively, and satisfactory measurement results can be acquired.

Using Cuckoo Search Algorithm with Q-Learning and Genetic Operation to Solve the Problem of Logistics Distribution Center Location

Cuckoo search (CS) algorithm is a novel swarm intelligence optimization algorithm, which is successfully applied to solve some optimization problems. However, it has some disadvantages, as it is easily trapped in local optimal solutions. Therefore, in this work, a new CS extension with Q-Learning step size and genetic operator, namely dynamic step size cuckoo search algorithm (DMQL-CS), is proposed. Step size control strategy is considered as action in DMQL-CS algorithm, which is used to examine the individual multi-step evolution effect and learn the individual optimal step size by calculating the Q function value. Furthermore, genetic operators are added to DMQL-CS algorithm. Crossover and mutation operations expand search area of the population and improve the diversity of the population. Comparing with various CS algorithms and variants of differential evolution (DE), the results demonstrate that the DMQL-CS algorithm is a competitive swarm algorithm. In addition, the DMQL-CS algorithm was applied to solve the problem of logistics distribution center location. The effectiveness of the proposed method was verified by comparing with cuckoo search (CS), improved cuckoo search algorithm (ICS), modified chaos-enhanced cuckoo search algorithm (CCS), and immune genetic algorithm (IGA) for both 6 and 10 distribution centers.

Fixed Point Analysis of Douglas--Rachford Splitting for Ptychography and Phase Retrieval

Douglas--Rachford splitting (DRS) methods based on the proximal point algorithms for the Poisson and Gaussian log-likelihood functions are proposed for ptychography and phase retrieval. Fixed point analysis shows that the DRS iterated sequences are always bounded explicitly in terms of the step size and that the fixed points are attracting if and only if the fixed points are regular solutions. This alleviates two major drawbacks of the classical Douglas--Rachford algorithm: slow convergence when the feasibility problem is consistent and divergent behavior when the feasibility problem is inconsistent. Fixed point analysis also leads to a simple, explicit expression for the optimal step size in terms of the spectral gap of an underlying matrix. When applied to the challenging problem of blind ptychography, which seeks to recover both the object and the probe simultaneously, alternating minimization with the DRS inner loops, even with a far from optimal step size, converges geometrically under the nearly minimum conditions established in the uniqueness theory.

Machine Learning-Based Signal Detection for CoMP Downlink in Ultra-Dense Small Cell Networks

In the next generation wireless communication systems, high-speed data rate, reliable link quality, and ubiquitous access are necessary requirement. To meet the next generation communication system requirements, the ultra-dense small cell network (SCN) is proposed as one of the possible candidate solutions. To achieve high data rate and reliable link quality, coordinated multiple point (CoMP) transmission is usually used in ultra-dense SCN to satisfy the performance target. However, to realize CoMP transmissions in ultra-dense SCN, the feedback load is heavy because of the large amount of small cells and users. In this study, we want to investigate the ultra-dense SCN environment and design an effective method for its downlink (DL) transmission. This method consists of iterative scheme which iteratively solves the received signal with proper step-size value $\gamma $ . The step-size value $\gamma $ is very important to the convergence performance of iterative scheme because it affects the convergence speed and the converged error level of the iterative algorithm. Therefore, in this paper, we propose a novel machine learning based method which creates data model from input data. When the model is well established, the optimal step-size can be well estimated and the feedback information can become rough and the bandwidth allocated for feedback can be saved for data transmission. The simulations show that, the proposed method can create good model and achieve better convergence performance. For example, about the distribution of the transmissions with convergence iteration number less than $10^{4}$ level, the proposed method can obtain 30% improvement than the traditional method with fixed step-size $\gamma =0.01$ .

Recursive search‐based identification algorithms for the exponential autoregressive time series model with coloured noise

This study focuses on the recursive parameter estimation problems for the non-linear exponential autoregressive model with moving average noise (the ExpARMA model for short). By means of the gradient search, an extended stochastic gradient (ESG) algorithm is derived. Considering the difficulty of determining the step-size in the ESG algorithm, a numerical approach is proposed to obtain the optimal step-size. In order to improve the parameter estimation accuracy, the authors employ the multi-innovation identification theory to develop a multi-innovation ESG (MI-ESG) algorithm for the ExpARMA model. Introducing a forgetting factor into the MI-ESG algorithm, the parameter estimation accuracy can be further improved. With an appropriate innovation length and forgetting factor, the variant of the MI-ESG algorithm is effective to identify all the unknown parameters of the ExpARMA model. A simulation example is provided to test the proposed algorithms.

Tracking Analysis of Gaussian Kernel Signed Error Algorithm for Time-Variant Nonlinear Systems

This brief establishes a novel kernel-based model with a random walk variation of the optimum weight coefficients to characterize the time-variant nonlinear system. Then, the steady-state tracking performance of the kernel signed error algorithm (KSEA) with Gaussian kernel is analyzed for the proposed time-variant nonlinear system in the presence of non-Gaussian impulsive noise. The theoretical findings enable us to determine the optimal step-size that minimizes the steady-state excess mean-square error under this non-stationary environment. Simulation results illustrate the usefulness and accuracy of the derived analytical models for characterizing the steady-state tracking behavior of Gaussian KSEA.

Parameters estimate of recurrent quantum stochastic filter for time variant frequency periodic signals

Designing optimal time and spatial difference step size is the key technology for quantum-random filtering (QSF) to realize time-varying frequency periodic signal filtering. In this paper, it was proposed to use the short-time Fourier transform (STFT) to dynamically estimate the signal to noise ratio (SNR) and relative frequency of the input time-varying frequency periodic signal. Then the model of time and space difference step size and signal to noise ratio (SNR) and relative frequency of quantum random filter is established by least square method. Finally, the parameters of the quantum filter can be determined step by step by analyzing the characteristics of the actual signal. The simulation results of single-frequency signal and frequency time-varying signal show that the proposed method can quickly and accurately design the optimal filter parameters based on the characteristics of the input signal, and achieve significant filtering effects.

Two-stage multi-innovation stochastic gradient algorithm for multivariate output-error ARMA systems based on the auxiliary model

ABSTRACTThis paper investigates the parameter estimation problem for multivariate output-error systems perturbed by autoregressive moving average noises. Since the identification model has two different kinds of parameters, a vector and a matrix, the gradient algorithm cannot be used directly. Therefore, we decompose the original system model into two sub-models and proceed the identification problem by the collaboration between the two sub-models. By employing the gradient search and determining the optimal step-sizes, we present an auxiliary model based two-stage projection algorithm. However, in order to alleviate the sensitivity to the noise, we reselect the step-sizes and derive the auxiliary model based two-stage stochastic gradient (AM-2S-SG) algorithm. Based on the AM-2S-SG algorithm, an auxiliary model based two-stage multi-innovation stochastic gradient algorithm is proposed to generate more accurate estimates. Finally, numerical simulations are provided to demonstrate the effectiveness of the proposed algorithms.

Optimally linearizing the alternating direction method of multipliers for convex programming

The alternating direction method of multipliers (ADMM) is being widely used in a variety of areas; its different variants tailored for different application scenarios have also been deeply researched in the literature. Among them, the linearized ADMM has received particularly wide attention in many areas because of its efficiency and easy implementation. To theoretically guarantee convergence of the linearized ADMM, the step size for the linearized subproblems, or the reciprocal of the linearization parameter, should be sufficiently small. On the other hand, small step sizes decelerate the convergence numerically. Hence, it is interesting to probe the optimal (largest) value of the step size that guarantees convergence of the linearized ADMM. This analysis is lacked in the literature. In this paper, we provide a rigorous mathematical analysis for finding this optimal step size of the linearized ADMM and accordingly set up the optimal version of the linearized ADMM in the convex programming context. The global convergence and worst-case convergence rate measured by the iteration complexity of the optimal version of linearized ADMM are proved as well.

Rank-two update algorithm versus Frank–Wolfe algorithm with away steps for the weighted Euclidean one-center problem

The weighted Euclidean one-center (WEOC) problem is one of the classic problems in facility location theory, which is also a generalization of the minimum enclosing ball (MEB) problem. Given m points in $${\mathbb {R}}^{n}$$, the WEOC problem computes a center point $$c\in {\mathbb {R}}^{n}$$ that minimizes the maximum weighted Euclidean distance to m given points. The rank-two update algorithm is an effective method for solving the minimum volume enclosing ellipsoid (MVEE) problem. It updates only two components of the solution at each iteration, which was previously proposed in Cong et al. (Comput Optim Appl 51(1):241–257, 2012). In this paper, we further develop and analyze the rank-two update algorithm for solving the WEOC problem. At each iteration, the calculation of the optimal step-size for the WEOC problem needs to distinguish four different cases, which is a challenge in comparison with the MVEE problem. We establish the theoretical results of the complexity and the core set size of the rank-two update algorithm for the WEOC problem, which are the generalizations of the currently best-known results for the MEB problem. In addition, by constructing an important inequality for the WEOC problem, we establish the linear convergence of this rank-two update algorithm. Numerical experiments show that the rank-two update algorithm is comparable to the Frank–Wolfe algorithm with away steps for the WEOC problem. In particular, the rank-two update algorithm is more efficient than the Frank–Wolfe algorithm with away steps for problem instances with $$m\gg n$$ under high precision.

Phaseless Terahertz Coded-Aperture Imaging for Sparse Target Based on Phase Retrieval Algorithm.

Phaseless terahertz coded-aperture imaging (PL-TCAI) is a novel radar computational imaging method that utilizes the coded aperture and the incoherent detector array to achieve forward-looking and high-resolution imaging without relying on relative motion. In this paper, we propose a more reasonable and compact architecture for the PL-TCAI system and derive the imaging model of PL-TCAI based on the random frequency-hopping signal. Since most phase retrieval algorithms for PL-TCAI utilize only the intensity of echo signals to accurately reconstruct the target, excessive measurement samples are usually required. In order to reduce the number of measurement samples required for imaging, this paper proposes a sparse Wirtinger flow algorithm with optimal stepsize (SWFOS) by using the sparse prior of the target. The specific procedures of the SWFOS algorithm include the support recovery, initialization by truncated spectral method, iteration via gradient descent scheme, hard threshold operation, and stepsize optimization of iteration. Numerical simulations are performed, and the results show that the SWFOS algorithm not only has good performance for the PR problem, but can also sharply reduce the number of measurement samples required for imaging in the PL-TCAI system.

Two-stage active noise control with online secondary-path filter based on an adapted scheduled-stepsize NLMS algorithm

A two-stage active noise control (ANC) system is proposed for non-stationary environments: a secondary-path filtering (SPF) stage and a control filtering (CF) stage. The secondary-path filter is roughly trained as quickly as possible in the SPF stage. Based on the trained secondary-path filter, the control filter is trained to minimize the residual errors sensed by an error microphone in the CF stage. A stage-switching algorithm is designed to exchange between the SPF stage and the CF stage based only on signals from the error microphone, which moves the CF stage to the SPF stage whenever the residual errors reach up to a certain level in which the control filter cannot suppress the residual errors mainly caused by the change of the secondary path. To train the secondary-path filter and the control filter quickly and robustly, a scheduled-stepsize normalized least mean square (NLMS) algorithm is adapted to handle not only measurement noises but also disturbances mutually generated between the training of the secondary-path filter and that of the control filter. Since the adapted scheduled-stepsize NLMS algorithm presets the optimal stepsizes for each iteration, the proposed ANC system trains quickly the filters without the additional computations and reduces the residual errors over other ANC systems.

Advanced MPPT Algorithm for Distributed Photovoltaic Systems

The basic and adaptive maximum power point tracking algorithms have been studied for distributed photovoltaic systems to maximize the energy production of a photovoltaic (PV) module. However, the basic maximum power point tracking algorithms using a fixed step size, such as perturb and observe and incremental conductance, suffer from a trade-off between tracking accuracy and tracking speed. Although the adaptive maximum power point tracking algorithms using a variable step size improve the maximum power point tracking efficiency and dynamic response of the basic algorithms, these algorithms still have the oscillations at the maximum power point, because the variable step size is sensitive to external factors. Therefore, this paper proposes an enhanced maximum power point tracking algorithm that can have fast dynamic response, low oscillations, and high maximum power point tracking efficiency. To achieve these advantages, the proposed maximum power point tracking algorithm uses two methods that can apply the optimal step size to each operating range. In the operating range near the maximum power point, a small fixed step size is used to minimize the oscillations at the maximum power point. In contrast, in the operating range far from the maximum power point, a variable step size proportional to the slope of the power-voltage curve of PV module is used to achieve fast tracking speed under dynamic weather conditions. As a result, the proposed algorithm can achieve higher maximum power point tracking efficiency, faster dynamic response, and lower oscillations than the basic and adaptive algorithms. The theoretical analysis and performance of the proposed algorithm were verified by experimental results. In addition, the comparative experimental results of the proposed algorithm with the other maximum power point tracking algorithms show the superiority of the proposed algorithm.

Revisiting the numerical solution of stochastic differential equations

Purpose The purpose of this paper is to revisit the numerical solutions of stochastic differential equations (SDEs). An important drawback when integrating SDEs numerically is the number of steps required to attain acceptable accuracy of convergence to the true solution. Design/methodology/approach This paper develops a bias reducing method based loosely on extrapolation. Findings The method is seen to perform acceptably well and for realistic steps sizes provides improved accuracy at no significant additional computational cost. In addition, the optimal step size of the bias reduction methods is shown to be consistent with theoretical analysis. Originality/value Overall, there is evidence to suggest that the proposed method is a viable, easy to implement competitor for other commonly used numerical schemes.

Analytical results for a generalized model of bursty gene expression with molecular memory.

The activation of a gene is a complex biochemical process and could involve small steps, creating a memory between individual events. However, the effect of this molecular memory was often neglected in previous work. How the molecular memory affects gene expression remains not fully explored. We analyze a stochastic model of bursty gene expression, where the waiting time from inactivation to activation is assumed to follow a nonexponential (in fact, Erlang) distribution. We derive the analytical expression for the gene-product distribution, which explicitly traces the effect of molecule memory. Interestingly, we find that the effect of molecular memory is equivalent to the introduction of feedback. In addition, we analytically show that the stationary distribution is always super-Poissonian, independent of the detail of the waiting-time distribution, and there is the optimal step size that minimizes the Fano factor for any given mean burst size and is a decreasing function of the mean burst size. These analytical results indicate that molecular memory is an unneglectable factor affecting gene expression.

Optimized composite finite difference schemes for atmospheric flow modeling

AbstractIn this paper, we use some finite difference methods in order to solve an atmospheric flow problem described by an advection–diffusion equation. This flow problem was solved by Clancy using forward‐time central space (FTCS) scheme and is challenging to simulate due to large errors in phase and amplitude which are generated especially over long propagation times. Clancy also derived stability limits for FTCS scheme. We use Von Neumann stability analysis and the approach of Hindmarsch et al. which is an improved technique over that of Clancy in order to obtain the region of stability of some methods such as FTCS, Lax–Wendroff (LW), Crank–Nicolson. We also construct a nonstandard finite difference (NSFD) scheme. Properties like stability and consistency are studied. To improve the results due to significant numerical dispersion or numerical dissipation, we derive a new composite scheme consisting of three applications of LW followed by one application of NSFD. The latter acts like a filter to remove the dispersive oscillations from LW. We further improve the composite scheme by computing the optimal temporal step size at a given spatial step size using two techniques namely; by minimizing the square of dispersion error and by minimizing the sum of squares of dispersion and dissipation errors.

Tracking Analysis of Minimum Kernel Risk-Sensitive Loss Algorithm Under General Non-Gaussian Noise

In this brief, the steady-state tracking performance of minimum kernel risk-sensitive loss in a non-stationary environment is analyzed. In order to model a non-stationary environment, a first-order random-walk model is used to describe the variations of optimum weight vector over time. Moreover, the measurement noise is considered to have non-Gaussian distribution. The energy conservation relation is utilized to extract an approximate closed-form expression for the steady-state excess mean square error (EMSE). Our analysis shows that unlike for the stationary case, the EMSE curve is not an increasing function of step-size parameter. Hence, the optimum step-size which minimizes the EMSE is derived. We also discuss that our approach can be used to extract steady-state EMSE for a general class of adaptive filters. The simulation results with different noise distributions support the theoretical derivations.

A dual boundary element based implicit differentiation method for determining stress intensity factor sensitivities for plate bending problems

A novel methodology for determining Stress Intensity Factor (SIF) sensitivities for plate bending problems using the Dual Boundary Element Method (DBEM) is presented. The direct derivatives of the DBEM integral equations for plate bending have been derived for the first time and are used as part of a DBEM-based Implicit Differentiation Method (IDM or DBEM-IDM) for calculating the sensitivities of SIFs to changes in different geometric parameters such as crack length and crack rotation angle. The SIFs and their sensitivities are calculated using the J-integral and the derivative of the J-integral respectively. A numerical example featuring a thick plate subjected to membrane, bending, and pressure loads is presented. In the first half of the numerical example, the SIF sensitivities from the IDM are compared with those obtained from the more common, but relatively crude, Finite Difference Method (FDM or DBEM-FDM). Results show that the IDM is a significantly more efficient and robust alternative to the FDM. The accuracy of the FDM showed significant dependence on the step size used, necessitating a time-consuming optimisation procedure to determine the optimal step size. Once this optimal step size was found, both methods provided very similar results. As part of the second half of the numerical example, a demonstration of one possible application of the SIF sensitivities from the IDM is presented. This involved carrying out reliability analyses using the First-Order Reliability Method (FORM) with a large number of design variables.

Forecasting air pollution PM2.5 in Beijing using weather data and multiple kernel learning

AbstractPM2.5 mass concentration prediction is an important research issue because of the increasing impact of air pollution on the urban environment. In this paper, a PM2.5 forecasting framework incorporating meteorological factors based on multiple kernel learning (MKL) is proposed to forecast the near future PM2.5. In addition, we develop a novel two‐step algorithm for solving the primal MKL problem. Compared with most existing MKL 2‐step algorithms, the proposed algorithm does not require the optimal step size for updating kernel combination coefficients by linear search. To demonstrate the performance of the proposed forecasting framework, its performance is compared to single kernel‐based support vector regression (SVR). Data sets of an inland city Beijing acquired from UCI are used to train and validate both of two methods. Experiments show that our proposed method outperforms the SVR.