Stochastic Approximation Algorithm Research Articles

Finite-Time High-Probability Bounds for Polyak–Ruppert Averaged Iterates of Linear Stochastic Approximation

This paper provides a finite-time analysis of linear stochastic approximation (LSA) algorithms with fixed step size, a core method in statistics and machine learning. LSA is used to compute approximate solutions of a d-dimensional linear system [Formula: see text] for which [Formula: see text] can only be estimated by (asymptotically) unbiased observations [Formula: see text]. We consider here the case where [Formula: see text] is an a sequence of independent and identically distributed random variables sequence or a uniformly geometrically ergodic Markov chain. We derive pth moment and high-probability deviation bounds for the iterates defined by LSA and its Polyak–Ruppert-averaged version. Our finite-time instance-dependent bounds for the averaged LSA iterates are sharp in the sense that the leading term we obtain coincides with the local asymptotic minimax limit. Moreover, the remainder terms of our bounds admit a tight dependence on the mixing time [Formula: see text] of the underlying chain and the norm of the noise variables. We emphasize that our result requires the LSA step size to scale only with logarithm of the problem dimension d. Funding: The work of A. Durmus and E. Moulines was partly supported by [Grant ANR-19-CHIA-0002]. This project received funding from the European Research Council [ERC-SyG OCEAN Grant 101071601]. The research of A. Naumov and S. Samsonov was prepared within the framework of the HSE University Basic Research Program.

Consensus of networked nonlinear systems based on the distributed stochastic approximation algorithm

Consensus of networked nonlinear nonparametric time-varying systems is studied, and the nonlinearity at input may possess arbitrary growth rate. In addition, the communication noise is also under consideration. The purely distributed control algorithm is designed on the basis of distributed stochastic approximation algorithm with expanding truncations (DSAAWET). The truncation mechanism neutralizes the divergent tendency caused by unknown nonlinearities and noises, which makes the algorithm well tackle more general nonlinearities with high growth rate and more complicated structure noises. We first prove the average consensus over fixed digraph, and then extend the results to averagely connected time-varying digraphs. Finally, the validity of the algorithm is justified by numerical simulations.

HK-SPSA based performance optimization method for steam generator liquid level control

Steam generator level control systems play a crucial role in ensuring safe, economical, and stable operation of nuclear power plants. However, traditional control systems exhibit limited efficiency in commissioning and are susceptible to human error. Additionally, traditional parameter tuning methods are typically experienced-based, cumbersome, and time-consuming, making it challenging to obtain optimal parameters. To address these issues, we propose a hybrid knowledge-guided improved simultaneous perturbation stochastic approximation algorithm (HK-SPSA) to optimize the control parameters of steam generator level control systems. Firstly, the algorithm utilizes iteration point adjacency information to approximate the current optimization process status and guide the adaptive adjustment of the iteration step size. Secondly, it utilizes composite gradient information generated by the estimated historical and current gradients to guide the optimization direction and the tuning of iteration step size. Simulation experiments have shown that HK-SPSA improves optimization performance and reduces the costs of optimization and adjustment compared to traditional methods.

Optimization of CO2 EOR and geological sequestration in high-water cut oil reservoirs

AbstractIn terms of the collaborative optimization of CO2 flooding for Enhanced Oil Recovery (EOR) and CO2 sequestration, previous studies have co-optimized both cumulative oil production and CO2 sequestration by various algorithms. However, these solutions fail to optimize the CO2 injection schemes for high-water cut oil reservoirs. This paper presents an optimization methodology for CO2 flooding and sequestration in high-water cut oil reservoirs. The production optimization was carried out by adjusting the injection and production rate. To solve the proposed objective functions, the simultaneous perturbation stochastic approximation (SPSA) algorithm is applied in this paper, and the CMG-GEM module is utilized to simulate the reservoir production performance. A typical high-water cut reservoir in the Shengli oilfield was used to verify the feasibility of the presented methodology. In this paper, the production performance and net present value (NPV) for continuous gas injection under different water cuts were analyzed. The optimal timing of transforming from water flooding to gas displacement for the high-water cut reservoir was optimized. In addition, the optimal water–gas ratios for Water-Alternating-Gas (WAG) flooding were determined. The sensitivity of NPV to gas injection price and carbon subsidy was analyzed. The results show that when the gas price is 0.178 $/m3 and the carbon subsidy is 0.0169 $/m3, the optimal timing of transforming from water flooding to gas injection should be earlier than the time when the water cut is 0.82. Through the combination of NPV, cumulative oil production rate, and CO2 sequestration volume for WAG flooding, the optimal WAG ratio should be 1:2. The presented method in this paper considers various economic indicators and can optimize CO2 flooding and sequestration in high-water cut oil reservoirs efficiently, which can provide some guidance for the design of CO2 flooding schemes in high-water cut oil reservoirs.

Data-Driven Dynamic Inversion Method for Complex Fractures in Unconventional Reservoirs

Abstract Hydraulic fracturing is a crucial technology for enhancing the recovery of oil and gas from unconventional reservoirs. Accurately describing fracture morphology is essential for accurately predicting production dynamics. This article proposes a new fracture inversion model based on dynamic data-driven methods, which is different from the conventional linear elastic fracture mechanics model. This method eliminates the need to consider complex mechanical mechanisms, resulting in faster simulation speeds. In the model, the fracture morphology is constrained by combining microseismic data and fracturing construction data, and the fracture tip propagation domain is introduced to characterize the multi-directionality of fracture propagation. The simulated fracture exhibits a multi-branch fracture network morphology, aligning more closely with geological understanding. In addition, the influence of microseismic signal intensity on the direction of fracture propagation is considered in this study. The general stochastic approximation (GSA) algorithm is employed to optimize the direction of fracture propagation. The proposed method is applied to both the single-stage fracturing model and the whole well fracturing model. The research findings indicate that in the single-stage fracturing model, the inverted fracture morphology aligns closely with the microseismic data, with a fitting rate of the fracturing construction curve exceeding 95%, and a microseismic data fitting rate exceeding 93%. In the whole well fracturing model, a total of 18 sections were inverted. The fitting rate between the overall fracture morphology and the microseismic data reached 90%. The simulation only took 5 minutes, demonstrating high computational efficiency and meeting the needs of large-scale engineering fracture simulation. This method can effectively support geological modeling and production dynamic prediction.

Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inference

In this paper, we study the effectiveness of using a constant stepsize in statistical inference via linear stochastic approximation (LSA) algorithms with Markovian data. After establishing a Central Limit Theorem (CLT), we outline an inference procedure that uses averaged LSA iterates to construct confidence intervals (CIs). Our procedure leverages the fast mixing property of constant-stepsize LSA for better covariance estimation and employs Richardson-Romberg (RR) extrapolation to reduce the bias induced by constant stepsize and Markovian data. We develop theoretical results for guiding stepsize selection in RR extrapolation, and identify several important settings where the bias provably vanishes even without extrapolation. We conduct extensive numerical experiments and compare against classical inference approaches. Our results show that using a constant stepsize enjoys easy hyperparameter tuning, fast convergence, and consistently better CI coverage, especially when data is limited.

Quantile Optimization via Multiple-Timescale Local Search for Black-Box Functions

Black-Box Quantile Optimization via Finite-Difference-Based Gradient Approximation Risk management necessitates consideration of metrics such as quantiles to supplement conventional mean performance measures. In “Quantile Optimization via Multiple-Timescale Local Search for Black-Box Functions,” J. Hu, M. Song, and M. C. Fu consider the problem where the goal is to optimize the quantile of a black-box output. They introduce two new iterative multitimescale stochastic approximation algorithms utilizing finite-difference-based gradient estimators. The first algorithm requires 2d + 1 samples of the black-box function per iteration, where d is the number of decision variables. The second employs a simultaneous-perturbation-based gradient estimator that uses only three samples per iteration, irrespective of the number of decision variables. The authors prove strong local convergence for both algorithms and analyze their finite-time convergence rates through a novel fixed-point argument. These algorithms perform well across a varied set of benchmark problems.

Technical Note—On the Convergence Rate of Stochastic Approximation for Gradient-Based Stochastic Optimization

Finite-Time Performance of Gradient-Based Stochastic Approximation Algorithms Stochastic approximation (SA) algorithms are among the most commonly used methods for addressing stochastic optimization problems. In “Technical Note: On the Convergence Rate of Stochastic Approximation for Gradient-Based Stochastic Optimization,” J. Hu and M. C. Fu provide a new error bound for SA algorithms when applied to a class of problems with convex differentiable structures. The bound allows a detailed characterization of an algorithm’s finite-time performance through directly analyzing the bias and variance of the employed gradient estimator. The main result is then applied to study and compare the efficiency of traditional Kiefer-Wolfowitz algorithms and those based on random direction finite-difference gradient estimators such as simultaneous perturbation SA. The analysis leads to new insights into when it may be advantageous to use such randomized gradient estimates under various problem and algorithm parameter settings.

A Hybrid Energy Storage System Integrated with a Wave Energy Converter: Data-Driven Stochastic Power Management for Output Power Smoothing

Beyond solar and wind energy, wave energy is gaining great interest due to its very high theoretical potential, although its stochastic nature causes intermittent and fluctuating power production. Energy storage system (ESS) integration to wave energy converter (WEC) plants represents a promising solution to mitigate this issue. To overcome the technological limits of the single storage devices, the hybridization of complementary ESSs represents an effective solution, extending the operating range over different timeframes. This paper analyzes the benefits of Li-ion battery–supercapacitor hybrid ESS integration into a grid-connected WEC, aiming at smoothing the produced power oscillations. The hybridization concept involves coupling a power-intensive technology, such as a supercapacitor devoted to managing fluctuations at higher frequency, with a battery technology exploited to manage power variations over longer timeframes to mitigate degradation issues. In this study, a multi-objective data-driven power management strategy, based on the simultaneous perturbation stochastic approximation (SPSA) algorithm, is implemented to minimize power fluctuations in terms of power ramp (representing the power variation between two consecutive values with a 1 s time step), both at the Point of Common Coupling (PCC) and the Li-ion battery terminals, thanks to the supercapacitor peak-shaving function. The SPSA management strategy, together with a suitable sizing procedure, allows a reduction of more than 70% in the power oscillations at the PCC with respect to those at the WEC terminals, while decreasing battery stress by more than 25% if compared to a non-hybrid ESS consisting of a Li-ion battery. This shows how supercapacitor features can extend battery lifespan when integrated in a hybrid ESS.

Two-stage appointment scheduling considering patient foldback under a stochastic approximation approach

Patient foldback frequently disrupts clinical operations, reduces productivity, and leads to patient congestion. To mitigate these negative impacts, this study aims to optimize appointment scheduling for a series of patients within service systems. These systems are defined by a two-stage service process, the occurrence of patient foldback behavior, and stochastic service durations. To achieve this objective, we propose a stochastic programming model to minimize the expected weighted sum of costs associated with physician idle time and patient wait times. We examine the characteristics of the sample path function and propose a stochastic approximation (SA) algorithm to address the problem effectively. Numerical experiments indicate that the proposed SA algorithm significantly decreases the solution time compared to the sample average approximation (SAA)-based algorithm, thereby confirming the efficiency and effectiveness of the proposed algorithm. Moreover, the analysis of the optimal appointment schedule characteristics, while considering patient foldback, reveals that the optimal job allowances retain a ‘dome’ shape; however, there is a marked steep increase at the start of the curve.

Automatic History Matching for Adjusting Permeability Field of Fractured Basement Reservoir Simulation Model Using Seismic, Well Log, and Production Data

Developing automatic history matching (AHM) methods to replace the traditional manual history matching (MHM) approach in adjusting the permeability distribution of the reservoir simulation model has been studied by many authors. Because permeability values need to be evaluated at hundreds of thousands of grid cells in a typical reservoir simulation model, it is necessary to apply a reparameterization technique to allow the optimization algorithms to be implemented with fewer variables. In basic reparameterization techniques including zonation and pilot point methods, the calibrations are usually based solely on the production data with no systematic link to the geological and geophysical data, and therefore, the obtained permeability distribution may be not geologically consistent. Several other reparameterization techniques have attempted to preserve geological consistency by incorporating 4D seismic data; however, these techniques cannot be applied to our fractured basement reservoirs (FBRs) as they do not have 4D seismic data. Taking into account these challenges, in this study, an AHM methodology and workflow have been developed using a new reparameterization technique. This approach attempts to minimize the potential for geological nonconsistency of the calibrated results by linking the permeability to geophysical data. The proposed methodology can be applied to fields with only traditional geophysical data (3D seismic and conventional well logs). In the proposed workflow, the spatial distributions of seismic attributes and geomechanical properties were calculated and estimated from 3D seismic data and well logs, respectively. After that, a feed-forward artificial neural network (ANN) model trained by the back-propagation algorithm of the relationship between initial permeability with seismic attributes and geomechanical properties of their grid cell values is developed. Then, the calibration of the permeability distribution is performed by adjustment of the ANN model. Modification of the ANN model is performed using the simultaneous perturbation stochastic approximation (SPSA) algorithm to calibrate transmission coefficients in the ANN model to minimize the discrepancy between the simulated results and observed data. The developed methodology is applied to calibrate the permeability distribution of a simulation model of Bach Ho FBR in Vietnam. The effectiveness of the methodology is evident by comparing the historical matches with an available manually history-matched simulation model. The application shows that the proposed methodology could be considered as a suitable practical approach for adjusting the permeability distribution for FBR reservoir simulation models.

A Modified Simultaneous Perturbation Stochastic Approximation Algorithm for Active Noise Control

Secondary path modeling is an important step in active noise control. Keeping the active noise control system stable, when the secondary path changes over time, is a widely studied problem. Using the simultaneous perturbation stochastic approximation algorithm, can update the control filter without secondary path modeling, avoiding the problem of increased noise caused by secondary path modeling. In this paper a variant of the simultaneous perturbation stochastic approximation algorithm is proposed, which incorporates historical coefficients of the control filter, to correct the direction of the gradient calculation. Based on an experimental setup with a realistic transfer path, this paper demonstrates that the proposed algorithm has superior performance in terms of convergence speed and noise reduction amount, than the general simultaneous perturbation stochastic approximation algorithm.

Finite-time error bounds for distributed linear stochastic approximation

This paper considers a novel multi-agent linear stochastic approximation algorithm driven by Markovian noise and general consensus-type interaction, in which each agent evolves according to its local stochastic approximation process which depends on the information from its neighbors. The interconnection structure among the agents is described by a time-varying directed graph. While the convergence of consensus-based stochastic approximation algorithms when the interconnection among the agents is described by doubly stochastic matrices (at least in expectation) has been studied, less is known about the case when the interconnection matrix is simply stochastic. For any uniformly strongly connected graph sequences whose associated interaction matrices are stochastic, the paper derives finite-time bounds on the mean-square error, defined as the deviation of the output of the algorithm from the unique equilibrium point of the associated ordinary differential equation. For the case of interconnection matrices being stochastic, the equilibrium point can be any unspecified convex combination of the local equilibria of all the agents in the absence of communication. Both the cases with constant and time-varying step-sizes are considered. In the case when the convex combination is required to be a straight average and interaction between any pair of neighboring agents may be uni-directional, so that doubly stochastic matrices cannot be implemented in a distributed manner, the paper proposes a push-sum-type distributed stochastic approximation algorithm and provides its finite-time bound for the time-varying step-size case by leveraging the analysis for the consensus-type algorithm with stochastic matrices and developing novel properties of the push-sum algorithm. Distributed temporal difference learning is discussed as an illustrative application.

A Lyapunov Theory for Finite-Sample Guarantees of Markovian Stochastic Approximation

The stochastic approximation (SA) method stands as the foundational mathematical tool for modern large-scale optimization and machine learning. Therefore, gaining a theoretical understanding of SA algorithms is of fundamental interest. In their paper titled “A Lyapunov Theory for Finite-Sample Guarantees of Markovian Stochastic Approximation,” Chen et al. present a unified Lyapunov framework for the finite-sample analysis of a Markovian SA algorithm under a contractive operator with respect to an arbitrary norm. The key novelty lies in the construction of a smooth Lyapunov function called the generalized Moreau envelope. The authors demonstrate the effectiveness of their SA results in the context of reinforcement learning (RL), specifically through popular algorithms such as variants of temporal difference (TD) learning and Q-learning. As byproducts, the results provide theoretical insights into the efficiency of bootstrapping in TD learning with eligibility traces and the bias-variance tradeoff in off-policy learning.

Online Deterministic Annealing for Classification and Clustering.

Inherent in virtually every iterative machine learning algorithm is the problem of hyperparameter tuning, which includes three major design parameters: 1) the complexity of the model, e.g., the number of neurons in a neural network; 2) the initial conditions, which heavily affect the behavior of the algorithm; and 3) the dissimilarity measure used to quantify its performance. We introduce an online prototype-based learning algorithm that can be viewed as a progressively growing competitive-learning neural network architecture for classification and clustering. The learning rule of the proposed approach is formulated as an online gradient-free stochastic approximation algorithm that solves a sequence of appropriately defined optimization problems, simulating an annealing process. The annealing nature of the algorithm contributes to avoiding poor local minima, offers robustness with respect to the initial conditions, and provides a means to progressively increase the complexity of the learning model, through an intuitive bifurcation phenomenon. The proposed approach is interpretable, requires minimal hyperparameter tuning, and allows online control over the performance-complexity tradeoff. Finally, we show that Bregman divergences appear naturally as a family of dissimilarity measures that play a central role in both the performance and the computational complexity of the learning algorithm.

Stochastic search for a parametric cost function approximation: Energy storage with rolling forecasts

Rolling forecasts have been almost overlooked in the renewable energy storage literature. In this paper, we provide a new approach for handling uncertainty not just in the accuracy of a forecast, but in the evolution of forecasts over time. Our approach shifts the focus from modeling the uncertainty in a lookahead model to accurate simulations in a stochastic base model. We develop a robust policy for making energy storage decisions by creating a parametrically modified lookahead model, where the parameters are tuned in the stochastic base model. Since computing unbiased stochastic gradients with respect to the parameters require restrictive assumptions, we propose a simulation-based stochastic approximation algorithm based on numerical derivatives to optimize these parameters. While numerical derivatives, calculated based on the noisy function evaluations, provide biased gradient estimates, an online variance reduction technique built in the framework of our proposed algorithm, will enable us to control the accumulated bias errors and establish the finite-time rate of convergence of the algorithm. Our numerical experiments show the performance of this algorithm in finding policies outperforming the deterministic benchmark policy.

Secrecy Rate Maximization of RIS-Assisted SWIPT Systems: A Two-Timescale Beamforming Design Approach

Reconfigurable intelligent surfaces (RISs) achieve high passive beamforming gains for signal enhancement or interference nulling by dynamically adjusting their reflection coefficients. Their employment is particularly appealing for improving both the wireless security and the efficiency of radio frequency (RF)-based wireless power transfer. Motivated by this, we conceive and investigate a RIS-assisted secure simultaneous wireless information and power transfer (SWIPT) system designed for information and power transfer from a base station (BS) to an information user (IU) and to multiple energy users (EUs), respectively. Moreover, the EUs are also potential eavesdroppers that may overhear the communication between the BS and IU. We adopt <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">two-timescale</i> transmission for reducing the signal processing complexity as well as channel training overhead, and aim for maximizing the average worst-case secrecy rate achieved by the IU. This is achieved by jointly optimizing the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">short-term</i> transmit beamforming vectors at the BS (including information and energy beams) as well as the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">long-term</i> phase shifts at the RIS, under the energy harvesting constraints considered at the EUs and the power constraint at the BS. The stochastic optimization problem formulated is non-convex with intricately coupled variables, and is non-smooth due to the existence of multiple EUs/eavesdroppers. No standard optimization approach is available for this challenging scenario. To tackle this challenge, we propose a smooth approximation aided stochastic successive convex approximation (SA-SSCA) algorithm. Furthermore, a low-complexity heuristic algorithm is proposed for reducing the computational complexity without unduly eroding the performance. Simulation results show the efficiency of the RIS in securing SWIPT systems. The significant performance gains achieved by our proposed algorithms over the relevant benchmark schemes are also demonstrated.

Approximate Kalman filtering by both M-robustified dynamic stochastic approximation and statistical linearization methods

The problem of designing a robustified Kalman filtering technique, insensitive to spiky observations, or outliers, contaminating the Gaussian observations has been presented in the paper. Firstly, a class of M-robustified dynamic stochastic approximation algorithms is derived by minimizing at each stage a specific time-varying M-robust performance index, that is, general for a family of algorithms to be considered. The gain matrix of a particular algorithm is calculated at each stage by minimizing an additional criterion of the approximate minimum variance type, with the aid of the statistical linearization method. By combining the proposed M-robust estimator with the one-stage optimal prediction, in the minimum mean-square error sense, a new statistically linearized M-robustified Kalman filtering technique has been derived. Two simple practical versions of the proposed M-robustified state estimator are derived by approximating the mean-square optimal statistical linearization coefficient with the fixed and the time-varying factors. The feasibility of the approaches has been analysed by the simulations, using a manoeuvring target radar tracking example, and the real data, related to an object video tracking using short-wave infrared camera.

Neural networks singular evolutive interpolated Kalman filter and its application to data assimilation for 2D water pollution model

Abstract The present study promotes a new algorithm for estimating the water pollution propagation with the primary goal of providing more reliable and high quality estimates to decision makers. To date, the widely used variational method suffers from the large computational burden, which limits its application in practice. Moreover, this method, considering the initial state as a control variable, is very sensitive in specifying initial error, especially for unstable dynamical systems. The Neural Network Filter (NNF), proposed in the present paper, is aimed at overcoming these two drawbacks in the variational method: by its nature, the NNF is sequential (no batch large assimilation window used) and stable even for unstable dynamics, with the gain parameters as control variables. The NNF, developed in the present paper, is a Neural Network Filter (NNF) version of the Singular Evolutive Interpolated Kalman Filter (SEIKF). One of the new versions of this NNF is that it uses structure of the gain of SEIKF0 taken by the SEIKF at the first time moment of correction process. To deal with the uncertainty of the system parameters and of the noise covariance, the proposed Neural Network SEIKF0 named by NNSEIKF0 makes use of the covariance of a reduced rank iterated during assimilation process and of some pertinent gain parameters tuned adaptively to yield the minimum prediction error for the system output. The computational burden in implementation of the NNSEIKF0 is reduced drastically due to applying the optimization tool known as a simultaneous perturbation stochastic approximation (SPSA) algorithm, which requires only two integrations of the numerical model. No iterative loop is required at each assimilation instant as usually happens with the standard gradient descent optimization algorithms. Data assimilation experiment, carried out by the SEIKF0 and NNSEIKF0, is implemented for the Thanh Nhan Lake in Hanoi and the performance comparison between the NNSEIKF0 and SEIKF0 is given to show the high efficiency of the proposed NNSEIKF0.

Simultaneous perturbation stochastic approximation: towards one-measurement per iteration

When measuring the value of a function to be minimized is not only expensive but also with noise, the popular simultaneous perturbation stochastic approximation (SPSA) algorithm requires only two function values in each iteration. In this paper, we present a method requiring only one function measurement value per iteration in the average sense. We prove the strong convergence and asymptotic normality of the new algorithm. Limited experimental results demonstrate the effectiveness and potential of our algorithm for solving low-dimensional problems.