Stochastic Approximation Framework Research Articles

Stochastic Approximation for Multi-period Simulation Optimization with Streaming Input Data

We consider a continuous-valued simulation optimization (SO) problem, where a simulator is built to optimize an expected performance measure of a real-world system while parameters of the simulator are estimated from streaming data collected periodically from the system. At each period, a new batch of data is combined with the cumulative data and the parameters are re-estimated with higher precision. The system requires the decision variable to be selected in all periods. Therefore, it is sensible for the decision-maker to update the decision variable at each period by solving a more precise SO problem with the updated parameter estimate to reduce the performance loss with respect to the target system. We define this decision-making process as the multi-period SO problem and introduce a multi-period stochastic approximation (SA) framework that generates a sequence of solutions. Two algorithms are proposed: Re-start SA ( ReSA ) reinitializes the stepsize sequence in each period, whereas Warm-start SA ( WaSA ) carefully tunes the stepsizes, taking both fewer and shorter gradient-descent steps in later periods as parameter estimates become increasingly more precise. We show that under suitable strong convexity and regularity conditions, ReSA and WaSA achieve the best possible convergence rate in expected sub-optimality either when an unbiased or a simultaneous perturbation gradient estimator is employed, while WaSA accrues significantly lower computational cost as the number of periods increases. In addition, we present the regularized ReSA which obviates the need to know the strong convexity constant and achieves the same convergence rate at the expense of additional computation.

A unified stochastic approximation framework for learning in games

A unified stochastic approximation framework for learning in games

Blind Post-Decision State-Based Reinforcement Learning for Intelligent IoT

Recent years have witnessed a renewed interest in reinforcement learning (RL) due to the rapid growth of the Internet-of-Things (IoT) and their associated intelligent information processing and decision-making demands. As the slow learning speed is one of the major stumbling blocks of the classic RL algorithms, substantial efforts have been devoted to developing faster RL algorithms. Among them, post-decision state (PDS) learning is a prominent one, which can often improve the learning speed by orders of magnitude by exploiting the structural property of the underlying Markov decision processes (MDPs). However, conventional PDS learning requires prior information about the PDS transition probability, which may not be always available in practice. To lift this limitation, a novel blind PDS (b-PDS) learning algorithm is proposed in this work by leveraging the generic two-timescale stochastic approximation framework. By introducing an extra estimating procedure about the PDS transition probability, b-PDS learning can achieve a similar improvement of learning speed as conventional PDS learning while excluding the need for prior information. In addition, by analyzing the globally asymptotically stable equilibrium of the corresponding ordinary differential equation (o.d.e.), the convergence and optimality of b-PDS learning are established. Moreover, extensive simulation results are provided to validate the effectiveness of the proposed algorithm. Over the considered random MDPs, it has been observed that, to reach 90% of the best possible time average reward, the proposed b-PDS learning can reduce the learning time by 70% compared to Q-learning and 30% compared to Dyna.

A new regularized stochastic approximation framework for stochastic inverse problems

We study the nonlinear inverse problem of estimating stochastic parameters in the fourth-order partial differential equation with random data. The primary focus is on developing a novel stochastic approximation framework for inverse problems consisting of three key components. As a first step, we reformulate the inverse problem into a stochastic convex optimization problem. The second step includes developing a new regularized stochastic extragradient framework for a nonlinear variational inequality, which subsumes the optimality conditions for the optimization formulation of the inverse problem. The third step involves modeling random variables by a Karhunen–Loève type finite-dimensional noise representation, allowing the direct and the inverse problems to be conveniently discretized. We show that the regularized extragradient methods are strongly convergent in a Hilbert space setting, and we also provide several auxiliary results for the inverse problem, including Lipschitz continuity and a derivative characterization of the solution map. We provide the outcome of computational experiments to estimate stochastic and deterministic parameters. The numerical results demonstrate the feasibility and effectiveness of the developed framework and validate stochastic approximation as an effective method for stochastic inverse problems.

A stochastic approximation approach to simultaneous feature weighting and selection for nearest neighbour learners

Nearest neighbour (NN) learners are some of the most popular methods in supervised machine learning with feature selection and weighting being two fundamental tools for improving their performance. In this study, we introduce a novel methodology for simultaneous feature selection and weighting for NN learners based on simultaneous perturbation stochastic approximation (SPSA). This is a pseudo-gradient descent optimisation algorithm that approximates gradient information from noisy objective function measurements without a need for an explicit functional form of the objective function nor its derivatives. In particular, we show how the process of simultaneous feature selection and weighting can be optimised within a stochastic approximation framework with repeated cross-validation (CV) performance as the objective function, which we call SPSA-FWS. We provide extensive computational experiments for assessment of this approach and we compare performance of SPSA-FWS to other feature weighting methods. Our results indicate that SPSA-FWS outperforms existing feature weighting algorithms for the most part and it stands as a competitive new method for this task. Specifically, when compared against its unweighted counterpart and feature weighting alone with 5-repeated 5-fold CV accuracy being the performance metric, SPSA-FWS provides improvements of up to 177.88% with a mean improvement of 20.52% for classification tasks and a mean improvement of 0.19 in the R-Squared metric for regression tasks respectively. In addition to its superior performance, SPSA-FWS has two attractive features: (1) it can be used in conjunction with any performance metric and any variant of nearest neighbour learners, and (2) it can be hybridised with other feature weighting methods.

Topology Optimization With Many Right-Hand Sides Using Mirror Descent Stochastic Approximation—Reduction From Many to a Single Sample

Abstract Practical engineering designs typically involve many load cases. For topology optimization with many deterministic load cases, a large number of linear systems of equations must be solved at each optimization step, leading to an enormous computational cost. To address this challenge, we propose a mirror descent stochastic approximation (MD-SA) framework with various step size strategies to solve topology optimization problems with many load cases. We reformulate the deterministic objective function and gradient into stochastic ones through randomization, derive the MD-SA update, and develop algorithmic strategies. The proposed MD-SA algorithm requires only low accuracy in the stochastic gradient and thus uses only a single sample per optimization step (i.e., the sample size is always one). As a result, we reduce the number of linear systems to solve per step from hundreds to one, which drastically reduces the total computational cost, while maintaining a similar design quality. For example, for one of the design problems, the total number of linear systems to solve and wall clock time are reduced by factors of 223 and 22, respectively.

Demand Calibration of Multimodal Microscopic Traffic Simulation using Weighted Discrete SPSA

This paper presents a stochastic approximation framework to solve a generalized problem of off-line calibration of demand for a multimodal microscopic (or mesoscopic) network simulation using aggregated sensor data. A key feature of this problem is that demand, although typically treated as a continuous variable is in fact discrete, particularly in the context of agent-based simulation. To address this, we first use a discrete version of the weighted simultaneous perturbation stochastic approximation (W-DSPSA) algorithm for minimizing a generalized least squares (GLS) objective (that measures the distance between simulated and observed measurements), defined over discrete sets. The algorithm computes the gradient at each iteration using a symmetric discrete perturbation of the calibration parameters and a multimodal weight matrix to improve the accuracy of the gradient estimate. The W-DSPSA algorithm is then applied to the large-scale calibration of multimodal origin–destination (OD) flows (including private vehicle (PVT) and public transit (PT) trips) in a microscopic network simulation model of Singapore. The results indicate that an acceptable margin of error on the vehicle loop count (VLC) and bus passenger count (BPC) are achieved at convergence with an improvement of 60%~80% in root mean squared errors. Lastly, we validate the calibration results with observed travel times on the network. Statistical comparison shows good agreements on both point-to-point travel time (PTT) and public buses’ stop-to-stop ride-time (SRT) with the field observations.

Stochastic approximation algorithms for rumor source inference on graphs

We revisit the classical problem of identifying the source of rumor in a network, assumed unique, treating as given the network topology and the set of rumor infected nodes. In addition, it is assumed that some partial information about the order in which the nodes were infected, is also available. Such form of information is commonly available to network monitoring agencies. Under this premise, we propose a class of estimators that is agnostic to the underlying stochastic model of rumor spread and further, generalizes other estimators popular in literature, e.g., rumor center, distance center and Jordan center. We also develop an MCMC-based stochastic approximation framework to implement this class of estimators. Results from extensive simulations on the Erdős–Rényi and Barabási–Albert random graph models indicate promising empirical performance and robustness of the proposed estimators to different graph topologies. Our stochastic approximation framework also extends easily to general statistical inference problems on graphs that are of combinatorial nature. In particular, we demonstrate a suitable modification of our framework that makes it amenable to be used for the popular problem of rank aggregation from noisy pairwise comparisons.

Distributed multiple access with a general link layer channel

This paper investigates the problem of distributed medium access control in a time slotted wireless multiple access network with an unknown finite number of homogeneous users. Assume that each user has a single transmission option. In each time slot, a user chooses either to idle or to transmit a packet. Under a general channel model, a distributed medium access control framework is proposed to adapt transmission probabilities of all users to a value that maximizes an arbitrarily chosen symmetric network utility. Probability target of each user in the proposed algorithm is calculated based upon a channel contention measure, which is defined as the success probability of a virtual packet. It is shown that the proposed algorithm falls into the classical stochastic approximation framework with guaranteed convergence when the contention measure can be directly obtained from the receiver. On the other hand, when the contention measure is not directly available, computer simulations show that a revised medium access control algorithm can still help the system to converge to the same designed equilibrium.

Learning Trans-Dimensional Random Fields with Applications to Language Modeling.

To describe trans-dimensional observations in sample spaces of different dimensions, we propose a probabilistic model, called the trans-dimensional random field (TRF) by explicitly mixing a collection of random fields. In the framework of stochastic approximation (SA), we develop an effective training algorithm, called augmented SA, which jointly estimates the model parameters and normalizing constants while using trans-dimensional mixture sampling to generate observations of different dimensions. Furthermore, we introduce several statistical and computational techniques to improve the convergence of the training algorithm and reduce computational cost, which together enable us to successfully train TRF models on large datasets. The new model and training algorithm are thoroughly evaluated in a number of experiments. The word morphology experiment provides a benchmark test to study the convergence of the training algorithm and to compare with other algorithms, because log-likelihoods and gradients can be exactly calculated in this experiment. For language modeling, our experiments demonstrate the superiority of the TRF approach in being computationally more efficient in computing data probabilities by avoiding local normalization and being able to flexibly integrate a richer set of features, when compared with n-gram models and neural network models.

Nonparametric stochastic approximation with large step-sizes

We consider the random-design least-squares regression problem within the reproducing kernel Hilbert space (RKHS) framework. Given a stream of independent and identically distributed input/output data, we aim to learn a regression function within an RKHS $\mathcal{H}$, even if the optimal predictor (i.e., the conditional expectation) is not in $\mathcal{H}$. In a stochastic approximation framework where the estimator is updated after each observation, we show that the averaged unregularized least-mean-square algorithm (a form of stochastic gradient), given a sufficient large step-size, attains optimal rates of convergence for a variety of regimes for the smoothnesses of the optimal prediction function and the functions in $\mathcal{H}$.

Fast Online EM for Big Topic Modeling

The expectation-maximization (EM) algorithm can compute the maximum-likelihood (ML) or maximum a posterior (MAP) point estimate of the mixture models or latent variable models such as latent Dirichlet allocation (LDA), which has been one of the most popular probabilistic topic modeling methods in the past decade. However, batch EM has high time and space complexities to learn big LDA models from big data streams. In this paper, we present a fast online EM (FOEM) algorithm that infers the topic distribution from the previously unseen documents incrementally with constant memory requirements. Within the stochastic approximation framework, we show that FOEM can converge to the local stationary point of the LDA's likelihood function. By dynamic scheduling for the fast speed and parameter streaming for the low memory usage, FOEM is more efficient for some lifelong topic modeling tasks than the state-of-the-art online LDA algorithms to handle both big data and big models (aka, big topic modeling) on just a PC.

Features for stochastic approximation based foreground detection

Foreground detection algorithms have sometimes relied on rather ad hoc procedures, even when probabilistic mixture models are defined. Moreover, the fact that the input features have different variances and that they are not independent from each other is often neglected, which hampers performance. Here we aim to obtain a background model which is not tied to any particular choice of features, and that accounts for the variability and the dependences among features. It is based on the stochastic approximation framework. A possible set of features is presented, and their suitability for this problem is assessed. Finally, the proposed procedure is compared with several state-of-the-art alternatives, with satisfactory results.

Simulated Stochastic Approximation Annealing for Global Optimization With a Square-Root Cooling Schedule

Simulated annealing has been widely used in the solution of optimization problems. As known by many researchers, the global optima cannot be guaranteed to be located by simulated annealing unless a logarithmic cooling schedule is used. However, the logarithmic cooling schedule is so slow that no one can afford to use this much CPU time. This article proposes a new stochastic optimization algorithm, the so-called simulated stochastic approximation annealing algorithm, which is a combination of simulated annealing and the stochastic approximation Monte Carlo algorithm. Under the framework of stochastic approximation, it is shown that the new algorithm can work with a cooling schedule in which the temperature can decrease much faster than in the logarithmic cooling schedule, for example, a square-root cooling schedule, while guaranteeing the global optima to be reached when the temperature tends to zero. The new algorithm has been tested on a few benchmark optimization problems, including feed-forward neural network training and protein-folding. The numerical results indicate that the new algorithm can significantly outperform simulated annealing and other competitors. Supplementary materials for this article are available online.

Fast Distributed Strategic Learning for Global Optima in Queueing Access Games

In this paper we examine combined fully distributed payoff and strategy learning (CODIPAS) in a queue-aware access game over a graph. The classical strategic learning analysis relies on vanishing or small learning rate and uses stochastic approximation tool to derive steady states and invariant sets of the underlying learning process. Here, the stochastic approximation framework does not apply due to non-vanishing learning rate. We propose a direct proof of convergence of the process. Interestingly, the convergence time to one of the global optima is almost surely finite and we explicitly characterize the convergence time. We show that pursuit-based CODIPAS learning is much faster than the classical learning algorithms in games. We extend the methodology to coalitional learning and proves a very fast formation of coalitions for queue-aware access games where the action space is dynamically changing depending on the location of the user over a graph.

Designing a Robust Activity Recognition Framework for Health and Exergaming Using Wearable Sensors

Detecting human activity independent of intensity is essential in many applications, primarily in calculating metabolic equivalent rates and extracting human context awareness. Many classifiers that train on an activity at a subset of intensity levels fail to recognize the same activity at other intensity levels. This demonstrates weakness in the underlying classification method. Training a classifier for an activity at every intensity level is also not practical. In this paper, we tackle a novel intensity-independent activity recognition problem where the class labels exhibit large variability, the data are of high dimensionality, and clustering algorithms are necessary. We propose a new robust stochastic approximation framework for enhanced classification of such data. Experiments are reported using two clustering techniques, K-Means and Gaussian Mixture Models. The stochastic approximation algorithm consistently outperforms other well-known classification schemes which validate the use of our proposed clustered data representation. We verify the motivation of our framework in two applications that benefit from intensity-independent activity recognition. The first application shows how our framework can be used to enhance energy expenditure calculations. The second application is a novel exergaming environment aimed at using games to reward physical activity performed throughout the day, to encourage a healthy lifestyle.

A Resampling-Based Stochastic Approximation Method for Analysis of Large Geostatistical Data

The Gaussian geostatistical model has been widely used in modeling of spatial data. However, it is challenging to computationally implement this method because it requires the inversion of a large covariance matrix, particularly when there is a large number of observations. This article proposes a resampling-based stochastic approximation method to address this challenge. At each iteration of the proposed method, a small subsample is drawn from the full dataset, and then the current estimate of the parameters is updated accordingly under the framework of stochastic approximation. Since the proposed method makes use of only a small proportion of the data at each iteration, it avoids inverting large covariance matrices and thus is scalable to large datasets. The proposed method also leads to a general parameter estimation approach, maximum mean log-likelihood estimation, which includes the popular maximum (log)-likelihood estimation (MLE) approach as a special case and is expected to play an important role in analyzing large datasets. Under mild conditions, it is shown that the estimator resulting from the proposed method converges in probability to a set of parameter values of equivalent Gaussian probability measures, and that the estimator is asymptotically normally distributed. To the best of the authors’ knowledge, the present study is the first one on asymptotic normality under infill asymptotics for general covariance functions. The proposed method is illustrated with large datasets, both simulated and real. Supplementary materials for this article are available online.

A Stochastic Approximation Framework for a Class of Randomized Optimization Algorithms

We study a class of random sampling-based algorithms for solving general non-differentiable optimization problems. These are iterative approaches that are based on sampling from and updating an underlying distribution function over the set of feasible solutions. In particular, we propose a novel and systematic framework to investigate the convergence and asymptotic convergence rates of these algorithms by exploiting their connections to the well-known stochastic approximation (SA) method. Such an SA framework unifies our understanding of these randomized algorithms and provides new insight into their design and implementation issues. Our preliminary numerical experiments indicate that new implementations of these algorithms based on the proposed framework may lead to improved performance over existing procedures.

On stochastic gradient and subgradient methods with adaptive steplength sequences

Traditionally, stochastic approximation (SA) schemes have been popular choices for solving stochastic optimization problems. However, the performance of standard SA implementations can vary significantly based on the choice of the steplength sequence, and in general, little guidance is provided about good choices. Motivated by this gap, we present two adaptive steplength schemes for strongly convex differentiable stochastic optimization problems, equipped with convergence theory, that aim to overcome some of the reliance on user-specific parameters. The first scheme, referred to as a recursive steplength stochastic approximation (RSA) scheme, optimizes the error bounds to derive a rule that expresses the steplength at a given iteration as a simple function of the steplength at the previous iteration and certain problem parameters. The second scheme, termed as a cascading steplength stochastic approximation (CSA) scheme, maintains the steplength sequence as a piecewise-constant decreasing function with the reduction in the steplength occurring when a suitable error threshold is met. Then, we allow for nondifferentiable objectives but with bounded subgradients over a certain domain. In such a regime, we propose a local smoothing technique, based on random local perturbations of the objective function, that leads to a differentiable approximation of the function. Assuming a uniform distribution on the local randomness, we establish a Lipschitzian property for the gradient of the approximation and prove that the obtained Lipschitz bound grows at a modest rate with problem size. This facilitates the development of an adaptive steplength stochastic approximation framework, which now requires sampling in the product space of the original measure and the artificially introduced distribution.

Dynamic topology learning with the probabilistic self-organizing graph

Self-organizing neural networks are usually focused on prototype learning, while the topology is held fixed during the learning process. Here a method to adapt the topology of the network so that it reflects the internal structure of the input distribution is proposed. This leads to a self-organizing graph, where each unit is a mixture component of a mixture of Gaussians (MoG). The corresponding update equations are derived from the stochastic approximation framework. This approach combines the advantages of probabilistic mixtures with those of self-organization. Experimental results are presented to show the self-organization ability of our proposal and its performance when used with multivariate datasets in classification and image segmentation tasks.