Stochastic Optimization Problem Research Articles

A theoretical and empirical study of new adaptive algorithms with additional momentum steps and shifted updates for stochastic non-convex optimization

Abstract It is known that adaptive optimization algorithms represent the key pillar behind the rise of the machine learning field. In the optimization literature numerous studies have been devoted to accelerated gradient methods but only recently adaptive iterative techniques were analyzed from a theoretical point of view. In the present paper we introduce new adaptive algorithms endowed with momentum terms for stochastic non-convex optimization problems. Our purpose is to show a deep connection between accelerated methods endowed with different inertial steps and AMSGrad-type momentum methods. Our methodology is based on the framework of stochastic and possibly non-convex objective mappings, along with some assumptions that are often used in the investigation of adaptive algorithms. In addition to discussing the finite-time horizon analysis in relation to a certain final iteration and the almost sure convergence to stationary points, we shall also look at the worst-case iteration complexity. This will be followed by an estimate for the expectation of the squared Euclidean norm of the gradient. Various computational simulations for the training of neural networks are being used to support the theoretical analysis. For future research we emphasize that there are multiple possible extensions to our work, from which we mention the investigation regarding non-smooth objective functions and the theoretical analysis of a more general formulation that encompasses our adaptive optimizers in a stochastic framework.

Neural network solution of stochastic Pontryagin maximum principle for urban road network perimeter control

In contemporary perimeter control strategies, reliance on simplified accumulation-based macroscopic fundamental diagram (MFD) models with minimal scatter in macroscopic variables is common. Yet, the inherent state-dependent stochasticity driving road network evolution challenges the efficacy of such deterministic approaches, potentially undermining desired objectives when supply and demand variabilities are ignored. To fill the gap, our study integrates an analytical state-dependent stochastic dynamics model into the MFD framework, incorporating Brownian noise to capture the stochastic essence of traffic evolution. This advancement enables the modeling of stochastic gridlock processes. We formulate a stochastic optimal control problem for perimeter regulation based on this novel MFD model, targeting both delay minimization and gridlock prevention. Utilizing the stochastic Pontryagin maximum principle (SPMP), we express the optimal control conditions as a coupled forward-backward stochastic differential equation (FBSDE). To solve this FBSDE, we reformulate it as an optimization problem and develop neural network-assisted solution methodologies. Our numerical studies validate the convergence of the proposed solution method and demonstrate significant performance enhancements: not only can gridlocks be effectively mitigated by accounting for the model’s stochastic nature and associated objectives, but also the total cumulative trip completion rate improves by up to 25% when optimizing for both delay reduction and gridlock avoidance. These findings underscore the potential of integrating stochastic dynamics into perimeter control strategies for more resilient and efficient traffic management systems.

Stochastic haul rate planning using two-stage stochastic programming and model predictive control

Abstract Optimising the scheduling of shovels and trucks is a critical task in open-pit mining, significantly impacting production efficiency and operational costs. Traditional models often assume ideal conditions without accounting for disturbances; however, in this paper, we address this gap by incorporating multiple uncertainties. These uncertainties are categorised into contingencies and variations. Contingencies are binary events that may or may not happen, such as shovel breakdowns. Variations are continuous factors such as fluctuations in mineral composition, truck travel time, maximum loading rate of shovels and maximum processing rate of crushers. We develop and test a robust mixed-integer programming (MILP) model for computing the optimal loading rate of shovels in open-pit mines. The problem is formulated as a two-stage stochastic optimisation problem which is then solved and executed in a receding horizon fashion. The proposed methodology is simulated for an open-pit mine in Australia’s Pilbara region and the results confirm the effectiveness of the proposed stochastic optimisation approach in minimising the effect of uncertainty on the operation.

An SDE perspective on stochastic inertial gradient dynamics with time-dependent viscosity and Hessian-driven damping

Our approach is part of the close link between continuous dissipative dynamical systems and optimization algorithms. Motivated by solving stochastic convex optimization problems in Hilbert spaces, we propose a class of stochastic inertial differential equations driven by the gradient of the objective function. This will provide a general mathematical framework for analyzing fast optimization algorithms with stochastic gradient input. Our goal is to develop convergence guarantees for second-order stochastic differential equations in time, incorporating a viscous time-dependent damping and a Hessian-driven damping. To develop this program, we rely on stochastic Lyapunov analysis. Assuming a square-integrability condition on the diffusion term times a function dependent on the viscous damping, and that the Hessian-driven damping is a positive constant, our first main result shows that almost surely, there is convergence of the objective values with a fast convergence rate in expectation. Besides, in the case where the Hessian-driven damping is zero, we get fast convergence of the values in expectation and in almost sure sense, and we also prove almost sure weak convergence of the trajectory. We provide a comprehensive complexity analysis by establishing several new convergence rates in expectation and in almost sure sense for the convex and strongly convex case.

Convergence of policy gradient for stochastic linear quadratic optimal control problems in infinite horizon

Convergence of policy gradient for stochastic linear quadratic optimal control problems in infinite horizon

Decision-focused linear pooling for probabilistic forecast combination

In real-world settings, decision-makers often have access to multiple forecasts for the same unknown quantity. Combining different forecasts has long been known to improve forecast quality, as measured by scoring rules in the case of probabilistic forecasting. However, improved forecast quality does not always translate into better decisions in a downstream problem that utilizes the resultant combined forecast as input. To this end, this work proposes a novel probabilistic forecast combination approach that accounts for the downstream stochastic optimization problem by which the decisions will be made. We propose a linear pool of probabilistic forecasts where the respective weights are learned by minimizing the expected decision cost of the induced combination, which we formulate as a nested optimization problem. Two methods are proposed for its solution: a gradient-based method that utilizes differential optimization layers, and a performance-based weighting method. The proposed decision-focused combination approach is validated in two integral problems associated with renewable energy integration in low-carbon power systems and compared against well-established combination methods. Namely, we examine an electricity market trading problem under stochastic solar production and a grid scheduling problem under stochastic wind production. The results illustrate that the proposed approach leads to lower expected downstream costs, while optimizing for forecast quality when estimating linear pool weights does not always translate into better decisions. Notably, optimizing for a combination of downstream cost and an accuracy-oriented scoring rule consistently leads to better decisions while also improving forecast quality.

Robust Gittins for Stochastic Scheduling

A common theme in stochastic optimization problems is that, theoretically, stochastic algorithms need to "know" relatively rich information about the underlying distributions. This is at odds with most applications, where distributions are rough predictions based on historical data. Thus, commonly, stochastic algorithms are making decisions using imperfect predicted distributions, while trying to optimize over some unknown true distributions. We consider the fundamental problem of scheduling stochastic jobs preemptively on a single machine to minimize expected mean completion time in the setting where the scheduler is only given imperfect predicted job size distributions. If the predicted distributions are perfect, then it is known that this problem can be solved optimally by the Gittins index policy. The goal of our work is to design a scheduling policy that is robust in the sense that it produces nearly optimal schedules even if there are modest discrepancies between the predicted distributions and the underlying real distributions. Our main contributions are: - We show that the standard Gittins index policy is not robust in this sense. If the true distributions are perturbed by even an arbitrarily small amount, then running the Gittins index policy using the perturbed distributions can lead to an unbounded increase in mean completion time. - We explain how to modify the Gittins index policy to make it robust, that is, to produce nearly optimal schedules, where the approximation depends on a new measure of error between the true and predicted distributions that we define. Looking forward, the approach we develop here can be applied more broadly to many other stochastic optimization problems to better understand the impact of mispredictions, and lead to the development of new algorithms that are robust against such mispredictions.

On the Relationship Between Stochastic Turnpike and Dissipativity Notions

In this paper, we introduce and study different dissipativity notions and different turnpike properties for discrete-time stochastic nonlinear optimal control problems. The proposed stochastic dissipativity notions extend the classic notion of Jan C. Willems to L r random variables and to probability measures. Our stochastic turnpike properties range from a formulation for random variables via turnpike phenomena in probability and in probability measures to the turnpike property for the moments. Moreover, we investigate how different metrics (such as Wasserstein or L évy-Prokhorov) can be leveraged in the analysis. Our results are built upon stationarity concepts in distribution and in random variables and on the formulation of the stochastic optimal control problem as a finite-horizon Markov decision process. We investigate how the proposed dissipativity notions connect to the various stochastic turnpike properties and we work out the link between different forms of dissipativity.

An improved pelican optimization algorithm for solving stochastic optimal power flow problem of power systems considering uncertainty of renewable energy resources

An improved pelican optimization algorithm for solving stochastic optimal power flow problem of power systems considering uncertainty of renewable energy resources

Optimal annual solar PV penetration for improved voltage regulation and power loss reduction under uncertainty conditions

Given their technological, economic, and environmental advantages, the widespread adoption of renewable distributed generators (RDGs) in distribution systems (DSs) is becoming more prevalent. However, Solar photovoltaic distributed generators (PV-DGs) face the challenge of intermittent behavior, which results in power output fluctuations and increased grid uncertainty. Therefore, addressing these uncertainties is crucial when determining their optimal allocation. The proposed method considers uncertainties related to both load demand and solar irradiation. The model is formulated as a stochastic mixed-integer nonlinear optimization problem, which is solved using the whale optimization algorithm (WOA). The standard IEEE 33-bus system is used to validate the proposed approach, and demand variations are modeled based on the IEEE reliability test system (IEEE-RTS). The objective is to simultaneously minimize total expected voltage deviation, real power loss, and reactive power loss while increasing solar PV penetration. The technique for order of preference by similarity to the ideal solution (TOPSIS) is applied to select the best solution. Simulated results indicate significant improvements: a 19.39% reduction in voltage deviation, an 18.42% decrease in total real power loss, and an 18.53% reduction in reactive power loss compared to the base case. Additionally, the model accommodates a total of 3.206625 MW of solar PV power in the DS.

Towards distributed two-stage stochastic optimization

Abstract The weighted vertex cover problem revolves around selecting a subset of vertices that covers a target edge set while minimizing the total cost of the selected vertices. We consider a variant of this classic optimization problem where the target edge set is not fully known; rather, it is characterized by a probability distribution. Adhering to the model of two-stage stochastic optimization, the execution is divided into two stages. In the first stage, the decision maker selects a vertex subset based on the probabilistic forecast of the target edge set. In the second stage, the target edge set is revealed, and the decision maker can augment the initial vertex subset with additional vertices to ensure coverage; however, this augmentation is more expensive due to increased vertex costs. This paper initiates the study of the two-stage stochastic vertex cover problem in the realm of distributed graph algorithms, where the decision-making process is distributed among the graph’s vertices. We consider two known stochastic optimization variants: the independent sampling model, where the edges in the target set are drawn independently from some probability distribution; and the finite scenario model, where the probability distribution over the target edge set is provided explicitly. For both variants, we devise efficient distributed algorithms based on a novel adaptation of the distributed primal-dual technique to linear programs resulting from the stochastic optimization problems’ relaxation.

Stochastic Differential Games and Optimization Problems with Controlled Point Process Arrivals

There is a very large literature on applications of stochastic control of jump diffusions and a smaller literature on such games. In many applications it is natural to assume that the arrival intensity is controlled, but except for two long-forgotten papers the literature instead assumes that it is the jump sizes that are controlled. The more natural assumption is typically avoided because a failed Lipschitz condition means that the classical existence and uniqueness proofs cannot be used. We here derive an asymptotic Markov equilibrium of the game with controlled jump intensities and show that it, at least in an example, is very similar to the Markov equilibrium of an analog game with controlled jump sizes. The paper thus makes two contributions: It supplies a way to solve some optimization problems and games with controlled jump intensities and it shows that the commonly used formulation with controlled jump sizes is quite defensible for at least some classes of games.

Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems

Two-level stochastic optimization formulations have become instrumental in a number of machine learning contexts such as continual learning, neural architecture search, adversarial learning, and hyperparameter tuning. Practical stochastic bilevel optimization problems become challenging in optimization or learning scenarios where the number of variables is high or there are constraints. In this paper, we introduce a bilevel stochastic gradient method for bilevel problems with nonlinear and possibly nonconvex lower-level constraints. We also present a comprehensive convergence theory that addresses both the lower-level unconstrained and constrained cases and covers all inexact calculations of the adjoint gradient (also called hypergradient), such as the inexact solution of the lower-level problem, inexact computation of the adjoint formula (due to the inexact solution of the adjoint equation or use of a truncated Neumann series), and noisy estimates of the gradients, Hessians, and Jacobians involved. To promote the use of bilevel optimization in large-scale learning, we have developed new low-rank practical bilevel stochastic gradient methods (BSG-N-FD and BSG-1) that do not require second-order derivatives and, in the lower-level unconstrained case, dismiss any matrix–vector products.

Necessary and Sufficient Conditions of Open-Loop and Closed-Loop Solvability for Delayed Stochastic LQ Optimal Control Problems

Necessary and Sufficient Conditions of Open-Loop and Closed-Loop Solvability for Delayed Stochastic LQ Optimal Control Problems

A Regularized Variance-Reduced Modified Extragradient Method for Stochastic Hierarchical Games

We consider an N-player hierarchical game in which the ith player’s objective comprises of an expectation-valued term, parametrized by rival decisions, and a hierarchical term. Such a framework allows for capturing a broad range of stochastic hierarchical optimization problems, Stackelberg equilibrium problems, and leader-follower games. We develop an iteratively regularized and smoothed variance-reduced modified extragradient framework for iteratively approaching hierarchical equilibria in a stochastic setting. We equip our analysis with rate statements, complexity guarantees, and almost-sure convergence results. We then extend these statements to settings where the lower-level problem is solved inexactly and provide the corresponding rate and complexity statements. Our model framework encompasses many game theoretic equilibrium problems studied in the context of power markets. We present a realistic application to the study of virtual power plants, emphasizing the role of hierarchical decision making and regularization. Preliminary numerics suggest that empirical behavior compares well with theoretical guarantees.

Research on Offloading and Resource Allocation for MEC with Energy Harvesting Based on Deep Reinforcement Learning

Mobile edge computing (MEC) systems empowered by energy harvesting (EH) significantly enhance sustainable computing capabilities for mobile devices (MDs). This paper investigates a multi-user multi-server MEC network, in which energy-constrained users dynamically harvest ambient energy to flexibly allocate resources among local computation, task offloading, or intentional task discarding. We formulate a stochastic optimization problem aiming to minimize the time-averaged weighted sum of execution delay, energy consumption, and task discard penalty. To address the energy causality constraints and temporal coupling effects, we develop a Lyapunov optimization-based drift-plus-penalty framework that decomposes the long-term optimization into sequential per-time-slot subproblems. Furthermore, to overcome the curse of dimensionality in high-dimensional action, we propose hierarchical deep reinforcement learning (DRL) solutions incorporating both Q-learning with experience replay and asynchronous advantage actor–critic (A3C) architectures. Extensive simulations demonstrate that our DRL-driven approach achieves lower costs compared with conventional model predictive control methods, while maintaining robust performance under stochastic energy arrivals and channel variations.

Stochastic Optimal Control for Uncertain Structural Systems Under Random Excitations Based on Bayes Optimal Estimation

Stochastic vibration control of uncertain structures under random loading is an important problem and its minimax optimal control strategy remains to be developed. In this paper, a stochastic optimal control strategy for uncertain structural systems under random excitations is proposed, based on the minimax stochastic dynamical programming principle and the Bayes optimal estimation method with the combination of stochastic dynamics and Bayes inference. The general description of the stochastic optimal control problem is presented including optimal parameter estimation and optimal state control. For the estimation, the posterior probability density conditional on observation states is expressed using the likelihood function conditional on system parameters according to Bayes’ theorem. The likelihood is replaced by the geometrically averaged likelihood, and the posterior is converted into its logarithmic expression to avoid numerical singularity. The expressions of state statistics are derived based on stochastic dynamics. The statistics are further transformed into those conditional on observation states based on optimal state estimation. Then, the obtained posterior will be more reliable and accurate, and the optimal estimation will greatly reduce uncertain parameter domains. For the control, the minimax strategy is designed by minimizing the performance index for the worst-parameter system, which is obtained by maximizing the performance index based on game theory. The dynamical programming equation for the uncertain system is derived according to the minimax stochastic dynamical programming principle. The worst parameters are determined by the maximization of the equation, and the optimal control is determined by the minimization of the resulting equation. The minimax optimal control by combining the Bayes optimal estimation and minimax stochastic dynamical programming will be more effective and robust. Finally, numerical results for a five-story frame structure under random excitations show the control effectiveness of the proposed strategy.

Adam Algorithm with Step Adaptation

Adam (Adaptive Moment Estimation) is a well-known algorithm for the first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. As shown by computational experiments, with an increase in the degree of conditionality of the problem and in the presence of interference, Adam is prone to looping, which is associated with difficulties in step adjusting. In this paper, an algorithm for step adaptation for the Adam method is proposed. The principle of the step adaptation scheme used in the paper is based on reproducing the state in which the descent direction and the new gradient are found during one-dimensional descent. In the case of exact one-dimensional descent, the angle between these directions is right. In case of inexact descent, if the angle between the descent direction and the new gradient is obtuse, then the step is large and should be reduced; if the angle is acute, then the step is small and should be increased. For the experimental analysis of the new algorithm, test functions of a certain degree of conditionality with interference on the gradient and learning problems with mini-batches for calculating the gradient were used. As the computational experiment showed, in stochastic optimization problems, the proposed Adam modification with step adaptation turned out to be significantly more efficient than both the standard Adam algorithm and the other methods with step adaptation that are studied in the work.

AMFL: Resource-Efficient Adaptive Metaverse-Based Federated Learning for the Human-Centric Augmented Reality Applications.

The emergence of 5G technology has enabled the development of Metaverse applications that provide users with immersive experiences through augmented reality (AR) devices, and the integration of federated learning (FL) with the Metaverse AR (MAR) systems can enable many edge intelligence services in 5G. However, the presence of nonindependent and identically distributed (Non-IID) data across all AR users' devices, coupled with limited edge communication resources, makes it challenging to achieve human-centric Metaverse-related applications such as target detection or image classification that combine virtual content with real-world. To address these challenges, we propose a novel adaptive resource-efficient Metaverse-based FL (AMFL) algorithm for AR applications that mitigates the negative effect of Non-IID data and reduces resource costs as well as improves the quality of experience (QoE). We first analyze the impact of wireless communication factors such as CPU frequency, bandwidth, and transmission power on FL training performance by a toy example in the MAR systems. Based on this analysis, furthermore, we establish a Non-IID degree, model accuracy, and resource consumption-related QoE maximization problem under given resource budgets, which is a stochastic optimization problem with strongly coupled variables, including bandwidth, CPU frequency, and transmission power. Guided by the theoretical analysis, to solve this issue, AMFL employs a deep reinforcement learning (DRL)-based method to adaptively allocate resources. Numerical results demonstrate that AMFL can significantly improve the QoE by up to 30.28%, and reduce communication round and energy costs by up to 81.08% and 72.20%, respectively, even under the worst Non-IID case, compared to benchmarks.

Stochastic Optimal Avoidance of Multiple Engagement Zones

This work formulates feedback control strategies for vehicles to reach a goal point amongst a field of dynamic risk regions. Whereas previous work has considered deterministic versions of this problem, this work considers the scenario in the context of uncertainty. This uncertainty could be due to unknown wind or other external disturbances. The risk regions’ dynamics are tightly coupled to the state of the vehicle, and as a result the task of navigating through a field dynamically changes as the vehicle traverses through it. Rather than using a nonlinear program solver, the approach taken here is one of formulating the problem as a stochastic optimal control problem wherein a feedback control law is derived. This feedback control law allows a single or multiple vehicles to reach the desired goal location efficiently and with a minimal risk incursion. When inside a risk region, there is a probability rate of the vehicle being neutralized. The stochastic dynamical system is modeled as a hybrid system, i.e., regime switching diffusion. The hybrid system is comprised of continuous spatial dynamics and discrete operational states corresponding to “normal” and “neutralized.” The optimal control problem is discretized into a field of vehicle states and control. Using Value Iteration, the control signal at each discrete location is optimized, providing a feedback control strategy over the field. Following a discussion of the methods and approaches, this problem is formulated, simulations are performed, and conclusions and remarks are made.¶