Value Iteration Research Articles

A Parallel Monte Carlo Algorithm for the Life Cycle Asset Allocation Problem

Life cycle asset allocation is a crucial aspect of financial planning, especially for pension funds. Traditional methods often face challenges in computational efficiency and applicability to different market conditions. This study aimed to innovatively transplant an algorithm from reinforcement learning that enhances the efficiency and accuracy of life cycle asset allocation. We synergized tabular methods with Monte Carlo simulations to solve the pension problem. This algorithm was designed to correspond states in reinforcement learning to key variables in the pension model: wealth, labor income, consumption level, and proportion of risky assets. Additionally, we used cleaned and modeled survey data from Chinese consumers to validate the model’s optimal decision-making in the Chinese market. Furthermore, we optimized the algorithm using parallel computing to significantly reduce computation time. The proposed algorithm demonstrated superior efficiency compared to the traditional value iteration method. Serial execution of our algorithm took 29.88 min, while parallel execution reduced this to 1.42 min, compared to the 41.15 min required by the value iteration method. These innovations suggest significant potential for improving pension fund management strategies, particularly in the context of the Chinese market.

Is Pessimism Provably Efficient for Offline Reinforcement Learning?

We study offline reinforcement learning (RL), which aims to learn an optimal policy based on a data set collected a priori. Because of the lack of further interactions with the environment, offline RL suffers from the insufficient coverage of the data set, which eludes most existing theoretical analysis. In this paper, we propose a pessimistic variant of the value iteration algorithm (PEVI), which incorporates an uncertainty quantifier as the penalty function. Such a penalty function simply flips the sign of the bonus function for promoting exploration in online RL, which makes it easily implementable and compatible with general function approximators. Without assuming the sufficient coverage of the data set (e.g., finite concentratability coefficients or uniformly lower-bounded densities of visitation measures), we establish a data-dependent upper bound on the suboptimality of PEVI for general Markov decision processes (MDPs). When specialized to linear MDPs, it matches the information-theoretic lower bound up to multiplicative factors of the dimension and horizon. In other words, pessimism is not only provably efficient but also, minimax optimal. In particular, given the data set, the learned policy serves as the “best effort” among all policies as no other policies can do better. Our theoretical analysis identifies the critical role of pessimism in eliminating a notion of spurious correlation, which arises from the “irrelevant” trajectories that are less covered by the data set and not informative for the optimal policy. Funding: Z. Yang acknowledges the Simons Institute (Theory of Reinforcement Learning). Z. Wang acknowledges the National Science Foundation [Awards 2048075, 2008827, 2015568, and 1934931], the Simons Institute (Theory of Reinforcement Learning), Amazon, J. P. Morgan, and Two bSigma.

Multi-agent reinforcement learning with synchronized and decomposed reward automaton synthesized from reactive temporal logic

Multi-agent systems (MAS) consist of multiple autonomous agents interacting to achieve collective objectives. Multi-agent reinforcement learning (MARL) enhances these systems by enabling agents to learn optimal behaviors through interaction, thus improving their coordination in dynamic environments. However, MARL faces significant challenges in adapting to complex dependencies on past states and actions, which are not adequately represented by the current state alone in reactive systems. This paper addresses these challenges by considering MAS operating under task specifications formulated as Generalized Reactivity of rank 1 (GR(1)). These synthesized strategies are used as a priori knowledge to guide the learning. To tackle the difficulties of handling non-Markovian tasks in reactive systems, we propose a novel synchronized decentralized training paradigm that guides agents to learn within the MARL framework using a reward structure constructed from decomposed synthesized strategies of GR(1). We initially formalize the synthesis of GR(1) strategies as a reachability problem of winning states of the system. Subsequently, we develop a decomposition mechanism that constructs individual reward structures for decentralized MARL, incorporating potential values calculated through value iteration. Theoretical proofs are provided to verify that the safety and liveness properties are preserved. We evaluate our approach against other state-of-the-art methods under various GR(1) specifications and scenario maps, demonstrating superior learning efficacy and optimal rewards per episode. Additionally, we show that the decentralized training paradigm outperforms the centralized training paradigm. The value iteration strategy used to calculate potential values for the reward structure is compared against two other strategies, showcasing its advantages.

Comparative Analysis of Value Iteration and Policy Iteration in Robotic Decision-Making

This study focuses on Markov Decision Processes (MDPs) as a framework for decision-making in robotics. MDPs are crucial for enabling autonomous systems to operate in dynamic environments. MDPs enable the development of optimal strategies that balance immediate and future rewards, making them essential for intelligent robotic behavior. The purpose of this study is to evaluate the performance differences of value iteration and strategy iteration algorithms in different robot tasks. The study highlights the strengths and weaknesses of each algorithm. This comparative analysis uniquely addresses a gap in existing research by evaluating both Value Iteration and Policy Iteration side by side, offering critical insights into their respective performances across diverse robotic tasks. Results indicate that Policy Iteration converges faster and adapts better in complex environments, making it ideal for real-time applications, while Value Iteration is more efficient in smaller state spaces.These findings provide critical insights for selecting algorithms tailored to specific robotic applications, highlighting their role in advancing intelligent robotic systems.

Dynamic Operational Planning in Warfare: A Stochastic Game Approach to Military Campaigns

ABSTRACTWe study a two‐player discounted zero‐sum stochastic game model for dynamic operational planning in military campaigns. At each stage, the players manage multiple commanders who order military actions on objectives that have an open line of control. When a battle over the control of an objective occurs, its stochastic outcome depends on the actions and the enabling support provided by the control of other objectives. Each player aims to maximize the cumulative number of objectives they control, weighted by their criticality. To solve this large‐scale stochastic game, we derive properties of its Markov perfect equilibria by leveraging the logistics and military operational command and control structure. We show the consequential isotonicity of the optimal value function with respect to the partially ordered state space, which in turn leads to a significant reduction of the state and action spaces. We also accelerate Shapley's value iteration algorithm by eliminating dominated actions and investigating pure equilibria of the matrix game solved at each iteration. We demonstrate the computational value of our equilibrium results on a case study that reflects representative operational‐level military campaigns with geopolitical implications. Our analysis reveals a complex interplay between the game's parameters and dynamics in equilibrium, resulting in new military insights for campaign analysts.

Learning‐based tracking control of AUV: Mixed policy improvement and game‐based disturbance rejection

AbstractA mixed adaptive dynamic programming (ADP) scheme based on zero‐sum game theory is developed to address optimal control problems of autonomous underwater vehicle (AUV) systems subject to disturbances and safe constraints. By combining prior dynamic knowledge and actual sampled data, the proposed approach effectively mitigates the defect caused by the inaccurate dynamic model and significantly improves the training speed of the ADP algorithm. Initially, the dataset is enriched with sufficient reference data collected based on a nominal model without considering modelling bias. Also, the control object interacts with the real environment and continuously gathers adequate sampled data in the dataset. To comprehensively leverage the advantages of model‐based and model‐free methods during training, an adaptive tuning factor is introduced based on the dataset that possesses model‐referenced information and conforms to the distribution of the real‐world environment, which balances the influence of model‐based control law and data‐driven policy gradient on the direction of policy improvement. As a result, the proposed approach accelerates the learning speed compared to data‐driven methods, concurrently also enhancing the tracking performance in comparison to model‐based control methods. Moreover, the optimal control problem under disturbances is formulated as a zero‐sum game, and the actor‐critic‐disturbance framework is introduced to approximate the optimal control input, cost function, and disturbance policy, respectively. Furthermore, the convergence property of the proposed algorithm based on the value iteration method is analysed. Finally, an example of AUV path following based on the improved line‐of‐sight guidance is presented to demonstrate the effectiveness of the proposed method.

Nonzero-sum game-based decentralized approximate optimal control of modular robot manipulators with coordinate operation tasks using value iteration

Nonzero-sum game-based decentralized approximate optimal control of modular robot manipulators with coordinate operation tasks using value iteration

A holistic matrix norm-based alternative solution method for Markov reward games

In this study, we focus on examining single-agent stochastic games, especially Markov reward games represented in the form of a decision tree. We propose an alternative solution method based on the matrix norms for these games. In contrast to the existing methods such as value iteration, policy iteration, and dynamic programming, which are state-and-action-based approaches, the proposed matrix norm-based method considers the relevant stages and their actions as a whole and solves it holistically for each stage without computing the effects of each action on each state's reward individually. The new method involves a distinct transformation of the decision tree into a payoff matrix for each stage and the utilization of the matrix norm of the obtained payoff matrix. Additionally, the concept of the moving matrix is integrated into the proposed method to incorporate the impacts of all actions on the stage simultaneously, rendering the method holistic. Moreover, we present an explanatory algorithm for the implementation of the method and also provide a comprehensive solution diagram explaining the method figuratively. As a result, we offer a new and alternative perspective for solving the games with the help of the proposed method due to the simplicity of utilization of the matrix norms in addition to the existing methods. For clarification of the matrix norm-based method, we demonstrate the figurative application of the method on a benchmark Markov reward game with 2-stages and 2-actions and a comprehensive implementation of the method on a game consisting of 3-stages and 3-actions.

Multiplayer hierarchical decision‐making for discrete‐time nonlinear networks of service via value iteration adaptive dynamic programming

AbstractIn this article, the multiplayer hierarchical decision‐making problem for discrete‐time nonlinear networks of service is studied from the perspective of Nash–Stackelberg–Nash games. A novel two‐level value iteration adaptive dynamic programming algorithm is developed to solve the coupled nonlinear Hamilton–Jacobi–Bellman equations associated with the game problem, which neither requires the system drift dynamics to be known as a priori nor requests the initial strategies to be admissible. Moreover, both the value function sequences and the strategy sequences generated by the algorithm converge to their theoretical optimality. An implementation framework for the algorithm is constructed by leveraging the regularized least‐squares method, and a convergence criterion that guarantees the admissibility of the approximated strategies is also proposed. The effectiveness of the proposed algorithm and implementation framework are finally demonstrated through two numerical simulation examples.

Iterative algorithms for solving one-sided partially observable stochastic shortest path games

Real-world scenarios often involve dynamic interactions among competing agents, where decisions are made considering actions taken by others. These situations can be modeled as partially observable stochastic games (POSGs), with zero-sum variants capturing strictly competitive interactions (e.g., security scenarios). While such models address a broad range of problems, they commonly focus on infinite-horizon scenarios with discounted-sum objectives. Using the discounted-sum objective, however, can lead to suboptimal solutions in cases where the length of the interaction does not directly affect the gained rewards of the players.We thus focus on games with undiscounted objective and an indefinite horizon where every realization of the game is guaranteed to terminate after some unspecified number of turns. To manage the computational complexity of solving POSGs in general, we restrict to games with one-sided partial observability where only one player has imperfect information while their opponent is provided with full information about the current situation. We introduce two novel algorithms based on the heuristic search value iteration (HSVI) algorithm that iteratively solve sequences of easier-to-solve approximations of the game using fundamentally different approaches for constructing the sequences: (1) in GoalHorizon, the game approximations are based on a limited number of turns in which players can change their actions, (2) in GoalDiscount, the game approximations are constructed using an increasing discount factor. We provide theoretical qualitative guarantees for algorithms, and we also experimentally demonstrate that these algorithms are able to find near-optimal solutions on pursuit-evasion games and a game modeling privilege escalation problem from computer security.

Dynamic soft-kill weapon-target assignment in naval environments

One of the most significant threats faced by ships is anti-ship missiles. Nowadays, these missiles, equipped with diverse guidance systems, can locate their trajectory and attack the ship. Consequently, ships need to utilize their weapons to attempt to neutralize these threats. This article aims to develop dynamic assignment algorithms to assign a ship’s defensive soft-kill weapons to a set of incoming missiles, to minimize the average damage inflicted on the ship. To this end, initially, a binary linear programming model is developed to solve the static weapon-target assignment problem. Subsequently, a simulation–optimization algorithm and a reinforcement learning-based approach, grounded in the value iteration algorithm, are developed to solve the dynamic weapon-target assignment problem. To compare and evaluate the performance of the developed solution methods, we employ a set of randomly generated test instances. Computational results indicate that the reinforcement learning approach, due to its inherent foresight, outperforms the simulation–optimization approach in reducing the inflicted damages. However, in terms of CPU run time, the simulation–optimization approach is more efficient.

System transformation and model-free value iteration algorithms for continuous-time linear quadratic stochastic optimal control problems

In this paper, we investigate a continuous-time linear quadratic stochastic optimal control (LQSOC) problem in an infinite horizon, where diffusion and drift terms of the corresponding stochastic system depend on both state and control variables. In light of the stochastic control theory, this LQSOC problem is reduced to solving a generalised algebraic Riccati equation (GARE). With the help of an existing model-based value iteration (VI) algorithm, we propose two data-driven VI algorithms to solve the GARE. The first one relies on transforming the stochastic system into a deterministic control system first and then solving the LQSOC problem by the data of the deterministic system. Consequently, this algorithm does not need the information of two system coefficients and has a lower algorithm complexity. The second algorithm directly uses the data generated by the stochastic system, and thus it circumvents the requirement of all system coefficients. We also provide convergence proofs of these two data-driven algorithms and validate these algorithms through two simulation examples.

Convergence and stability analysis of value iteration Q-learning under non-discounted cost for discrete-time optimal control

This paper presents a theoretical analysis of the value iteration Q-learning with non-discounted costs. The analysis focuses on two main aspects: the convergence of the iterative Q-function and the stability of the system under the final iterative control policy. Unlike previous theoretical results on Q-learning, our analysis takes into account the effect of approximation errors, leading to a more comprehensive investigation. We first discuss the effect of approximation errors on the iterative Q-function update. Then, considering the presence of approximation errors in each iteration, we analyze the convergence of the iterative Q-function. Furthermore, we establish a sufficient condition, also accounting for the approximation errors, to ensure the stability of the system under the final iterative control policy. Finally, two simulation cases are conducted to validate the presented convergence and stability results.

Complex behavior from intrinsic motivation to occupy future action-state path space

Most theories of behavior posit that agents tend to maximize some form of reward or utility. However, animals very often move with curiosity and seem to be motivated in a reward-free manner. Here we abandon the idea of reward maximization and propose that the goal of behavior is maximizing occupancy of future paths of actions and states. According to this maximum occupancy principle, rewards are the means to occupy path space, not the goal per se; goal-directedness simply emerges as rational ways of searching for resources so that movement, understood amply, never ends. We find that action-state path entropy is the only measure consistent with additivity and other intuitive properties of expected future action-state path occupancy. We provide analytical expressions that relate the optimal policy and state-value function and prove convergence of our value iteration algorithm. Using discrete and continuous state tasks, including a high-dimensional controller, we show that complex behaviors such as “dancing”, hide-and-seek, and a basic form of altruistic behavior naturally result from the intrinsic motivation to occupy path space. All in all, we present a theory of behavior that generates both variability and goal-directedness in the absence of reward maximization.

Joint optimization of preventive maintenance and triggering mechanism for k-out-of-n: F systems with protective devices based on periodic inspection

Systems with protective devices (PDs) have attracted increasing research interests due to their extensive engineering applications. In practice, the PDs are triggered to function under certain conditions. However, existing studies focus on the reliability modelling and analysis for such systems, and the joint optimization of maintenance and PD's triggering mechanism has not been investigated. Consequently, this paper explores the joint optimization of maintenance and triggering mechanism for k-out-of-n: F systems with PDs. A new triggering mechanism of the PD is proposed based on the number of failed or unhealthy components. The working PD can reduce the impacts of shocks on components. Based on the detected system status through periodic inspections, corresponding actions including corrective maintenance for components or the PD, preventive maintenance for components or shutting down the PD can be decided to improve the system reliability. The operation process of the established system is modelled by Markov decision process, and optimal actions under diverse system states are obtained via value iteration algorithm. The optimal inspection interval and triggering mechanism are derived by solving the optimization model aiming to minimize the expected total cost. Finally, the applicability of the proposed model is validated by abundant numerical examples.

Projection methods for finding the greatest element of the intersection of max-closed convex sets

We focus on the problem of finding the greatest element of the intersection of max-closed convex sets. For this purpose, we analyze the famous method of cyclic projections and show that, if this method is suitably initialized and applied to max-closed convex sets, it converges to the greatest element of their intersection. Moreover, we propose another projection method, called the decreasing projection, which turns out both theoretically and practically preferable to Euclidean projections in this particular case. Next, we argue that several known algorithms, such as Bellman-Ford and Floyd-Warshall algorithms for shortest paths or Gauss-Seidel variant of value iteration in Markov decision processes, can be interpreted as special cases of iteratively applying decreasing projections onto certain max-closed convex sets. Finally, we link decreasing projections (and thus also the aforementioned algorithms) to bounds consistency in constraint programming.

Anterior cingulate cortex provides the neural substrates for feedback-driven iteration of decision and value representation

Adjusting decision-making under uncertain and dynamic situations is the hallmark of intelligence. It requires a system capable of converting feedback information to renew the internal value. The anterior cingulate cortex (ACC) involves in error and reward events that prompt switching or maintenance of current decision strategies. However, it is unclear whether and how the changes of stimulus-action mapping during behavioral adaptation are encoded, nor how such computation drives decision adaptation. Here, we tracked ACC activity in male mice performing go/no-go auditory discrimination tasks with manipulated stimulus-reward contingencies. Individual ACC neurons integrate the outcome information to the value representation in the next-run trials. Dynamic recruitment of them determines the learning rate of error-guided value iteration and decision adaptation, forming a non-linear feedback-driven updating system to secure the appropriate decision switch. Optogenetically suppressing ACC significantly slowed down feedback-driven decision switching without interfering with the execution of the established strategy.

Approximate dynamic programming for liner shipping network design

AbstractThe global containerised trade heavily relies on liner shipping services, facilitating the worldwide movement of large cargo volumes along fixed routes and schedules. The profitability of shipping companies hinges on how efficiently they design their shipping network; a complex optimization problem known as the liner shipping network design problem (LSNDP). In recent years, approximate dynamic programming (ADP), also known as reinforcement learning, has emerged as a promising approach for large-scale optimisation. This paper introduces a novel Markov decision process for the LSNDP and investigates the potential of ADP. We show that ADP methods based on value iteration produce optimal solutions to small instances, but their scalability is hindered by high memory demands. An ADP method based on a deep neural network requires less memory and successfully obtains feasible solutions. The quality of solutions, however, declines for larger instances, possibly due to the discrete nature of high-dimensional state and action spaces.

On the Convex Formulations of Robust Markov Decision Processes

Robust Markov decision processes (MDPs) are used for applications of dynamic optimization in uncertain environments and have been studied extensively. Many of the main properties and algorithms of MDPs, such as value iteration and policy iteration, extend directly to RMDPs. Surprisingly, there is no known analog of the MDP convex optimization formulation for solving RMDPs. This work describes the first convex optimization formulation of RMDPs under the classical sa-rectangularity and s-rectangularity assumptions. By using entropic regularization and exponential change of variables, we derive a convex formulation with a number of variables and constraints polynomial in the number of states and actions, but with large coefficients in the constraints. We further simplify the formulation for RMDPs with polyhedral, ellipsoidal, or entropy-based uncertainty sets, showing that, in these cases, RMDPs can be reformulated as conic programs based on exponential cones, quadratic cones, and nonnegative orthants. Our work opens a new research direction for RMDPs and can serve as a first step toward obtaining a tractable convex formulation of RMDPs. Funding: The work in the paper was supported, in part, by NSF [Grants 2144601 and 1815275]; and Agence Nationale de la Recherche [Grant 11-LABX-0047].

A Hybrid Handover Scheme for Vehicular VLC/RF Communication Networks.

Visible light communication (VLC) is a promising complementary technology to its radio frequency (RF) counterpart to satisfy the high quality-of-service (QoS) requirements of intelligent vehicular communications by reusing LED street lights. In this paper, a hybrid handover scheme for vehicular VLC/RF communication networks is proposed to balance QoS and handover costs by considering the vertical handover and horizontal handover together judging from the mobile state of the vehicle. A Markov decision process (MDP) is formulated to describe this hybrid handover problem, with a cost function balancing the handover consumption, delay, and reliability. A value iteration algorithm was applied to solve the optimal handover policy. The simulation results demonstrated the performance of the proposed hybrid handover scheme in comparison to other benchmark schemes.