Partially Observable Markov Decision Process Research Articles

Reinforcement Learning for Jointly Optimal Coding and Control Policies for a Markovian System Controlled over a Communication Channel

This paper develops approximation and optimality results for the optimal control of a networked system. In this system, a Markovian process is managed over a finite-rate, noiseless communication channel. Solving this problem involves determining joint optimal coding and control policies that minimize cost over time. While theoretical results on optimal coding and control structures exist, practical implementation has remained largely infeasible for non-linear systems due to the computational complexities and uncountable state spaces. This research introduces an approach that combines structural results with reinforcement learning (RL) techniques. The method uses regularity properties of the system to approximate the uncountable state space with a countable one, which allows reinforcement learning algorithms to achieve near-optimal solutions. Specifically, we establish that finite model approximations (where infinite state spaces are quantized to finite ones) and sliding finite window approximations (where a finite memory "window" of past control actions is maintained at each time step) can be employed to develop near-optimality. These approximations allow the system to be reformulated as a Markov Decision Problem (MDP) with a finite state space, making RL algorithms implementable. The convergence of the reinforcement learning algorithm to a near-optimal policy (under both the previous approximations) is supported by theoretical analysis and performance simulations. We ensure that the solutions obtained are not only computationally feasible but also nearly optimal with respect to the original problem. The applications of this work extend to many networked control systems, especially those with zero-delay coding and partially observable Markov decision processes (POMDPs). By integrating structural results with learning algorithms, this paper provides a practical framework for implementing optimal control in finite-rate environments.

No compromise in solution quality: Speeding up belief-dependent continuous partially observable Markov decision processes via adaptive multilevel simplification

Continuous Partially Observable Markov Decision Processes (POMDPs) with general belief-dependent rewards are notoriously difficult to solve online. In this paper, we present a complete provable theory of adaptive multilevel simplification for the setting of a given externally constructed belief tree and Monte Carlo Tree Search (MCTS) that constructs the belief tree on the fly using an exploration technique. Our theory allows to accelerate POMDP planning with belief-dependent rewards without any sacrifice in the quality of the obtained solution. We rigorously prove each theoretical claim in the proposed unified theory. Using the general theoretical results, we present three algorithms to accelerate continuous POMDP online planning with belief-dependent rewards. Our two algorithms, SITH-BSP and LAZY-SITH-BSP, can be utilized on top of any method that constructs a belief tree externally. The third algorithm, SITH-PFT, is an anytime MCTS method that permits to plug-in any exploration technique. All our methods are guaranteed to return exactly the same optimal action as their unsimplified equivalents. We replace the costly computation of information-theoretic rewards with novel adaptive upper and lower bounds which we derive in this paper, and are of independent interest. We show that they are easy to calculate and can be tightened by the demand of our algorithms. Our approach is general; namely, any bounds that monotonically converge to the reward can be utilized to achieve a significant speedup without any loss in performance. Our theory and algorithms support the challenging setting of continuous states, actions, and observations. The beliefs can be parametric or general and represented by weighted particles. We demonstrate in simulation a significant speedup in planning compared to baseline approaches with guaranteed identical performance.

Multi-objective joint optimization of task offloading based on MADRL in internet of things assisted by satellite networks

The Internet of Things (IoT) integrates a large number of heterogeneous terminals and systems, possessing ubiquitous sensing and computing capabilities. Satellite networks are a crucial supplement to terrestrial networks, particularly in remote areas where network infrastructures are sparingly distributed or unavailable. Combining edge computing with satellite networks provides on-orbit computing capabilities for IoT applications, reducing service delay and enhancing service quality. Due to the resource constraints of satellites, achieving collaborative services through task offloading among multiple satellites becomes essential. Both the privacy leakage risk arising from frequent data interactions and the load imbalance resulting from offloading preferences cannot be overlooked. The key challenge of task offloading is to safeguard the privacy of offloaded data and ensure the system’s load balance while minimizing the delay and energy consumption. In this paper, the task offloading problem is formulated as a Partially Observable Markov Decision Process (POMDP), and a task offloading algorithm based on multi-objective joint optimization using multi-agent deep reinforcement learning in a distributed architecture is proposed. The simulation results validate the efficacy of our model and algorithm, demonstrating that our proposed algorithm achieves better performance in minimizing comprehensive offloading costs.

Battery health-considered energy management strategy for a dual-motor two-speed battery electric vehicle based on a hybrid soft actor-critic algorithm with memory function

Energy management strategies (EMSs) ease mileage anxiety and improve the health performance of battery electric vehicles (BEVs). This study proposes a soft actor-critic (SAC)-based EMS for dual-motor two-speed BEV to minimize electrical energy consumption and battery capacity losses. In addition, two optimization tricks are employed to optimize the original SAC. The linear mapping trick is integrated into the actor-network of SAC to enable agents to search for optimal EMS in a discrete (driving mode)-continuous (torque distribution) hybrid action space. The SAC-based EMS is modeled as a partially observable Markov decision process (POMDP) to obtain more information about the real state of the environment. Based on this, another optimization trick, the long short-term memory (LSTM) network, is integrated into the actor and critic of SAC to make full use of both historical and current environmental information. Simulation results show that the proposed method learns the optimal EMS, its converged episodes are shortened to the 97th, the health performance of the battery is the closest to the dynamic programming (DP)-based EMS, and maintains battery operating at a range of 30–40°C. In addition, the proposed EMS has the best adaptability in test cycles, reaching 99.42% and 99.29% of DP-based EMS, respectively.

Active Inference and Reinforcement Learning: A Unified Inference on Continuous State and Action Spaces under Partial Observability.

Reinforcement learning (RL) has garnered significant attention for developing decision-making agents that aim to maximize rewards, specified by an external supervisor, within fully observable environments. However, many real-world problems involve partial or noisy observations, where agents cannot access complete and accurate information about the environment. These problems are commonly formulated as partially observable Markov decision processes (POMDPs). Previous studies have tackled RL in POMDPs by either incorporating the memory of past actions and observations or by inferring the true state of the environment from observed data. Nevertheless, aggregating observations and actions over time becomes impractical in problems with large decision-making time horizons and high-dimensional spaces. Furthermore, inference-based RL approaches often require many environmental samples to perform well, as they focus solely on reward maximization and neglect uncertainty in the inferred state. Active inference (AIF) is a framework naturally formulated in POMDPs and directs agents to select actions by minimizing a function called expected free energy (EFE). This supplies reward-maximizing (or exploitative) behavior, as in RL, with information-seeking (or exploratory) behavior. Despite this exploratory behavior of AIF, its use is limited to problems with small time horizons and discrete spaces due to the computational challenges associated with EFE. In this article, we propose a unified principle that establishes a theoretical connection between AIF and RL, enabling seamless integration of these two approaches and overcoming their limitations in continuous space POMDP settings. We substantiate our findings with rigorous theoretical analysis, providing novel perspectives for using AIF in designing and implementing artificial agents. Experimental results demonstrate the superior learning capabilities of our method compared to other alternative RL approaches in solving partially observable tasks with continuous spaces. Notably, our approach harnesses information-seeking exploration, enabling it to effectively solve reward-free problems and rendering explicit task reward design by an external supervisor optional.

Digital twins-boosted intelligent maintenance of ageing bridge hangers exposed to coupled corrosion–fatigue deterioration

The corrosion–fatigue coupled deterioration of ageing steel bridge hangers presents significant structural challenges, demanding rigorous condition assessments and timely maintenance interventions. This paper introduces a framework of digital twins-boosted intelligent maintenance (DTIM) tailored for ageing hangers, integrating the prediction model, monitoring data and inspection results. The DTIM features a suite of algorithms adaptation and innovations, including dynamic Partially Observable Markov Decision Processes (POMDP), Asynchronous Advantage Actor Critic (A3C), and Bayesian Dynamic Linear Models (BDLM). The DTIM emphasises regular early-life repairs, strategic inspections, and timely replacements towards life-end, tailored to the condition of specific bridge hangers in the case study presented. By coordinating actions across hangers, the DTIM enables opportunistic maintenance to further optimise resource allocation. The output highlights digital twins in exploring the add-on value of monitoring and inspection for the proactive and sustainable maintenance of ageing infrastructure, promising enhanced structural integrity and serviceability in a cost-effective and well-informed manner.

A Workflow for Building Computationally Rational Models of Human Behavior

Computational rationality explains human behavior as arising due to the maximization of expected utility under the constraints imposed by the environment and limited cognitive resources. This simple assumption, when instantiated via partially observable Markov decision processes (POMDPs), gives rise to a powerful approach for modeling human adaptive behavior, within which a variety of internal models of cognition can be embedded. In particular, such an instantiation enables the use of methods from reinforcement learning (RL) to approximate the optimal policy solution to the sequential decision-making problems posed to the cognitive system in any given setting; this stands in contrast to requiring ad hoc hand-crafted rules for capturing adaptive behavior in more traditional cognitive architectures. However, despite their successes and promise for modeling human adaptive behavior across everyday tasks, computationally rational models that use RL are not easy to build. Being a hybrid of theoretical cognitive models and machine learning (ML) necessitates that model building take into account appropriate practices from both cognitive science and ML. The design of psychological assumptions and machine learning decisions concerning reward specification, policy optimization, parameter inference, and model selection are all tangled processes rife with pitfalls that can hinder the development of valid and effective models. Drawing from a decade of work on this approach, a workflow is outlined for tackling this challenge and is accompanied by a detailed discussion of the pros and cons at key decision points.

Transfer entropy on collective motion with undeclared loose leader–follower (LLF) structure

During an emergency or evacuation, individuals focus on the movement of their neighbors and follow those in sight, even without social affiliation, forming the undeclared Loose Leader-Follower (LLF) group, rather than the usual assumption of following certain rules in most of evacuation models. Inspired by the real-life experience and experimental data, the study develops a novel stochastic process-based evacuation model, integrated the LLF structure in collective motion to clarify the mechanism underlying individual interaction and refine the walking process according to attainable information. Transfer entropy (TE), a model-free measure of the direction of information flow, is used to detect the role of the individuals in the LLF structure from the time-series reconstructed motion information (e.g., the walking coordinate series). To reduce the error of the role detection, according to the 3σ principle for TE values, the detection threshold is decided based on the surrogate walking data. For simulation of their motion decision in the LLF structure, the TE is integrated into the walking behavior mode (TE-POMDP) using the Partially Observable Markov Decision Process (POMDP), that produces their motion behavior under the typical attained information condition. The results show that the TE-POMDP can provide more realistic trajectories. Additionally, we find that a ‘Delta-like’ formation of leading group distribution was presented and remained stable in the procession after a short period of confusion in early stage.

Rapid data collection and processing in dense urban edge computing networks with drone assistance

In the edge computing network systems for the Internet of Things (IoT), there is growing attention to utilizing drones for collecting data and maintaining the freshness of data processing. This study focuses on analyzing the problems related to trajectory planning and task scheduling in a single drone-assisted edge computing network within a dense, three-dimensional urban environment. We first design an edge computing network architecture and establish an air-to-ground channel model between the drone and ground mobile devices to address the blockage caused by buildings in urban environments. Subsequently, to provide effective edge computing services, we structure the problem as a Partially Observable Markov Decision Process (POMDP) and introduce an optimization framework based on reinforcement learning. This improves data timeliness and reduces energy consumption.

When to monitor or control: Informed invasive species management using a partially observable Markov decision process (POMDP) framework

Abstract Resource allocation for invasive species management requires information about the size of the invasive population, which may be expensive and time‐consuming to obtain. The trade‐off between investment in monitoring and control efforts is a challenging decision problem, and existing mathematical tools are often difficult to interpret, and/or limited to a specific case study. We propose a partially observable Markov decision process (POMDP) framework to help decision‐makers understand effective monitoring and control policymaking. POMDPs can deal with uncertainty in both the model and state of the system but are more challenging to solve due to the continuous and high‐dimensional state space. Rather than limiting the possible states of the system, as do most previously proposed methods, we work through the development of a density projection approach, which reduces the dimensionality of the space of beliefs by restricting them to a parametrised family of probability distributions. This serves to align the mathematical representation of the problem with the real‐world quantities relevant to human decision‐making. The result of our model is a sequence of actions, which minimises the expected cost incurred in managing the invasive species, where the recommendation depends on an estimate of the species' abundance, and the uncertainty in this estimate. We demonstrate the effectiveness of our proposed framework with a case study on tropical fire ant (Solenopsis geminata) control and two generic case studies of varying complexity. Furthermore, we investigate sensitivity of the results to the choices of control cost and efficacy, and monitoring cost and error. The framework proposed by this paper makes the powerful machinery of POMDPs available to environmental managers. It computes the optimal course of action to manage a growing population of an invasive species, incorporating a varying time horizon and multiple control interventions. We sidestep the computational difficulties of general POMDPs to provide a clear, visual overview of decision‐making recommendations, and how these decisions change in new situations. Initial results and scenario‐based analysis show promising results, and the framework could be extended to the related field of disease management.

Research on mobile robot path planning in complex environment based on DRQN algorithm

A deep reinforcement Q learning algorithm (DRQN) based on radial neural network is proposed to achieve path planning and obstacle avoidance for mobile robots in complex ground environments with different types of obstacles, including static and dynamic obstacles. Firstly, the path planning problem is represented as a partially-observed Markov decision process. Steering angle, running characteristics, and other elements are introduced into the state-action decision space and the greedy factor is dynamically adjusted using a simulated annealing algorithm, which improves the mobile robot’s environment exploration and action selection accuracy. Secondly, the Q-learning algorithm is improved by replacing the Q-table structure with an RBF neural network to enhance the approximation ability of the algorithm’s function values, and the parameters of the implicit layer and the weights between the implicit and the output layer are trained using the dynamic clustering and least-mean methods respectively, which improves the convergence speed and enhances the ability of mobile robots to handle large-scale computation. Lastly, the double reward mechanism is set up to prevent the mobile robot from blind searching in unknown environments, which enhances the learning ability and improves path planning safety and flexibility at the same time. Different types of scenarios are set up for simulation experiments, and the results verified the superiority of the DQRN algorithm. Taking the 30 * 30 complex scene as an example, using the DQRN algorithm for path planning reduces the values of distance, turning angle, and planning time by 27.04%, 7.76%, and 28.05%, respectively, compared to the average values of Q-learning, optimized Q-learning, deep Q-learning, and DDPG algorithms, which can effectively improve the path planning efficiency for mobile robots in complex environments.

Competitive pricing for ride-sourcing platforms with MARL

With the advancement of autonomous driving, the competition between shared autonomous vehicles (SAVs) and the existing human-driven vehicles (HVs) ride-sourcing platforms has emerged as a prominent research area within the fields of transportation and management. Solving the pricing problem, considering the competition between HV and SAV platforms, is critical for the platform’s operation. The competitive pricing problem is hard to solve since it is a partially observable Markov decision process (POMDP), that is, only the strategies of other platforms (such as price and wage) are observable, while their states are unobservable. It is also a multivariable sequential problem pricing since spatial-temporal pricing varies with region and time. To tackle this complex problem, this paper implements the multi-agent reinforcement learning (MARL) approach of multi-agent deep deterministic policy gradient (MADDPG) to an mesoscopic simulation system. The simulation replicates the intricate dynamics within a competitive environment where SAV and HV platforms coexist. MADDPG effectively resolves the competitive pricing issue using data derived from simulation. The findings indicate that the SAV platform secures greater profits with a smaller fleet, as it eliminates the need to compensate drivers. Meanwhile, for the same reason, even with fewer vehicles, the SAV platform can set lower prices to attract more passengers and thereby generate higher profits. We further evaluate the efficacy of dynamic pricing versus spatial-temporal pricing and discover that spatial-temporal pricing yields greater profits.

Examining chronic kidney disease screening frequency among diabetics: a POMDP approach.

Forty percent of diabetics will develop chronic kidney disease (CKD) in their lifetimes. However, as many as 50% of these CKD cases may go undiagnosed. We developed screening recommendations stratified by age and previous test history for individuals with diagnosed diabetes and unknown proteinuria status by race and gender groups. To do this, we used a Partially Observed Markov Decision Process (POMDP) to identify whether a patient should be screened at every three-month interval from ages 30-85. Model inputs were drawn from nationally-representative datasets, the medical literature, and a microsimulation that integrates this information into group-specific disease progression rates. We implement the POMDP solution policy in the microsimulation to understand how this policy may impact health outcomes and generate an easily-implementable, non-belief-based approximate policy for easier clinical interpretability. We found that the status quo policy, which is to screen annually for all ages and races, is suboptimal for maximizing expected discounted future net monetary benefits (NMB). The POMDP policy suggests more frequent screening after age 40 in all race and gender groups, with screenings 2-4 times a year for ages 61-70. Black individuals are recommended for screening more frequently than their White counterparts. This policy would increase NMB from the status quo policy between $1,000to $8,000 per diabetic patient at a willingness-to-pay of $150,000 per quality-adjusted life year (QALY).

Trajectory planning for airborne radar in extended target tracking based on deep reinforcement learning

In this article, an effective algorithm is proposed for trajectory optimization in airborne radar for extended target tracking (ETT). The key idea of the proposed airborne radar trajectory planning (ARTP) scheme is to model this problem as a partially observable Markov decision process (POMDP) and plan the waypoints of airborne radar using deep reinforcement learning (DRL) in order to improve the target's kinematic state and shape tracking accuracy, subject to the airborne radar maneuverability limitation. Specifically, at each tracking interval, the predicted Gaussian Wasserstein Distance (GWD) is analytically calculated and subsequently adopted as a reward function to evaluate the performance of extended target tracking. Then the DRL agent autonomously learns trajectory planning strategies to maximize ETT performance. Simulation results demonstrate that the proposed ARTP algorithm can deliver superior performance in terms of maximizing the overall ETT performance compared with other benchmarks.

Strong Simple Policies for POMDPs

The synthesis problem for partially observable Markov decision processes (POMDPs) is to compute a policy that provably adheres to one or more specifications. Yet, the general problem is undecidable, and policies require full (and thus potentially unbounded) traces of execution history. To provide good approximations of such policies, POMDP agents often employ randomization over action choices. We consider the problem of computing simpler policies for POMDPs, and provide several approaches to still ensure their expressiveness. Key aspects are (1) the combination of an arbitrary number of specifications the policies need to adhere to, (2) a restricted form of randomization, and (3) a light-weight preprocessing of the POMDP model to encode memory. We provide a novel encoding as a mixed-integer linear program as baseline to solve the underlying problems. Our experiments demonstrate that the policies we obtain are more robust, smaller, and easier to implement for an engineer than those obtained from state-of-the-art POMDP solvers.

POMDP-based probabilistic decision making for path planning in wheeled mobile robot

Path Planning in a collaborative mobile robot system has been a research topic for many years. Uncertainty in robot states, actions, and environmental conditions makes finding the optimum path for navigation highly challenging for the robot. To achieve robust behavior for mobile robots in the presence of static and dynamic obstacles, it is pertinent that the robot employs a path-finding mechanism that is based on the probabilistic perception of the uncertainty in various parameters governing its movement. Partially Observable Markov Decision Process (POMDP) is being used by many researchers as a proven methodology for handling uncertainty. The POMDP framework requires manually setting up the state transition matrix, the observation matrix, and the reward values. This paper describes an approach for creating the POMDP model and demonstrates its working by simulating it on two mobile robots destined on a collision course. Selective test cases are run on the two robots with three categories – MDP (POMDP with belief state spread of 1), POMDP with distribution spread of belief state over ten observations, and distribution spread across two observations. Uncertainty in the sensor data is simulated with varying levels of up to 10 %. The results are compared and analyzed. It is demonstrated that when the observation probability spread is increased from 2 to 10, collision reduces from 34 % to 22 %, indicating that the system's robustness increases by 12 % with only a marginal increase of 3.4 % in the computational complexity.

Optimal Joint Defense and Monitoring for Networks Security under Uncertainty: A POMDP-Based Approach

The increasing interconnectivity in our infrastructure poses a significant security challenge, with external threats having the potential to penetrate and propagate throughout the network. Bayesian attack graphs have proven to be effective in capturing the propagation of attacks in complex interconnected networks. However, most existing security approaches fail to systematically account for the limitation of resources and uncertainty arising from the complexity of attacks and possible undetected compromises. To address these challenges, this paper proposes a partially observable Markov decision process (POMDP) model for network security under uncertainty. The POMDP model accounts for uncertainty in monitoring and defense processes, as well as the probabilistic attack propagation. This paper develops two security policies based on the optimal stationary defense policy for the underlying POMDP state process (i.e., a network with known compromises): the estimation-based policy that performs the defense actions corresponding to the optimal minimum mean square error state estimation and the distribution-based policy that utilizes the posterior distribution of network compromises to make defense decisions. Optimal monitoring policies are designed to specifically support each of the defense policies, allowing dynamic allocation of monitoring resources to capture network vulnerabilities/compromises. The performance of the proposed policies is examined in terms of robustness, accuracy, and uncertainty using various numerical experiments.

Decision Making for Self-Adaptation Based on Partially Observable Satisfaction of Non-Functional Requirements

Approaches that support the decision-making of self-adaptive and autonomous systems (SAS) often consider an idealized situation where (i) the system’s state is treated as fully observable by the monitoring infrastructure, and (ii) adaptation actions are assumed to have known, deterministic effects over the system. However, in practice, the system’s state may not be fully observable, and the adaptation actions may produce unexpected effects due to uncertain factors. This article presents a novel probabilistic approach to quantify the uncertainty associated with the effects of adaptation actions on the state of a SAS. Supported by Bayesian inference and POMDPs (Partially-Observable Markov Decision Processes), these effects are translated into the satisfaction levels of the non-functional requirements (NFRs) to, therefore, drive the decision-making. The approach has been applied to two substantial case studies from the networking and Internet of Things (IoT) domains, using two different POMDP solvers. The results show that the approach delivers statistically significant improvements in supporting decision-making for SAS.

L2SR: learning to sample and reconstruct for accelerated MRI via reinforcement learning

Magnetic resonance imaging (MRI) is a widely used medical imaging technique, but its long acquisition time can be a limiting factor in clinical settings. To address this issue, researchers have been exploring ways to reduce the acquisition time while maintaining the reconstruction quality. Previous works have focused on finding either sparse samplers with a fixed reconstructor or finding reconstructors with a fixed sampler. However, these approaches do not fully utilize the potential of joint learning of samplers and reconstructors. In this paper, we propose an alternating training framework for jointly learning a good pair of samplers and reconstructors via deep reinforcement learning. In particular, we consider the process of MRI sampling as a sampling trajectory controlled by a sampler, and introduce a novel sparse-reward partially observed Markov decision process (POMDP) to formulate the MRI sampling trajectory. Compared to the dense-reward POMDP used in existing works, the proposed sparse-reward POMDP is more computationally efficient and has a provable advantage. Moreover, the proposed framework, called learning to sample and reconstruct (L2SR), overcomes the training mismatch problem that arises in previous methods that use dense-reward POMDP. By alternately updating samplers and reconstructors, L2SR learns a pair of samplers and reconstructors that achieve state-of-the-art reconstruction performances on the fastMRI dataset. Codes are available at https://github.com/yangpuPKU/L2SR-Learning-to-Sample-and-Reconstruct.

Downlink Scheduler for Delay Guaranteed Services Using Deep Reinforcement Learning

In this paper, we propose a novel scheduling scheme to guarantee per-packet delay in single-hop wireless networks for delay-critical applications. We consider several classes of packets with different delay requirements, where high-class packets yield high utility after successful transmission. Considering the correla-tionship of delays among competing packets, we apply a delay-laxity concept and introduce a new output gain function for scheduling decisions. Particularly, the selection of a packet takes into account not only its output gain but also the delay-laxity of other packets. In this context, we formulate a multi-objective optimization problem aiming to minimize the average queue length while maximizing the average output gain under the constraint of guaranteeing per-packet delay. However, due to the uncertainty in the environment (e.g., time-varying channel conditions and random packet arrivals), it is difficult and often impractical to solve this problem using traditional optimization techniques. We develop a deep reinforcement learning (DRL)-based framework to solve it. Specifically, we decompose the original optimization problem into a set of scalar optimization subproblems and model each of them as a partially observable Markov Decision Process (POMDP). We then resort to a Double Deep Q Network (DDQN)-based algorithm to learn an optimal scheduling policy for each subproblem, which can overcome the large-scale state space and reduce Q-value overestimation. Simulation results show that our proposed DDQN-based algorithm outperforms the conventional Q-learning algorithm in terms of reward and learning speed. In addition, our proposed scheduling scheme can achieve significant reductions in average delay and delay outage drop rate compared to other benchmark schemes.