Partially Observable Markov Decision Process Research Articles

When New Experience Leads to New Knowledge: A Computational Framework for Formalizing Epistemically Transformative Experiences.

The discovery of a new kind of experience can teach an agent what that kind of experience is like. Such a discovery can be epistemically transformative, teaching an agent something they could not have learned without having that kind of experience. However, learning something new does not always require new experience. In some cases, an agent can merely expand their existing knowledge using, e.g., inference or imagination that draws on prior knowledge. We present a computational framework, grounded in the language of partially observable Markov Decision Processes (POMDPs), to formalize this distinction. We propose that epistemically transformative experiences leave a measurable "signature" distinguishing them from experiences that are not epistemically transformative. For epistemically transformative experiences, learning in a new environment may be comparable to "learning from scratch" (since prior knowledge has become obsolete). In contrast, for experiences that are not transformative, learning in a new environment can be facilitated by prior knowledge of that same kind (since new knowledge can be built upon the old). We demonstrate this in a synthetic experiment inspired by Edwin Abbott's Flatland, where an agent learns to navigate a 2D world and is subsequently transferred either to a 3D world (epistemically transformative change) or to an expanded 2D world (epistemically non-transformative change). Beyond the contribution to understanding epistemic change, our work shows how tools in computational cognitive science can formalize and evaluate philosophical intuitions in new ways.

A partitioning Monte Carlo approach for consensus tasks in crowdsourcing

The planner’s role in crowdsourcing involves determining the time to stop collecting information (i.e., timed decision, TD). Previous studies have modeled the uncertain crowdsourcing environment as Partially Observable Markov Decision Processes (POMDPs) and utilized the value of information (VOI) to balance the utilities and costs of information collection. However, there is still room for optimization in solving the TD problem within single-agent POMDPs. In this paper, we propose a partitioning Monte Carlo approach for consensus tasks based on the option-candidate (OC) model partitioning. We simplify the state representation for the OC model and introduce a multi-agent POMDP representation. We establish a correspondence between the single-agent and multi-agent models using two consistency theorems. We propose three progressively improved partitioning Monte Carlo (PMC) algorithms to solve the TD problem within the multi-agent POMDP. We conducted experiments in synthetic domains and a citizen science project, demonstrating that the proposed algorithms exhibit continuous improvements in runtime advantages and consistently outperform the SOTA single-agent Monte Carlo sampling-based algorithm while providing nearly consistent output results.

Multi-Agent DRL for Air-to-Ground Communication Planning in UAV-Enabled IoT Networks.

In this paper, we present a novel method to enhance the sum-rate effectiveness in full-duplex unmanned aerial vehicle (UAV)-assisted communication networks. Existing approaches often couple uplink and downlink associations, resulting in suboptimal performance, particularly in dynamic environments where user demands and network conditions are unpredictable. To overcome these limitations, we propose a decoupling of uplink and downlink associations for ground-based users (GBUs), significantly improving network efficiency. We formulate a comprehensive optimization problem that integrates UAV trajectory design and user association, aiming to maximize the overall sum-rate efficiency of the network. Due to the problem's non-convexity, we reformulate it as a Partially Observable Markov Decision Process (POMDP), enabling UAVs to make real-time decisions based on local observations without requiring complete global information. Our framework employs multi-agent deep reinforcement learning (MADRL), specifically the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which balances centralized training with distributed execution. This allows UAVs to efficiently learn optimal user associations and trajectory controls while dynamically adapting to local conditions. The proposed solution is particularly suited for critical applications such as disaster response and search and rescue missions, highlighting the practical significance of utilizing UAVs for rapid network deployment in emergencies. By addressing the limitations of existing centralized and distributed solutions, our hybrid model combines the benefits of centralized training with the adaptability of distributed inference, ensuring optimal UAV operations in real-time scenarios.

Adaptive Compensation for Robotic Joint Failures Using Partially Observable Reinforcement Learning

Robotic manipulators are widely used in various industries for complex and repetitive tasks. However, they remain vulnerable to unexpected hardware failures. In this study, we address the challenge of enabling a robotic manipulator to complete tasks despite joint malfunctions. Specifically, we develop a reinforcement learning (RL) framework to adaptively compensate for a nonfunctional joint during task execution. Our experimental platform is the Franka robot with seven degrees of freedom (DOFs). We formulate the problem as a partially observable Markov decision process (POMDP), where the robot is trained under various joint failure conditions and tested in both seen and unseen scenarios. We consider scenarios where a joint is permanently broken and where it functions intermittently. Additionally, we demonstrate the effectiveness of our approach by comparing it with traditional inverse kinematics-based control methods. The results show that the RL algorithm enables the robot to successfully complete tasks even with joint failures, achieving a high success rate with an average rate of 93.6%. This showcases its robustness and adaptability. Our findings highlight the potential of RL to enhance the resilience and reliability of robotic systems, making them better suited for unpredictable environments.

Online Learning Strategy Induction through Partially Observable Markov Decision Process-Based Cognitive Experience Model

Inducing learning strategies is a crucial component of intelligent tutoring systems. Previous research has predominantly focused on the induction of offline learning strategies. Although the existing offline learning strategy induction methods can also be used for real-time updates of learning strategies, their update efficiency is not high, making it difficult to capture the characteristics exhibited by learners during the learning process in a timely manner. With the superior performance of the Partially Observable Markov Decision Process (POMDP), this paper proposes a POMDP-based cognitive experience model, which can be quickly updated during interactions and enables the real-time induction of learning strategies by weighting the learning experiences of different learners. Experimental results demonstrate that the learning strategies induced by PCEM are more personalized and exhibit superior performance.

Multi-agent reinforcement learning for task offloading with hybrid decision space in multi-access edge computing

Multi-access Edge Computing (MEC) has become a significant technology for supporting the computation-intensive and time-sensitive applications on the Internet of Things (IoT) devices. However, it is challenging to jointly optimize task offloading and resource allocation in the dynamic wireless environment with constrained edge resource. In this paper, we investigate a multi-user and multi-MEC servers system with varying task request and stochastic channel condition. Our purpose is to minimize the total energy consumption and time delay by optimizing the offloading decision, offloading ratio and computing resource allocation simultaneously. As the users are geographically distributed within an area, we formulate the problem of task offloading and resource allocation in MEC system as a partially observable Markov decision process (POMDP) and propose a novel multi-agent deep reinforcement learning (MADRL) -based algorithm to solve it. In particular, two aspects have been modified for performance enhancement: (1) To make fine-grained control, we design a novel neural network structure to effectively handle the hybrid action space arisen by the heterogeneous variables. (2) An adaptive reward mechanism is proposed to reasonably evaluate the infeasible actions and to mitigate the instability caused by manual configuration. Simulation results show the proposed method can achieve 7.12%−20.97% performance enhancements compared with the existing approaches.

Active Inference and Reinforcement Learning: A Unified Inference on Continuous State and Action Spaces Under Partial Observability.

Reinforcement learning (RL) has garnered significant attention for developing decision-making agents that aim to maximize rewards, specified by an external supervisor, within fully observable environments. However, many real-world problems involve partial or noisy observations, where agents cannot access complete and accurate information about the environment. These problems are commonly formulated as partially observable Markov decision processes (POMDPs). Previous studies have tackled RL in POMDPs by either incorporating the memory of past actions and observations or by inferring the true state of the environment from observed data. Nevertheless, aggregating observations and actions over time becomes impractical in problems with large decision-making time horizons and high-dimensional spaces. Furthermore, inference-based RL approaches often require many environmental samples to perform well, as they focus solely on reward maximization and neglect uncertainty in the inferred state. Active inference (AIF) is a framework naturally formulated in POMDPs and directs agents to select actions by minimizing a function called expected free energy (EFE). This supplies reward-maximizing (or exploitative) behavior, as in RL, with information-seeking (or exploratory) behavior. Despite this exploratory behavior of AIF, its use is limited to problems with small time horizons and discrete spaces due to the computational challenges associated with EFE. In this article, we propose a unified principle that establishes a theoretical connection between AIF and RL, enabling seamless integration of these two approaches and overcoming their limitations in continuous space POMDP settings. We substantiate our findings with rigorous theoretical analysis, providing novel perspectives for using AIF in designing and implementing artificial agents. Experimental results demonstrate the superior learning capabilities of our method compared to other alternative RL approaches in solving partially observable tasks with continuous spaces. Notably, our approach harnesses information-seeking exploration, enabling it to effectively solve reward-free problems and rendering explicit task reward design by an external supervisor optional.

Multi-objective joint optimization of task offloading based on MADRL in internet of things assisted by satellite networks

The Internet of Things (IoT) integrates a large number of heterogeneous terminals and systems, possessing ubiquitous sensing and computing capabilities. Satellite networks are the crucial supplement to terrestrial networks, particularly in remote areas where network infrastructures are sparingly distributed or unavailable. Combining edge computing with satellite networks provides on-orbit computing capabilities for IoT applications, reducing service delay and enhancing service quality. Due to the resource constraints of satellites, achieving collaborative services through task offloading among multiple satellites becomes essential. Both the privacy leakage risk arising from frequent data interactions and the load imbalance resulting from offloading preferences cannot be overlooked. The key challenge of task offloading is to safeguard the privacy of offloaded data and ensure the system’s load balance while minimizing the delay and energy consumption. In this paper, the task offloading problem is formulated as a Partially Observable Markov Decision Process (POMDP), and a task offloading algorithm based on multi-objective joint optimization using multi-agent deep reinforcement learning in a distributed architecture is proposed. The simulation results validate the efficacy of our model and algorithm, demonstrating that our proposed algorithm achieves better performance in minimizing comprehensive offloading costs.

Reinforcement Learning for Jointly Optimal Coding and Control Policies for a Markovian System Controlled over a Communication Channel

This paper develops approximation and optimality results for the optimal control of a networked system. In this system, a Markovian process is managed over a finite-rate, noiseless communication channel. Solving this problem involves determining joint optimal coding and control policies that minimize cost over time. While theoretical results on optimal coding and control structures exist, practical implementation has remained largely infeasible for non-linear systems due to the computational complexities and uncountable state spaces. This research introduces an approach that combines structural results with reinforcement learning (RL) techniques. The method uses regularity properties of the system to approximate the uncountable state space with a countable one, which allows reinforcement learning algorithms to achieve near-optimal solutions. Specifically, we establish that finite model approximations (where infinite state spaces are quantized to finite ones) and sliding finite window approximations (where a finite memory "window" of past control actions is maintained at each time step) can be employed to develop near-optimality. These approximations allow the system to be reformulated as a Markov Decision Problem (MDP) with a finite state space, making RL algorithms implementable. The convergence of the reinforcement learning algorithm to a near-optimal policy (under both the previous approximations) is supported by theoretical analysis and performance simulations. We ensure that the solutions obtained are not only computationally feasible but also nearly optimal with respect to the original problem. The applications of this work extend to many networked control systems, especially those with zero-delay coding and partially observable Markov decision processes (POMDPs). By integrating structural results with learning algorithms, this paper provides a practical framework for implementing optimal control in finite-rate environments.

No compromise in solution quality: Speeding up belief-dependent continuous partially observable Markov decision processes via adaptive multilevel simplification

Continuous Partially Observable Markov Decision Processes (POMDPs) with general belief-dependent rewards are notoriously difficult to solve online. In this paper, we present a complete provable theory of adaptive multilevel simplification for the setting of a given externally constructed belief tree and Monte Carlo Tree Search (MCTS) that constructs the belief tree on the fly using an exploration technique. Our theory allows to accelerate POMDP planning with belief-dependent rewards without any sacrifice in the quality of the obtained solution. We rigorously prove each theoretical claim in the proposed unified theory. Using the general theoretical results, we present three algorithms to accelerate continuous POMDP online planning with belief-dependent rewards. Our two algorithms, SITH-BSP and LAZY-SITH-BSP, can be utilized on top of any method that constructs a belief tree externally. The third algorithm, SITH-PFT, is an anytime MCTS method that permits to plug-in any exploration technique. All our methods are guaranteed to return exactly the same optimal action as their unsimplified equivalents. We replace the costly computation of information-theoretic rewards with novel adaptive upper and lower bounds which we derive in this paper, and are of independent interest. We show that they are easy to calculate and can be tightened by the demand of our algorithms. Our approach is general; namely, any bounds that monotonically converge to the reward can be utilized to achieve a significant speedup without any loss in performance. Our theory and algorithms support the challenging setting of continuous states, actions, and observations. The beliefs can be parametric or general and represented by weighted particles. We demonstrate in simulation a significant speedup in planning compared to baseline approaches with guaranteed identical performance.

Battery health-considered energy management strategy for a dual-motor two-speed battery electric vehicle based on a hybrid soft actor-critic algorithm with memory function

Energy management strategies (EMSs) ease mileage anxiety and improve the health performance of battery electric vehicles (BEVs). This study proposes a soft actor-critic (SAC)-based EMS for dual-motor two-speed BEV to minimize electrical energy consumption and battery capacity losses. In addition, two optimization tricks are employed to optimize the original SAC. The linear mapping trick is integrated into the actor-network of SAC to enable agents to search for optimal EMS in a discrete (driving mode)-continuous (torque distribution) hybrid action space. The SAC-based EMS is modeled as a partially observable Markov decision process (POMDP) to obtain more information about the real state of the environment. Based on this, another optimization trick, the long short-term memory (LSTM) network, is integrated into the actor and critic of SAC to make full use of both historical and current environmental information. Simulation results show that the proposed method learns the optimal EMS, its converged episodes are shortened to the 97th, the health performance of the battery is the closest to the dynamic programming (DP)-based EMS, and maintains battery operating at a range of 30–40°C. In addition, the proposed EMS has the best adaptability in test cycles, reaching 99.42% and 99.29% of DP-based EMS, respectively.

Digital twins-boosted intelligent maintenance of ageing bridge hangers exposed to coupled corrosion–fatigue deterioration

The corrosion–fatigue coupled deterioration of ageing steel bridge hangers presents significant structural challenges, demanding rigorous condition assessments and timely maintenance interventions. This paper introduces a framework of digital twins-boosted intelligent maintenance (DTIM) tailored for ageing hangers, integrating the prediction model, monitoring data and inspection results. The DTIM features a suite of algorithms adaptation and innovations, including dynamic Partially Observable Markov Decision Processes (POMDP), Asynchronous Advantage Actor Critic (A3C), and Bayesian Dynamic Linear Models (BDLM). The DTIM emphasises regular early-life repairs, strategic inspections, and timely replacements towards life-end, tailored to the condition of specific bridge hangers in the case study presented. By coordinating actions across hangers, the DTIM enables opportunistic maintenance to further optimise resource allocation. The output highlights digital twins in exploring the add-on value of monitoring and inspection for the proactive and sustainable maintenance of ageing infrastructure, promising enhanced structural integrity and serviceability in a cost-effective and well-informed manner.

A Workflow for Building Computationally Rational Models of Human Behavior

Computational rationality explains human behavior as arising due to the maximization of expected utility under the constraints imposed by the environment and limited cognitive resources. This simple assumption, when instantiated via partially observable Markov decision processes (POMDPs), gives rise to a powerful approach for modeling human adaptive behavior, within which a variety of internal models of cognition can be embedded. In particular, such an instantiation enables the use of methods from reinforcement learning (RL) to approximate the optimal policy solution to the sequential decision-making problems posed to the cognitive system in any given setting; this stands in contrast to requiring ad hoc hand-crafted rules for capturing adaptive behavior in more traditional cognitive architectures. However, despite their successes and promise for modeling human adaptive behavior across everyday tasks, computationally rational models that use RL are not easy to build. Being a hybrid of theoretical cognitive models and machine learning (ML) necessitates that model building take into account appropriate practices from both cognitive science and ML. The design of psychological assumptions and machine learning decisions concerning reward specification, policy optimization, parameter inference, and model selection are all tangled processes rife with pitfalls that can hinder the development of valid and effective models. Drawing from a decade of work on this approach, a workflow is outlined for tackling this challenge and is accompanied by a detailed discussion of the pros and cons at key decision points.

Transfer entropy on collective motion with undeclared loose leader–follower (LLF) structure

During an emergency or evacuation, individuals focus on the movement of their neighbors and follow those in sight, even without social affiliation, forming the undeclared Loose Leader-Follower (LLF) group, rather than the usual assumption of following certain rules in most of evacuation models. Inspired by the real-life experience and experimental data, the study develops a novel stochastic process-based evacuation model, integrated the LLF structure in collective motion to clarify the mechanism underlying individual interaction and refine the walking process according to attainable information. Transfer entropy (TE), a model-free measure of the direction of information flow, is used to detect the role of the individuals in the LLF structure from the time-series reconstructed motion information (e.g., the walking coordinate series). To reduce the error of the role detection, according to the 3σ principle for TE values, the detection threshold is decided based on the surrogate walking data. For simulation of their motion decision in the LLF structure, the TE is integrated into the walking behavior mode (TE-POMDP) using the Partially Observable Markov Decision Process (POMDP), that produces their motion behavior under the typical attained information condition. The results show that the TE-POMDP can provide more realistic trajectories. Additionally, we find that a ‘Delta-like’ formation of leading group distribution was presented and remained stable in the procession after a short period of confusion in early stage.

Rapid data collection and processing in dense urban edge computing networks with drone assistance

In the edge computing network systems for the Internet of Things (IoT), there is growing attention to utilizing drones for collecting data and maintaining the freshness of data processing. This study focuses on analyzing the problems related to trajectory planning and task scheduling in a single drone-assisted edge computing network within a dense, three-dimensional urban environment. We first design an edge computing network architecture and establish an air-to-ground channel model between the drone and ground mobile devices to address the blockage caused by buildings in urban environments. Subsequently, to provide effective edge computing services, we structure the problem as a Partially Observable Markov Decision Process (POMDP) and introduce an optimization framework based on reinforcement learning. This improves data timeliness and reduces energy consumption.

When to monitor or control: Informed invasive species management using a partially observable Markov decision process (POMDP) framework

Abstract Resource allocation for invasive species management requires information about the size of the invasive population, which may be expensive and time‐consuming to obtain. The trade‐off between investment in monitoring and control efforts is a challenging decision problem, and existing mathematical tools are often difficult to interpret, and/or limited to a specific case study. We propose a partially observable Markov decision process (POMDP) framework to help decision‐makers understand effective monitoring and control policymaking. POMDPs can deal with uncertainty in both the model and state of the system but are more challenging to solve due to the continuous and high‐dimensional state space. Rather than limiting the possible states of the system, as do most previously proposed methods, we work through the development of a density projection approach, which reduces the dimensionality of the space of beliefs by restricting them to a parametrised family of probability distributions. This serves to align the mathematical representation of the problem with the real‐world quantities relevant to human decision‐making. The result of our model is a sequence of actions, which minimises the expected cost incurred in managing the invasive species, where the recommendation depends on an estimate of the species' abundance, and the uncertainty in this estimate. We demonstrate the effectiveness of our proposed framework with a case study on tropical fire ant (Solenopsis geminata) control and two generic case studies of varying complexity. Furthermore, we investigate sensitivity of the results to the choices of control cost and efficacy, and monitoring cost and error. The framework proposed by this paper makes the powerful machinery of POMDPs available to environmental managers. It computes the optimal course of action to manage a growing population of an invasive species, incorporating a varying time horizon and multiple control interventions. We sidestep the computational difficulties of general POMDPs to provide a clear, visual overview of decision‐making recommendations, and how these decisions change in new situations. Initial results and scenario‐based analysis show promising results, and the framework could be extended to the related field of disease management.

Research on mobile robot path planning in complex environment based on DRQN algorithm

A deep reinforcement Q learning algorithm (DRQN) based on radial neural network is proposed to achieve path planning and obstacle avoidance for mobile robots in complex ground environments with different types of obstacles, including static and dynamic obstacles. Firstly, the path planning problem is represented as a partially-observed Markov decision process. Steering angle, running characteristics, and other elements are introduced into the state-action decision space and the greedy factor is dynamically adjusted using a simulated annealing algorithm, which improves the mobile robot’s environment exploration and action selection accuracy. Secondly, the Q-learning algorithm is improved by replacing the Q-table structure with an RBF neural network to enhance the approximation ability of the algorithm’s function values, and the parameters of the implicit layer and the weights between the implicit and the output layer are trained using the dynamic clustering and least-mean methods respectively, which improves the convergence speed and enhances the ability of mobile robots to handle large-scale computation. Lastly, the double reward mechanism is set up to prevent the mobile robot from blind searching in unknown environments, which enhances the learning ability and improves path planning safety and flexibility at the same time. Different types of scenarios are set up for simulation experiments, and the results verified the superiority of the DQRN algorithm. Taking the 30 * 30 complex scene as an example, using the DQRN algorithm for path planning reduces the values of distance, turning angle, and planning time by 27.04%, 7.76%, and 28.05%, respectively, compared to the average values of Q-learning, optimized Q-learning, deep Q-learning, and DDPG algorithms, which can effectively improve the path planning efficiency for mobile robots in complex environments.

Competitive pricing for ride-sourcing platforms with MARL

With the advancement of autonomous driving, the competition between shared autonomous vehicles (SAVs) and the existing human-driven vehicles (HVs) ride-sourcing platforms has emerged as a prominent research area within the fields of transportation and management. Solving the pricing problem, considering the competition between HV and SAV platforms, is critical for the platform’s operation. The competitive pricing problem is hard to solve since it is a partially observable Markov decision process (POMDP), that is, only the strategies of other platforms (such as price and wage) are observable, while their states are unobservable. It is also a multivariable sequential problem pricing since spatial-temporal pricing varies with region and time. To tackle this complex problem, this paper implements the multi-agent reinforcement learning (MARL) approach of multi-agent deep deterministic policy gradient (MADDPG) to an mesoscopic simulation system. The simulation replicates the intricate dynamics within a competitive environment where SAV and HV platforms coexist. MADDPG effectively resolves the competitive pricing issue using data derived from simulation. The findings indicate that the SAV platform secures greater profits with a smaller fleet, as it eliminates the need to compensate drivers. Meanwhile, for the same reason, even with fewer vehicles, the SAV platform can set lower prices to attract more passengers and thereby generate higher profits. We further evaluate the efficacy of dynamic pricing versus spatial-temporal pricing and discover that spatial-temporal pricing yields greater profits.

Examining chronic kidney disease screening frequency among diabetics: a POMDP approach.

Forty percent of diabetics will develop chronic kidney disease (CKD) in their lifetimes. However, as many as 50% of these CKD cases may go undiagnosed. We developed screening recommendations stratified by age and previous test history for individuals with diagnosed diabetes and unknown proteinuria status by race and gender groups. To do this, we used a Partially Observed Markov Decision Process (POMDP) to identify whether a patient should be screened at every three-month interval from ages 30-85. Model inputs were drawn from nationally-representative datasets, the medical literature, and a microsimulation that integrates this information into group-specific disease progression rates. We implement the POMDP solution policy in the microsimulation to understand how this policy may impact health outcomes and generate an easily-implementable, non-belief-based approximate policy for easier clinical interpretability. We found that the status quo policy, which is to screen annually for all ages and races, is suboptimal for maximizing expected discounted future net monetary benefits (NMB). The POMDP policy suggests more frequent screening after age 40 in all race and gender groups, with screenings 2-4 times a year for ages 61-70. Black individuals are recommended for screening more frequently than their White counterparts. This policy would increase NMB from the status quo policy between $1,000to $8,000 per diabetic patient at a willingness-to-pay of $150,000 per quality-adjusted life year (QALY).

Trajectory planning for airborne radar in extended target tracking based on deep reinforcement learning

In this article, an effective algorithm is proposed for trajectory optimization in airborne radar for extended target tracking (ETT). The key idea of the proposed airborne radar trajectory planning (ARTP) scheme is to model this problem as a partially observable Markov decision process (POMDP) and plan the waypoints of airborne radar using deep reinforcement learning (DRL) in order to improve the target's kinematic state and shape tracking accuracy, subject to the airborne radar maneuverability limitation. Specifically, at each tracking interval, the predicted Gaussian Wasserstein Distance (GWD) is analytically calculated and subsequently adopted as a reward function to evaluate the performance of extended target tracking. Then the DRL agent autonomously learns trajectory planning strategies to maximize ETT performance. Simulation results demonstrate that the proposed ARTP algorithm can deliver superior performance in terms of maximizing the overall ETT performance compared with other benchmarks.