MDP Model Research Articles

Research on pure azimuth passive positioning of unmanned aerial vehicle formation in electromagnetic silence environment

In the electromagnetic silence environment, for the azimuth-only passive localization problem in the conical formation, it was concluded that there was a deviation between the ideal standard formation and the actual formation. These disturbance deviations should be eliminated to make the UAV reach the ideal formation state. Therefore, the adjustment model of individual UAV, the directed graph model of formation node and the MDP model of adjustment strategy are constructed. Based on the relevant factor model constructed, the azimuth-only passive localization model in conical formation is established and solved in MATLAB. Finally, experiments and analysis are carried out in the simulation environment, and the effectiveness and feasibility of the proposed algorithm are proved.

Dynamic Scene Path Planning of UAVs Based on Deep Reinforcement Learning

Traditional unmanned aerial vehicle path planning methods focus on addressing planning issues in static scenes, struggle to balance optimality and real-time performance, and are prone to local optima. In this paper, we propose an improved deep reinforcement learning approach for UAV path planning in dynamic scenarios. Firstly, we establish a task scenario including an obstacle assessment model and model the UAV’s path planning problem using the Markov Decision Process. We translate the MDP model into the framework of reinforcement learning and design the state space, action space, and reward function while incorporating heuristic rules into the action exploration policy. Secondly, we utilize the Q function approximation of an enhanced D3QN with a prioritized experience replay mechanism and design the algorithm’s network structure based on the TensorFlow framework. Through extensive training, we obtain reinforcement learning path planning policies for both static and dynamic scenes and innovatively employ a visualized action field to analyze their planning effectiveness. Simulations demonstrate that the proposed algorithm can accomplish UAV dynamic scene path planning tasks and outperforms classical methods such as A*, RRT, and DQN in terms of planning effectiveness.

Research on the effective way of mental health education based on artificial intelligence technology

Abstract In response to the imperfect development of the mental health education enhancement system, this paper uses artificial intelligence technology to explore and innovate effective ways to enhance mental health education. This paper firstly constructs the MDP model, value function, strategy evaluation, and strategy enhancement based on artificial intelligence technology to build a feed-forward neural network model. Secondly, it analyzes three aspects of mental health education: the course offering or not, the frequency of the course offering, and the relevant content of the course. Finally, the MDP model with artificial intelligence technology and feed-forward neural network model was used to verify the enhancement path of mental health education, and four aspects of mental health education were analyzed in terms of the cognitive situation, arrangement of content, course offering, and educational teachers and training. The results showed that only 5.875% of the teachers knew a lot about mental health education in schools, 37.25% said they knew more, and 57% said they did not know much about mental health education, which indicates that most teachers do not pay much attention to mental health education. Thus, it is feasible to explore ways to improve mental health education based on the context of artificial intelligence.

Compensation guarantees in crowdsourced delivery: Impact on platform and driver welfare

Crowdsourced delivery and other sharing economy platforms attract freelance workers by offering them flexibility in scheduling their own work hours. Those platforms, however, have been criticized for the lack of protection they offer workers. Since workers are treated as independent contractors, they do not receive minimum wage and other protection measures under labor law. In this paper, we examine the integration of driver compensation guarantees in a platform’s dynamic matching decisions. We study the problem of designing dynamic matching policies that guarantee a particular level of compensation for active workers over a time period, while maintaining work hour flexibility. We model three types of policies, that are either wage-based or utilization-based. We propose an MDP model to capture the dynamic and stochastic nature of the problem, then design a value function approximation algorithm to efficiently solve the large-scale MDP model. Extensive computational testing is conducted to assess the performance of the proposed solution methodology and the compensation guarantees, using synthetic and real-world datasets. Our findings suggest that the utilization policy results in the highest earning for drivers, though at the expense of longer empty miles from drivers’ origins to the pickup locations of matched orders. On the other hand, the effective wage policy leads to shorter average distance to pickup, but slightly lower earning to drivers. Both policies result in only a slight decrease in platform profit as compared to the base case, and exhibit lower dispersion in the distribution of driver earning while active. In contrast, the nominal wage policy shows a comparable trend to the base-case policy in terms of average driver earnings, suggesting minimal benefits for drivers.

Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects

In safe MDP planning, a cost function based on the current state and action is often used to specify safety aspects. In real world, often the state representation used may lack sufficient fidelity to specify such safety constraints. Operating based on an incomplete model can often produce unintended negative side effects (NSEs). To address these challenges, first, we associate safety signals with state-action trajectories (rather than just immediate state-action). This makes our safety model highly general. We also assume categorical safety labels are given for different trajectories, rather than a numerical cost function, which is harder to specify by the problem designer. We then employ a supervised learning model to learn such non-Markovian safety patterns. Second, we develop a Lagrange multiplier method, which incorporates the safety model and the underlying MDP model in a single computation graph to facilitate agent learning of safe behaviors. Finally, our empirical results on a variety of discrete and continuous domains show that this approach can satisfy complex non-Markovian safety constraints while optimizing agent's total returns, is highly scalable, and is also better than previous best approach for Markovian NSEs.

Imperfect-Information Game AI Agent Based on Reinforcement Learning Using Tree Search and a Deep Neural Network

In the field of computer intelligence, it has always been a challenge to construct an agent model that can be adapted to various complex tasks. In recent years, based on the planning algorithm of Monte Carlo tree search (MCTS), a new idea has been proposed to solve the AI problems of two-player zero-sum games such as chess and Go. However, most of the games in the real environment rely on imperfect information, so it is impossible to directly use the normal tree search planning algorithm to construct a decision-making model. Mahjong, which is a popular multiplayer game with a long history in China, attracts great attention from AI researchers because it contains a large game state space and a lot of hidden information. In this paper, we utilize an agent learning approach that leverages deep learning, reinforcement learning, and dropout learning techniques to implement a Mahjong AI game agent. First, we improve the state transition of the tree search based on the learned MDP model, the player position variable and transition information are introduced into the tree search algorithm to construct a multiplayer search tree. Then, the model training based on a deep reinforcement learning method ensures the stable and sustainable training process of the learned MDP model. Finally, we utilize the strategy data generated by the tree search and use the dropout learning method to train the normal decision-making agent. The experimental results demonstrate the efficiency and stability performance of the agent trained by our proposed method compared with existing agents in terms of test data accuracy, tournament ranking performance, and online match performance. The agent plays against human players and acts like real humans.

Obtaining emergent behaviors for swarm robotics singling with deep reinforcement learning

ABSTRACT Isolating (singling) an individual from a group can be essential for protection, rescue or capture tasks. In this paper a system with multiple shepherds who must coordinate the sheep to achieve a specific singleness is proposed. We present a realistically modeled system that will be finally tested in a real robotic system. We want to encourage the adaptability of the system and provide different solutions by promoting the emergence of the swarm. In this line we will focus on the use of reinforcement learning, avoiding a manual design of the behavior in order to not restrict the resulting behaviors and to facilitate their adaptation. A detailed MDP model will be specified as well as the keys to reduce its dimensionality and facilitate its training. We will check the results of the obtained singling policy with respect to a greedy policy and focus on evaluating different behavioral strategies that can solve the problem in different ways. In addition, one of the obtained policies will be analyzed in detail to check both its robustness and its scalability with respect to the number of shepherds and sheep. This policy will be finally tested on a physical robotic swarm.

Increasing Biomanufacturing Yield with Bleed–Feed: Optimal Policies and Insights

Problem definition: Bleed–feed is a novel technology that allows biomanufacturers to skip intermediary bioreactor setups. However, the specific time at which bleed–feed is performed is critical for success. The process is stringently regulated, and its implementation involves unique tradeoffs in operational decision making. Our analysis formalizes the operational challenges related to bleed–feed decisions, and our results inform biomanufacturers and policymakers on the potential impact of this technology on current practice. Academic/practical relevance: Operations management (OM) methodologies have not yet been widely adopted in the biomanufacturing industry. This research presents one of the first attempts to demonstrate how OM can complement biomanufacturing to improve operational decisions. We present a practically relevant problem and a rigorous solution approach that is relevant to both OM and biomanufacturing. Methodology: We develop a finite-horizon, discrete-time Markov decision processes model and analyze the structural characteristics of optimal bleed–feed policies. Moreover, we characterize the behavior of the value function as a function of regulatory restrictions. As a salient feature, the MDP model captures both the biological dynamics of fermentation and the operational tradeoffs in biomanufacturing. Results: We show that optimal bleed–feed policies have a three-way control-limit structure under mild conditions that were validated with industry data. Moreover, our analysis reveals that the marginal benefits of bleed–feed diminish as additional bleed–feeds are performed. As a practically relevant benchmark, we consider a risk-averse heuristic and identify sufficient conditions for its optimality. Managerial implications: Our analysis (supported with an industry case study) shows that bleed–feed implementation can provide benefits. Real-world implementation at MSD resulted in an 82.5% improvement in the batch yield per setup (using one bleed–feed). We find that low-risk fermentation systems benefit the most from bleed–feed implementation. We also find that the performance gap between optimal policies and the risk-averse heuristic is higher when the failure risks or the critical biomass levels are lower. Funding: This work was supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek, NWO-VENI scheme. Supplemental Material: The online appendices are available at https://doi.org/10.1287/msom.2022.1163 .

Development of a hybrid model to plan segment based optimal promotion strategy

The study addresses the long-term effects of promotions in terms of movement in a value-based segmentation (lead, iron, gold, platinum), instead of simply looking at response rates that occur shortly after the promotion. The study develops a framework for planning an optimal promotion strategy via Markov Decision Processes and Machine Learning methods for an online department store. In the first phase, the states are set as the customer profitability segments in order to conduct the MDPs. Then, MDP model is solved, and the optimal decision for each segment is determined. In the second phase, in order to aid the company for making their plans for the next year, the segment that the customer will belong to next year should be predicted. Prediction of the future customer profitability segment is performed by using several machine learning algorithms, and the best performing model is selected. Using this best performing model, the company can predict the future (potential) profitability segment of the customer and make plans which include the optimal promotions that will be directed to the customers depending on their segments (these optimal promotions are the outcomes of the first phase). The proposed framework can be applied by practitioners in e-commerce companies which keep customer data.

Anxiety-Sensitive Planning: From Formal Foundations to Algorithms and Applications

Anxiety is the most prominent source of stress, harmful behaviours, and psychological disorders. AI systems, usually built for maximizing performance, increase the worldwide exposition to anxiety. This foundational paper introduces Anxiety-Aware Markov Decision Processes (AA-MDPs), the first formalism rooted in fundamental psychology research for modelling the anxiety tied to policies. In addition, this paper formalizes models and practical polynomial algorithms for generating anxiety-sensitive policies. Empirical validation demonstrates that AA-MDPs policies replicate the influence of anxiety on human decision-making observed by fundamental psychology research. Last, this paper demonstrates that AA-MDPs are directly applicable for social good, through a real-world use case (Anxiety-Sensitive Itinerary Planning), the immediate applicability for augmenting any formerly-defined MDP model with anxiety-awareness, and direct tracks developing future high-impact models.

A Migration Method for Vehicle Mobility Services Based on Road Segmentation Markov Model

The emergence of edge computing provides a new solution to solve resource-intensive applications, but how to formulate the most efficient service migration strategy according to users’ real-time location to ensure users to receive low latency services has become a new challenge. For the vehicle terminals running on the road, mobility of vehicle terminals is a factor that cannot be ignored. Ensuring the continuity of vehicle mobility service is an urgent problem to be solved. For the vehicles running in the roads, we focus on the establishment of service migration model based on Markov decision process according to road zoning, taking the loss and delay caused by service migration as the index to measure the advantages and disadvantages of service migration strategies. Then, a reinforcement learning algorithm is used to transform the service migration model into MDP model, and the measurement index is transformed into the reward function to formulate the service migration strategies for service migration process of the vehicle’s travel. Finally, through the simulation experiments on different scenarios, we compared our method with other methods in terms of loss and reward from service migration, which proves the effectiveness of our method.

How RL Agents Behave When Their Actions Are Modified

Reinforcement learning in complex environments may require supervision to prevent the agent from attempting dangerous actions. As a result of supervisor intervention, the executed action may differ from the action specified by the policy. How does this affect learning? We present the Modified-Action Markov Decision Process, an extension of the MDP model that allows actions to differ from the policy. We analyze the asymptotic behaviours of common reinforcement learning algorithms in this setting and show that they adapt in different ways: some completely ignore modifications while others go to various lengths in trying to avoid action modifications that decrease reward. By choosing the right algorithm, developers can prevent their agents from learning to circumvent interruptions or constraints, and better control agent responses to other kinds of action modification, like self-damage.

An Algorithmic Study to Maximize 5G Network Throughput Based on the Markov Decision Process

In this article, the network throughput optimization problem is investigated based on the theory of Markov decision process, combining the device-to-device direct selection problem with the finite stage discount MDP model problem. First, models for the device-to-device communication selection using MDP are built; second, the optimal mode selection strategy is derived using a finite stage backward iterative algorithm; and finally, the given mode selection strategy is evaluated by conducting a large number of simulation experiments. The results show that the MDP-based mode selection method proposed in this article has better performance in maximizing throughput and can yield better mode selection strategies with the advantage of obtaining larger system throughput.

Learning the Minimal Representation of a Dynamic System from Transition Data

This paper proposes a novel framework for learning a concise MDP model of a continuous state space dynamic system from given observed transition data. Most existing methods in offline reinforcement learning construct functional approximations of the value or the transition and reward functions, requiring complex and often not interpretable function approximators. Our approach instead relies on partitioning the system's feature space into regions constituting states of a finite deterministic MDP representing the system. We discuss what is theoretically the minimal MDP representation that preserves the values, and therefore the optimal policy, of the dynamic system. We define formally the problem of learning such a concise representation from transition data without exploration. To solve this problem, we introduce an in-sample property on partitions of the feature space we name coherence, and show that if the class of possible partitions is of finite VC dimension, any coherent partition with the transition data converges to the minimal representation of the system with provable finite-sample PAC convergence guarantees. This theoretical insight motivates our Minimal Representation Learning (MRL) algorithm that constructs from transition data an MDP representation that approximates the minimal representation of the system. We illustrate the effectiveness of the proposed framework through numerical experiments.

Web service composition on IoT reliability test based on cross entropy

AbstractWeb service has developed the managed IoT application to let connected devices easily and securely interact with cloud applications and other devices. As an important factor for web service, the reliability of web services refers to the probability of web service running success. For modeling web service composition, we should abstract the process of web service composition. Due to the diversity and complexity of web service composition, it is unlikely to do exhaustive testing. In order to improve the quality of web service composition test cases and find out which path leads to the greatest probability of service combination failure, heuristic test case generation method is adopted to obtain the optimal test path. First, the web service composition test is abstracted into the MDP model. The QoS of the web service composition is taken as the software test optimization goal, and the cross‐entropy strategy is used to optimize the test case. The experimental results show that the test profile given by the cross‐strategy is better than the random test strategy. Detect and exclude the same number of software defects. Cross‐entropy strategy can significantly reduce the number of test cases, reduce test costs, and improve defect detection efficiency.

Appointment Scheduling for a Health Care Facility with Series Patients

This study focuses on determining the appointment scheduling for healthcare facilities with series patients. “Series” patients are patients who are scheduled for a series of appointments instead of a single appointment. Examples of healthcare services with series patients include radiotherapy/chemotherapy for cancer, physical therapy, kidney dialysis, diabetes treatment, etc. The aim of this study is to design appointment scheduling policies taking into account revenues per service per patient, costs of staffing, overtime, overbooking and delay. The appointment scheduling problem is formulated using an MDP model. However, due to the huge state space, computing the optimal policy is impractical. Hence, we propose the Index Policy (IP) based on a one‐step policy improvement algorithm applied to the MDP model. We study a further simplification obtained by approximating the distribution of the number of patient visits by a Geometric distribution. A key analytical contribution is to prove the MDP to be a uni‐chain, which implies that there exists an optimal policy that maximizes the long‐run average profit. We also find that the IP provides a significant improvement over the other policies. Especially with the Geometric approximation, the IP requires minimal effort in implementing, and works almost as well. To test the effectiveness of our proposed IP in a real‐world setting, we use the data from a local physical therapy center to compare its performance with two other commonly used policies, namely, the Next Available Day Policy and the Shortest Queue Policy. We recommend the IP with Geometric approximation for series patients’ scheduling, which is computationally efficient and can significantly increases profits by incorporating the series feature of the patients’ appointments. Finally, we provide analysis that incorporates several practical considerations such as accounting for patient heterogeneity in number of visits and inter‐visit times, the option to reject new patients when the system is at full capacity, and incorporating patients with known number of visits at the time of the scheduling decision.

Multiuser Resource Control With Deep Reinforcement Learning in IoT Edge Computing

By leveraging the concept of mobile edge computing (MEC), massive amount of data generated by a large number of Internet of Things (IoT) devices could be offloaded to MEC server at the edge of wireless network for further computational intensive processing. However, due to the resource constraint of IoT devices and wireless network, both communications and computation resources need to be allocated and scheduled efficiently for better system performance. In this article, we propose a joint computation offloading and multiuser scheduling algorithm for IoT edge computing system to minimize the long-term average weighted sum of delay and power consumption under stochastic traffic arrival. We formulate the dynamic optimization problem as an infinite-horizon average-reward continuous-time Markov decision process (CTMDP) model. One critical challenge in solving this MDP problem for the multiuser resource control is the curse-of-dimensionality problem, where the state space of the MDP model and the computation complexity increase exponentially with the growing number of users or IoT devices. In order to overcome this challenge, we use the deep reinforcement learning (RL) techniques and propose a neural network architecture to approximate the value functions for the post-decision system states. The designed algorithm to solve the CTMDP problem supports semidistributed auction-based implementation, where the IoT devices submit bids to the BS to make the resource control decisions centrally. The simulation results show that the proposed algorithm provides significant performance improvement over the baseline algorithms, and also outperforms the RL algorithms based on other neural network architectures.

Matrix-Based Information Sharing Approach for Dynamic Robot Teams

In wireless connected robot swarms, information sharing is necessary for robots to cooperatively achieve joint goals under a partially observable environment. Despite significant progress on this problem, existing researchers have not adequately dealt with the case that robots broadcast messages to multiple members with a dynamically changing communication medium. In this paper, we propose an effective information sharing approach for dynamic robot teams. By using a matrix representation for information distribution, the network structure and information needs, this model allows quick reasoning through a single computation. Following our theoretical analysis, this approach can perform as the MDP model for small teams when states are fully observable. In addition, we designed heuristic algorithms to help robots adapt to dynamic network connections and information distribution in large teams. Empirical simulation results manifest that the approach performs well in uncertain environments to effectively balance sharing information with minimizing communication costs.

Feature Extraction and Mapping Construction for Mobile Robot via Ultrasonic MDP and Fuzzy Model.

This paper presents a modeling approach to feature classification and environment mapping for indoor mobile robotics via a rotary ultrasonic array and fuzzy modeling. To compensate for the distance error detected by the ultrasonic sensor, a novel feature extraction approach termed “minimum distance of point” (MDP) is proposed to determine the accurate distance and location of target objects. A fuzzy model is established to recognize and classify the features of objects such as flat surfaces, corner, and cylinder. An environmental map is constructed for automated robot navigation based on this fuzzy classification, combined with a cluster algorithm and least-squares fitting. Firstly, the platform of the rotary ultrasonic array is established by using four low-cost ultrasonic sensors and a motor. Fundamental measurements, such as the distance of objects at different rotary angles and with different object materials, are carried out. Secondly, the MDP feature extraction algorithm is proposed to extract precise object locations. Compared with the conventional range of constant distance (RCD) method, the MDP method can compensate for errors in feature location and feature matching. With the data clustering algorithm, a range of ultrasonic distances is attained and used as the input dataset. The fuzzy classification model—including rules regarding data fuzzification, reasoning, and defuzzification—is established to effectively recognize and classify the object feature types. Finally, accurate environment mapping of a service robot, based on MDP and fuzzy modeling of the measurements from the ultrasonic array, is demonstrated. Experimentally, our present approach can realize environment mapping for mobile robotics with the advantages of acceptable accuracy and low cost.

Wireless Power Transfer and Data Collection in Wireless Sensor Networks

In a rechargeable wireless sensor network, the data packets are generated by sensor nodes at a specific data rate, and transmitted to a base station. Moreover, the base station transfers power to the nodes by using wireless power transfer (WPT) to extend their battery life. However, inadequately scheduling WPT and data collection causes some of the nodes to drain their battery and have their data buffer overflow, whereas the other nodes waste their harvested energy, which is more than they need to transmit their packets. In this paper, we investigate a novel optimal scheduling strategy, called EHMDP, aiming to minimize data packet loss from a network of sensor nodes in terms of the nodes' energy consumption and data queue state information. The scheduling problem is first formulated by a centralized MDP model, assuming that the complete states of each node are well known by the base station. This presents the upper bound of the data that can be collected in a rechargeable wireless sensor network. Next, we relax the assumption of the availability of full state information so that the data transmission and WPT can be semidecentralized. The simulation results show that, in terms of network throughput and packet loss rate, the proposed algorithm significantly improves the network performance.