Markov Decision-making Process Research Articles

Research on reactive power voltage control strategy of distribution networks containing distributed PV

Abstract With the large-scale increase in photovoltaic power generation, its integration into the grid will affect the system voltage of the distribution network. This paper presents a reactive voltage control strategy for the distribution network based on deep reinforcement learning. Firstly, the objective function model is established based on the node voltage and network loss of the distribution network. Secondly, each photovoltaic inverter is modeled as an agent of the enhanced learning environment. The collaborative control problem of the photovoltaic inverter is transformed into an observable Markov decision-making process through the distribution network partition. The paper uses a double delay depth deterministic gradient algorithm to solve the problem. Finally, combined with the improvement of 33 nodes for verification, the proposed strategy has the stability of improving voltage, presenting an ability to effectively reduce network loss and verify the effectiveness of the proposed strategy.

Modeling the Decision and Coordination Mechanism of Power Battery Closed-Loop Supply Chain Using Markov Decision Processes

With the rapid growth of the new energy vehicle market, efficient management of the closed-loop supply chain of power batteries has become an important issue. Effective closed-loop supply chain management is very critical, which is related to the efficient utilization of resources, environmental responsibility, and the realization of economic benefits. In this paper, the Markov Decision Process (MDP) is used to model the decision-making and coordination mechanism of the closed-loop supply chain of power batteries in order to cope with the challenges in the management process, such as cost, quality, and technological progress. By constructing the MDP model for different supply chain participants, this paper investigates the optimization strategy of the supply chain and applies two solution methods: dynamic programming and reinforcement learning. The case study results show that the model can effectively identify optimized supply chain decisions, improve the overall efficiency of the supply chain, and coordinate the interests among parties. The contribution of this study is to provide a new modeling framework for power battery recycling and to demonstrate the practicality and effectiveness of the method with empirical data. This study demonstrates that the Markov decision-making process can be a powerful tool for closed-loop supply chain management, promotes a deeper understanding of the complex decision-making environment of the supply chain, and provides a new solution path for decision-making and coordination in the supply chain.

Research on Reinforcement Learning Algorithm-Based Practical Teaching and Reform Oriented to Enhancement of College Students’ Vocational Ability in Colleges and Universities

Abstract In this paper, we first explored the mathematical model of reinforcement learning - Markov decision-making process, solved the Bellman optimal solution by strategy iteration and strategy search method, and realized dynamic planning according to the Bellman optimal solution. Then, the students’ satisfaction with practical teaching and the level of students’ vocational ability cultivation were evaluated, and the influence of practical teaching on students’ vocational ability and vocational adaptability was analyzed. Finally, the reform path for practice teaching in colleges and universities was proposed using the reinforcement learning algorithm and impact analysis. The results show that the learning effect of students in the practice teaching environment is 3.8, the learning effect in the ordinary teaching environment is 3.25, and the effect of practice teaching is significant. The correlation coefficient between practice effect and students’ vocational ability is 0.537, which indicates that practice teaching affects students’ vocational ability. This study provides an important reference value for the construction of practical teaching in colleges and universities, as well as for the reform of the college education system.

Multi-agent optimal scheduling for integrated energy system considering the global carbon emission constraint

In a multi-regional integrated energy system (MIES), optimal scheduling under random renewable supply and user demand is crucial to promote the process of carbon neutrality. Further, the total carbon emission of multiple regions is expected to strictly restricted under a threshold, while intensifying the complex coupling of multiple agents. To address the optimal dispatching problem, we establish a typical MIES model with the global carbon emission constraint, which is formulated as a partially observable Markov decision-making process (POMDP). Then we propose an improved multi-agent deep deterministic policy gradient (MADDPG) method, which utilizes a centralized training and decentralized execution (CTDE) framework to effectively improve the multi-agent stationarity. Moreover, an attention mechanism is employed to enhance the efficiency of communication and coordination among agents. Experiments are carried out on multi-regional datasets, and the results certify that the proposed algorithm can decrease system operation costs, reduce carbon emissions, and speed up the convergence of the multi-agent system.

Collaborative Optimization Scheduling of Resilience and Economic Oriented Islanded Integrated Energy System under Low Carbon Transition

With the development of new energy sources and the increase in the installed scale of energy coupling equipment, the low-carbon transformation of the energy supply of the integrated energy system (IES) has a serious impact on the reliability of the IES supply, and there is an urgent need for a reasonable and accurate assessment and trade-off between the IES resilience and economics. In this regard, this paper models the overall optimization of the resilience and economic configuration and operation scheduling of the IES in the islanded operation mode after grid faults, proposes a two-layer optimization strategy model of resilience and economy, and solves the unit configuration, coupled output characteristics, and optimal scheduling of the islanded IES using the Markov decision-making process and forbearing stratified sequencing method, and evaluates and analyzes the resilience and cost of the various types of IES configuration schemes. Resilience and cost are also evaluated and analyzed. Finally, an example analysis is carried out in an electric-heat-cooling integrated energy system. The results show that the proposed two-tier optimization strategy model can optimize the IES configuration scheme and coordinate the scheduling of each equipment, and the overall annualized cost of the energy system decreases by CNY 45.21 thousand, or a percentage decrease of 5.24%, compared to the same configuration of the conventional strategy. The typical day toughness index improved by 7.33%, 7.56%, and 13.01% in the spring, summer, and autumn, respectively.

Collaborative planning of multi-tier sustainable supply chains: A reinforcement learning enhanced heuristic approach

Despite the growing importance of collaboration in achieving sustainability-related advantages for companies, existing studies lack a systematic framework to determine how multiple supply chains can jointly facilitate strategical decision-making to achieve the objectives in the triple-bottom-line (3BL). In this study, we suggest a comprehensive mixed-integer linear programming model for multi-network collaboration considering 3BL sustainability indicators and develop a heuristic approach enhanced by reinforcement learning to solve the model. The proposed model allows for optimal decision-making across multiple sustainable supply chains, simultaneously minimizing total costs and environmental impacts as well as maximizing social responsibility. The heuristic algorithm integrates a Markov decision-making process and information accumulation mechanism with the exploration of the solution space. It effectively learns from the solving process, and applies the most appropriate operator to iteratively improve the current solution according to the knowledge learnt. Extensive experiments based on real-world data are conducted and the results demonstrate that the proposed model and solution framework yield an effective collaborative supply chain design for each actor with superior efficiency and accuracy. Compared with CPLEX, the average solving times for medium-to-large instance scales are reduced by 16.34% to 87.59%, and 86.67% of the instances saw an improvement of solution quality, with an average improvement of 5.67%. Moreover, the inclusion of horizontal transportation in the proposed model provides a significant improvement of 51.24% in the economic bottom-line, as well as an improvement of 3.42% in the environmental bottom-line.

A Full-State Reliability Analysis Method for Remanufactured Machine Tools Based on Meta Action and a Markov Chain Using an Exercise Machine (EM) as an Example

The reliability of an RMT can be regarded as an important indicator customers can use to recognize its quality; however, it is difficult to implement a full-state reliability analysis of an RMT due to its complicated structural functions. Therefore, a full-state reliability analysis model is proposed herein based on meta action (MA) and a Markov chain for remanufactured exercise machine tools (REMTs). First, an analysis was carried out on individual levels by integrating the MAU decomposition method, and an MAU fault tree model was established layer by layer for the REMT. Second, full-state modeling was performed in view of the MAU characteristics of the REMT, whose operation processes are divided into MAU normal and failure states. A Markov decision-making process was introduced to integrate MAU states and establish our model, which was solved by means of an analytical method for the evaluation of reliability. Finally, an example of a remanufactured machine tool spindle is given to verify the effectiveness of the method.

Dynamic Regimes for Corporate Human Capital Development Used Reinforcement Learning Methods

Corporate human capital is a critical driver of sustainable economic growth, which is becoming increasingly important in the changing nature of work. Due to the expansion of various areas of human activity, the employee’s profile becomes multifaceted. Therefore, the problem of human capital management based on the individual trajectories of professional development, aimed at increasing the labor efficiency and contributing to the growth of the corporate operational efficiency, is relevant, timely, socially, and economically significant. The paper proposes a methodology for the dynamic regimes for human capital development (DRHC) to design individual trajectories for the employee’s professional development, based on reinforcement learning methods. The DRHC develops an optimal management regime as a set of programs aimed at developing an employee in the professional field, taking into account their individual characteristics (health quality, major and interdisciplinary competencies, motivation, and social capital). The DRHC architecture consists of an environment—an employee model—as a Markov decision-making process and an agent—decision-making center of a company. The DRHC uses DDQN, SARSA, and PRO algorithms to maximize the agent’s utility function. The implementation of the proposed DRHC policy would improve the quality of corporate human capital, increase labor resource efficiency, and ensure the productivity growth of companies.

A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat

Unmanned combat air vehicle (UCAV) combat is a challenging scenario with high-dimensional continuous state and action space and highly nonlinear dynamics. In this article, we propose a general hierarchical framework to resolve the within-vision-range (WVR) air-to-air combat problem under six dimensions of degree (6-DOF) dynamics. The core idea is to divide the whole decision-making process into two loops and use reinforcement learning (RL) to solve them separately. The outer loop uses a combat policy to decide the macro command according to the current combat situation. Then the inner loop uses a control policy to answer the macro command by calculating the actual input signals for the aircraft. We design the Markov decision-making process for the control policy and the Markov game between two aircraft. We present a two-stage training mechanism. For the control policy, we design an effective reward function to accurately track various macro behaviors. For the combat policy, we present a fictitious self-play mechanism to improve the combat performance by combating against the historical combat policies. Experiment results show that the control policy can achieve better tracking performance than conventional methods. The fictitious self-play mechanism can learn competitive combat policy, which can achieve high winning rates against conventional methods.

Spatiotemporal path tracking via deep reinforcement learning of robot for manufacturing internal logistics

The development of engineering technology, the logistics system composed of wheeled mobile robots (WMR) and automated guided vehicles (AGV) have been widely used in industrial scenes. However, traditional manufacturing scenes with dense layout, such as weaving workshops, have higher requirements for automatic transportation of materials. How to realize the intelligent transportation and management of materials in the manufacturing workshop and balance the efficiency and safety of material processing and transportation is still a great challenge. To address this problem, we designed a logistics system based on cloth-roll handling robot (CHR) and its path tracking hybrid deep reinforcement learning (DRL) considering spatiotemporal efficiency and safety in weaving workshop. This research first focuses on the design of a dynamic observation Markov decision-making process that integrates scene features. Further, a deep reinforcement learning considering the heterogeneity of observation data is proposed to obtain the optimal action solution. Then, a distributed scenarios training is implemented to improve the interaction ability between agents and the environment in complex scenes. In addition, the balance between dynamic observation and on-site calculation is considered in the path tracking for actual weaving workshop.

End-to-end active non-cooperative target tracking of free-floating space manipulators

Active non-cooperative target tracking is an important ability for free-floating space manipulator (FFSM) systems. Aiming at the difficulty of pose and posture estimation of non-cooperative targets and the complexity of FFSM planning and control, the paper proposed an end-to-end target tracking algorithm using deep reinforcement learning (DRL) algorithm. The algorithm uses raw camera images as input and directly outputs the manipulator joint velocities. To deal with the partially observable Markov decision processes (POMDP) problem caused by single image input, we combine the soft actor–critic algorithm with recurrent neural networks. Therefore, the velocity information of the target is introduced to transform the task into a fully observable Markov decision-making process (MDP). Our approach is completely data-driven. It avoids the position and posture estimation of non-cooperative target and the complex modeling motion planning process of FFSM. We trained and tested the algorithm by building a simulation environment in CoppeliaSim. The results showed that our approach achieved good results in active non-cooperative target tracking of FFSM systems and demonstrated the great potential of DRL algorithms in solving space tasks.

DHGECON: A multi-round conversational recommendation method based on dynamic heterogeneous encoding

Multi-round conversational recommendation (MCR), fulfilling a real–time recommendation task for users through interactively asking attributes and recommending items, can be regarded as a multi-step Markov decision-making process. Thus, in MCR, the key point is how to appropriately guide and characterize the dynamic interactive process in the conversation and to mine the dynamic relationship among the user, items and attributes to determine such policies as when to ask (attributes) and recommend (items), what (attributes) to ask and what (items) to recommend. Recent works mainly use statistical information involved in the conversation process to characterize the conversation without comprehensively considering the dynamic relationship among the user, items and attributes and may consequently have a negative impact on accurate capture of the user’s real-time preferences. To address this issue, we propose a multi-round conversational recommendation method based on dynamic heterogeneous encoding called DHGECON. Firstly, a dynamic heterogeneous graph with three types of node (users, items, and attributes) is constructed to characterize the proceeding conversation. Secondly, a heterogeneous graph-based encoder which adaptively updates the attention weight of nodes is designed to mine and represent the dynamic high-order semantic relationship among the user, items and attributes. Finally, the encoded information is fed into the decision-making module to get the action (i.e., recommending items or asking attributes) for guiding the next round conversation. Experimental results show that, compared with the state-of-the-art existing methods, the proposed method has a significant improvement in terms of major evaluation metrics over four real-world datasets.

Research on the Control of Multi-Agent Microgrid with Dual Neural Network Based on Priority Experience Storage Policy

In this paper, an improved dual neural network control method based on multi-agent system is proposed to solve the problem of rating the frequency deviation and voltage deviation of the microgrid system due to the uneven impedance distribution of the circuit. The microgrid multi-agent system control model is constructed; the microgrid operation problem is transformed into Markov decision-making process, and the frequency error model of distributed secondary control adjusting system is established. In the course of training, the priority experience replay mechanism is introduced to accelerate the training reward return by using the experience of high feedback reward, and the frequency and voltage bias of the microgrid system are reduced. The model of isolated island microgrid of distributed power supply communication topology is established, and the control strategy of double neural network is simulated. Compared with the traditional sagging control method, the double neural network algorithm proposed in this paper stabilizes the frequency of the grid at rated frequency and improves the convergence speed. Simulation results show that the proposed method is helpful to provide stable and high-quality power resources for enterprises.

A Machine Learning-Based Energy Management Agent for Fine Dust Concentration Control in Railway Stations

This study developed a reinforcement learning-based energy management agent that controls the fine dust concentration by controlling facilities such as blowers and air conditioners to efficiently manage the fine dust concentration in the station. To this end, we formulated an optimization problem based on the Markov decision-making process and developed a model for predicting the concentration of fine dust in the station by training an artificial neural network (ANN) based on supervised learning to develop the transfer function. In addition to the prediction model, the optimal policy for controlling the blower and air conditioner according to the current state was obtained based on the ANN to which the Deep Q-Network (DQN) algorithm was applied. In the case study, it is confirmed that the ANN and DQN of the predictive model were trained based on the actual data of Nam-Gwangju Station to converge to the optimal policy. The comparison between the proposed method and conventional method shows that the proposed method can use less power consumption but achieved better performance on reducing fine dust concentration than the conventional method. In addition, by increasing the value of the ratio that represents the compensation due to the fine dust reduction, the learned agent achieved more reduction on the fine dust concentration by increasing the power consumption of the blower and air conditioner.

Curriculum Personalization through the Application of Reinforcement Learning Algorithms

Individual curriculum Personalization contributes to improving the education quality. It takes into account the individual issues of each student and changes the program accordingly. This article discusses the application of recommender algorithms and Markov decision-making process to the problem of making an individual students schedule. The algorithm is based on the collected historical data of assessments both to identify the most effective lesson chains among all students, and to combine individual lesson chains.

An unsolvable power flow adjustment method for weak power grid based on transmission channel positioning and deep reinforcement learning

To solve the problem of human resources and time consumption caused by unsolvable power flow of weak power grid, an adjustment method for unsolvable power flow based on weak transmission channels and deep reinforcement learning (DRL) is proposed. Weak horizontal and vertical channels are defined, and their relationship with unsolvable power flow is demonstrated. Then, based on the idea of reducing the load level first and then gradually recovering it, actionable device and action margin are determined based on transmission channel positioning, adjustment sensitivity and solvability degree, so as to form a power flow adjustment strategy. Next, the Markov decision-making process of power flow adjustment is constructed, the prior probability of action is given by adjustment sensitivity of the actionable device, and the action is mapped to the adjustment amounts of generators and capacitors/reactors. Based on soft actor-critic, a DRL model suitable for power flow adjustment is established. Finally, the simulation results show that the proposed method performs well both in the New England 39-bus standard system and an actual power grid of China. Compared with the results obtained by other methods, the superiority of the proposed method is verified.

BLMDP: A new bi-level Markov decision process approach to joint bidding andtask-scheduling in cloud spot market

In the cloud computing market (CCM), computing services are traded between cloud providers and consumers in the form of the computing capacity of virtual machines (VMs). The Amazon spot market is one of the most well-known markets in which the surplus capacity of data centers is auctioned off in the form of VMs at relatively low prices. For each submitted task, the user can offer a price that is higher than the current price. However, uncertainty in the market environment confronts the user with challenges such as the variable price of VMs and the variable number of users. An appropriate strategy should both maximize the user?s utility and determine the best bid for him/her. In this paper, we aim to solve the problem of joint minimizing the cost of processing tasks and maximizing user satisfaction. Our proposed method, which we call bi-level Markov decision-making process (BLMDP), works on two levels. At the top level, it selects the most appropriate user bids and adjusts the spot price to minimize the cost of VMs on the cloud provider side. At a low level, it decides to admit tasks in such a way as to maximize the user side satisfaction. Performance evaluation based on real data collected from Amazon web services shows that BLMDP manages to minimize cloud provider's costs and maximize user gain more effectively compared to heuristic methods.

Data‐model driven rescheduling considering both rotor angle stability and transient voltage stability constraints

Facing the problem of workforce and time costs caused by the rescheduling of large-scale power grids with renewable energy penetration, and the lack of parallel control research on rotor angle instability and transient voltage instability, a rescheduling method based on trajectory sensitivity and deep reinforcement learning (DRL) is proposed to meet the requirements of rotor angle stability and transient voltage stability at the same time. By introducing the process of rescheduling to meet the transient stability constraint, the Markov decision-making process of rescheduling is first constructed. Then, a simple dominant instability type identification method is proposed according to the occurrence time, location, voltage pattern, and position of the oscillation centre. Moreover, a rescheduling strategy is formulated based on the dominant instability mode identification, trajectory sensitivity calculation, actionable device selection, action amount computation, and action of the device. Next, according to the improved distributed distributive deep deterministic policy gradients (D4PG), a DRL model is established to map the action to actionable generator pairs and capacitors/reactors, so as to realize parallel rescheduling of rotor angle instability and transient voltage instability. Finally, the improved New England 39-bus system and an actual power grid are used to verify the effectiveness and advantages of the method.

The Quantization Algorithm Based on DRL

Maximizing returns is always people’s investment goal. Gold and Bitcoin have some hedging ability, and their prices fluctuate greatly, making them popular varieties for investors. However, the market is risky, and different economic, political and environmental factors will impact the market. As a result, Bitcoin and gold prices fluctuate sharply, leading to uncertain investment and uncertain returns. To maximize returns, we model time and price, analyze and model the price trends of gold and bitcoin in problems, give the best trading model, and analyze the transaction cost sensitivity of transaction risk and trading strategies. Predictive techniques are used to move the future and to build some heuristic-based robots to make decisions. Use (DRL) deep reinforcement learning to simulate stock trading based on the Markov decision-making process of avoiding risk aversion, reducing trading costs, maintaining liquidity, and assuming that the stock market is not affected by enhanced trading agents. The second step is to use the risk analysis of the Sharp ratio to compare with the portfolio-managed Markowitz effective boundary model to select a portfolio with medium risk and high yield in the third step, to effectively calculate the sensitivity of the transaction strategy to the transaction cost, we used the time series to calculate the change difference between the final result and the standard value after the transaction cost change. A system of equations based on a number of yield values can be established to meet the optimization requirements, introduce computational historical data, and replace the relevant transaction costs with the above model to clarify the change curve of the final return results. Later, ac- cording to the needs of the trading strategy, complete the sensitivity analysis of the transaction strategy on the trading cost (calculate the impact of variable factors on profit), determine the sensitivity anal- ysis mode, and evaluate the advantages and disadvantages of the model combined with the above results. Ultimately, our strategy and models are passed on to traders as memos.

Transfer-Reinforcement-Learning-Based rescheduling of differential power grids considering security constraints

The power system rescheduling based on model-free methods has obvious defects in practical application, such as poor scenario transferability, long data training time, and waste of domain knowledge. To overcome the above defects, a transfer-reinforcement-learning-based rescheduling method of differential power grids considering security constraints is proposed. When constructing the Markov decision-making process of security-constrained rescheduling, both the off-limits of line power and node voltage are considered in the reward. The action space of deep reinforcement learning is narrowed by calculating the sensitivities of devices and mapped to control the active and reactive power regulating devices to reschedule active and reactive power simultaneously. According to the change degree of transfer object, the applications of transfer learning are divided into two scenarios. For the security-constrained rescheduling transfer scenario of different structures of the same power grid, a domain-adaption transfer learning method is formed, realizing good data adaptability after structure changes with the original model. Moreover, a policy-based transfer learning method is constructed for the security-constrained rescheduling transfer scenario of different power grids, enhancing the training speed and training effect of target power grid. Two standard systems and two actual power grids are utilized to verify the effectiveness of the method. For the actual power grids, the effects of the two scenarios are improved by 5.8% and 3.9% with transfer learning. Compared with other methods, this method not only has obvious advantages in transferability, but also has a shorter learning process and lower control cost.