Actor-critic Reinforcement Learning Approach Research Articles

Simultaneous shovel allocation and grade control decisions for short-term production planning of industrial mining complexes – an actor-critic approach

ABSTRACT Short-term planning typically involves optimising different mining complex components individually, preventing unlocking value from intercorrelated activities. This paper proposes an integrated framework that defines diglines and shovel allocation decisions to improve the operation’s financial performance. This involves a clustering approach, defining diglines and material destination; a simulator of the mining complex operations, forecasting the material flow from benches to processors; and an actor-critic reinforcement learning approach, assigning shovels to mining areas given operating requirements. A case study at a copper mining complex shows 27% cash flow improvements compared to a non-adaptive baseline approach provided by recent advances in the field.

Integrating short-term stochastic production planning updating with mining fleet management in industrial mining complexes: an actor-critic reinforcement learning approach

Short-term production planning in industrial mining complexes involves defining daily, weekly or monthly decisions that aim to achieve production targets established by long-term planning. Operational requirements must be considered when defining fleet allocation and production scheduling decisions. Thus, this paper presents an actor-critic reinforcement learning (RL) method to make mining equipment allocation and production scheduling decisions that maximize the profitability of a mining operation. Two RL agents are proposed. The first agent allocates shovels to mining fronts by considering some operational requirements. The second agent defines the processing destination and the number of trucks required for transportation. A simulator of mining complex operations is proposed to forecast the material flow from the mining fronts to the destinations. This simulator provides new states and rewards to the RL agents, so shovel allocation and production scheduling decisions can be improved. Additionally, as the mining complex operates, sensors collect ore quality data, which are used to update the uncertainty associated with the orebody models. The improvement in material supply characterization allows the RL agents to make more informed decisions. A case study applied at a copper mining complex highlights the method’s ability to make informed decisions while collecting new data. The results show a 47% improvement in cash flow by adapting the shovel and truck allocation and material destination compared to a base case with predefined fleet assignments.

Learning-Based Traffic Scheduling in Non-Stationary Multipath 5G Non-Terrestrial Networks

In non-terrestrial networks, where low Earth orbit satellites and user equipment move relative to each other, line-of-sight tracking and adapting to channel state variations due to endpoint movements are a major challenge. Therefore, continuous line-of-sight estimation and channel impairment compensation are crucial for user equipment to access a satellite and maintain connectivity. In this paper, we propose a framework based on actor-critic reinforcement learning for traffic scheduling in non-terrestrial networks scenario where the channel state is non-stationary due to the variability of the line of sight, which depends on the current satellite elevation. We deploy the framework as an agent in a multipath routing scheme where the user equipment can access more than one satellite simultaneously to improve link reliability and throughput. We investigate how the agent schedules traffic in multiple satellite links by adopting policies that are evaluated by an actor-critic reinforcement learning approach. The agent continuously trains its model based on variations in satellite elevation angles, handovers, and relative line-of-sight probabilities. We compare the agent’s retraining time with the satellite visibility intervals to investigate the effectiveness of the agent’s learning rate. We carry out performance analysis while considering the dense urban area of Paris, where high-rise buildings significantly affect the line of sight. The simulation results show how the learning agent selects the scheduling policy when it is connected to a pair of satellites. The results also show that the retraining time of the learning agent is up to 0.1times the satellite visibility time at given elevations, which guarantees efficient use of satellite visibility.

Actor-critic reinforcement learning for bidding in bilateral negotiation

Designing an effective and intelligent bidding strategy is one of the most compelling research challenges in automated negotiation, where software agents negotiate with each other to find a mutual agreement when there is a conflict of interests. Instead of designing a hand-crafted decision-making module, this work proposes a novel bidding strategy adopting an actor-critic reinforcement learning approach, which learns what to offer in a bilateral negotiation. An entropy reinforcement learning framework called Soft Actor-Critic (SAC) is applied to the bidding problem, and a self-play approach is employed to train the model. Our model learns to produce the target utility of the coming offer based on previous offer exchanges and remaining time. Furthermore, an imitation learning approach called behavior cloning is adopted to speed up the learning process. Also, a novel reward function is introduced that does take not only the agent?s own utility but also the opponent?s utility at the end of the negotiation. The developed agent is empirically evaluated. Thus, a large number of negotiation sessions are run against a variety of opponents selected in different domains varying in size and opposition. The agent?s performance is compared with its opponents and the performance of the baseline agents negotiating with the same opponents. The empirical results show that our agent successfully negotiates against challenging opponents in different negotiation scenarios without requiring any former information about the opponent or domain in advance. Furthermore, it achieves better results than the baseline agents regarding the received utility at the end of the successful negotiations.

Blockchain and MEC-assisted reliable billing data transmission over electric vehicular network: An actor—critic RL approach

Recently, electric vehicles (EVs) have been widely used under the call of green travel and environmental protection, and diverse requirements for charging are also increasing gradually. In order to ensure the authenticity and privacy of charging information interaction, blockchain technology is proposed and applied in charging station billing systems. However, there are some issues in blockchain itself, including lower computing efficiency of the nodes and higher energy consumption in the consensus process. To handle the above issues, in this paper, combining blockchain and mobile edge computing (MEC), we develop a reliable billing data transmission scheme to improve the computing capacity of nodes and reduce the energy consumption of the consensus process. By jointly optimizing the primary and replica nodes offloading decisions, block size and block interval, the transaction throughput of the blockchain system is maximized, as well as the latency and energy consumption of the system are minimized. Moreover, we formulate the joint optimization problem as a Markov decision process (MDP). To tackle the dynamic and continuity of the system state, the reinforcement learning (RL) is introduced to solve the MDP problem. Finally, simulation results demonstrate that the performance improvement of the proposed scheme through comparison with other existing schemes.

Online and energy-efficient task-processing for distributed edge networks

User equipment produces a series of tasks that are processed locally or remotely, falling into three categories: (i) local computing only, (ii) a fraction of the task is computed locally and the remaining task unprocessed is offloaded for remote computation, and (iii) the entire task is offloaded. Each case has attracted substantial attention in recent studies, where a delay-constrained non-linear optimization problem is often formulated. The solutions employed are either based on Lagrange duality, heuristic search, or dynamic programming. To our knowledge, there is no unifying task-processing orchestrator that is an online tailored solver for learning the model-free problems, encapsulating the three cases above. We fill this gap and present the first attempt on an innovative actor-critic reinforcement learning approach in consideration of the energy-efficiency, to compute the asymptotically optimal solutions via decomposing the comprehensive optimization into sub-problems. Rigorous theoretical analyses and experience-driven simulations demonstrate significant advantages over the benchmark approaches, in terms of task-processing delay, power efficiency, and convergence time.

CRSAL

Recommender systems have been attracting much attention from both academia and industry because of their ability to capture user interests and generate personalized item recommendations. As the life pace in contemporary society speeds up, traditional recommender systems are inevitably limited by their disconnected interaction styles and low adaptivity to users’ evolving demands. Consequently, conversational recommender systems emerge as a prospective research area, where an intelligent dialogue agent is integrated with a recommender system. Conversational recommender systems possess the ability to accurately understand end-users’ intent or request and generate human-like dialogue responses when performing recommendations. However, existing conversational recommender systems only allow the systems to ask users for more preference information, while users’ further questions and concerns about the recommended items (e.g., enquiring the location of a recommended restaurant) can hardly be addressed. Though the recent task-oriented dialogue systems allow for two-way communications, they are not easy to train because of their high dependence on human guidance in terms of user intent recognition and system response generation. Hence, to enable two-way human-machine communications and tackle the challenges brought by manually crafted rules, we propose Conversational Recommender System with Adversarial Learning (CRSAL), a novel end-to-end system to tackle the task of conversational recommendation. In CRSAL, we innovatively design a fully statistical dialogue state tracker coupled with a neural policy agent to precisely capture each user’s intent from limited dialogue data and generate conversational recommendation actions. We further develop an adversarial Actor-Critic reinforcement learning approach to adaptively refine the quality of generated system actions, thus ensuring coherent human-like dialogue responses. Extensive experiments on two benchmark datasets fully demonstrate the superiority of CRSAL on conversational recommendation tasks.

Design and Experimental Validation of a Cooperative Adaptive Cruise Control System Based on Supervised Reinforcement Learning

This paper presents a supervised reinforcement learning (SRL)-based framework for longitudinal vehicle dynamics control of cooperative adaptive cruise control (CACC) system. A supervisor network trained by real driving data is incorporated into the actor-critic reinforcement learning approach. In the SRL training process, the actor and critic network are updated under the guidance of the supervisor and the gain scheduler. As a result, the training success rate is improved, and the driver characteristics can be learned by the actor to achieve a human-like CACC controller. The SRL-based control policy is compared with a linear controller in typical driving situations through simulation, and the control policies trained by drivers with different driving styles are compared using a real driving cycle. Furthermore, the proposed control strategy is demonstrated by a real vehicle-following experiment with different time headways. The simulation and experimental results not only validate the effectiveness and adaptability of the SRL-based CACC system, but also show that it can provide natural following performance like human driving.

User Scheduling and Resource Allocation in HetNets With Hybrid Energy Supply: An Actor-Critic Reinforcement Learning Approach

Densely deployment of various small-cell base stations in cellular networks to increase capacity will lead to heterogeneous networks (HetNets), and meanwhile, embedding the energy harvesting capabilities in base stations as an alternative energy supply is becoming a reality. How to make efficient utilization of radio resource and renewable energy is a brand-new challenge. This paper investigates the optimal policy for user scheduling and resource allocation in HetNets powered by hybrid energy with the purpose of maximizing energy efficiency of the overall network. Since wireless channel conditions and renewable energy arrival rates have stochastic properties and the environment’s dynamics are unknown, the model-free reinforcement learning approach is used to learn the optimal policy through interactions with the environment. To solve our problem with continuous-valued state and action variables, a policy-gradient-based actor-critic algorithm is proposed. The actor part uses the Gaussian distribution as the parameterized policy to generate continuous stochastic actions, and the policy parameters are updated with the gradient ascent method. The critic part uses compatible function approximation to estimate the performance of the policy and helps the actor learn the gradient of the policy. The advantage function is used to further reduce the variance of the policy gradient. Using the numerical simulations, we demonstrate the convergence property of the proposed algorithm and analyze network energy efficiency.