Reinforcement Learning Procedure Research Articles

Reliable Data Delivery Using Fuzzy Reinforcement Learning in Wireless Sensor Networks

Wireless sensor networks (WSNs) has been envisioned as a potential paradigm in sensing technologies. Achieving energy efficiency in a wireless sensor network is challenging since sensor nodes have confined energy. Due to the multi-hop communication, sensor nodes spend much energy re-transmitting dropped packets. Packet loss may be minimized by finding efficient routing paths. In this research, a routing using fuzzy logic and reinforcement learning procedure is designed for WSNs to determine energy-efficient paths; to achieve reliable data delivery. Using the node’s characteristics, the reward is determined via fuzzy logic. For this paper, we employ reinforcement learning to improve the rewards, computed by considering the quality of the link, available free buffer of node, and residual energy. Further, simulation efforts have been made to illustrate the proposed mechanism’s efficacy in energy consumption, delivery delay of the packets, number of transmissions, and lifespan.

An Integrated Model of Collaborative Skill Acquisition: Anticipation, Control Tuning, and Role Adoption.

We studied collaborative skill acquisition in a dynamic setting with the game Co-op Space Fortress. While gaining expertise, the majority of subjects became increasingly consistent in the role they adopted without being able to communicate. Moreover, they acted in anticipation of the future task state. We constructed a collaborative skill acquisition model in the cognitive architecture ACT-R that reproduced subject skill acquisition trajectory. It modeled role adoption through reinforcement learning and predictive processes through motion extrapolation and learned relevant control parameters using both a reinforcement learning procedure and a new to ACT-R supervised learning procedure. This is the first integrated cognitive model of collaborative skill acquisition and, as such, gives us valuable insights into the multiple cognitive processes that are involved in learning tocollaborate.

Anomaly Detection and Correction of Optimizing Autonomous Systems With Inverse Reinforcement Learning.

This article considers autonomous systems whose behaviors seek to optimize an objective function. This goes beyond standard applications of condition-based maintenance, which seeks to detect faults or failures in nonoptimizing systems. Normal agents optimize a known accepted objective function, whereas abnormal or misbehaving agents may optimize a renegade objective that does not conform to the accepted one. We provide a unified framework for anomaly detection and correction in optimizing autonomous systems described by differential equations using inverse reinforcement learning (RL). We first define several types of anomalies and false alarms, including noise anomaly, objective function anomaly, intention (control gain) anomaly, abnormal behaviors, noise-anomaly false alarms, and objective false alarms. We then propose model-free inverse RL algorithms to reconstruct the objective functions and intentions for given system behaviors. The inverse RL procedure for anomaly detection and correction has the training phase, detection phase, and correction phase. First, inverse RL in the training phase infers the objective function and intention of the normal behavior system using offline stored data. Second, in the detection phase, inverse RL infers the objective function and intention for online observed test system behaviors using online observation data. They are then compared with that of the nominal system to identify anomalies. Third, correction is executed for the anomalous system to learn the normal objective and intention. Simulations and experiments on a quadrotor unmanned aerial vehicle (UAV) verify the proposed methods.

Policy-Independent Behavioral Metric-Based Representation for Deep Reinforcement Learning

Behavioral metrics can calculate the distance between states or state-action pairs from the rewards and transitions difference. By virtue of their capability to filter out task-irrelevant information in theory, using them to shape a state embedding space becomes a new trend of representation learning for deep reinforcement learning (RL), especially when there are explicit distracting factors in observation backgrounds. However, due to the tight coupling between the metric and the RL policy, such metric-based methods may result in less informative embedding spaces which can weaken their aid to the baseline RL algorithm and even consume more samples to learn. We resolve this by proposing a new behavioral metric. It decouples the learning of RL policy and metric owing to its independence on RL policy. We theoretically justify its scalability to continuous state and action spaces and design a practical way to incorporate it into an RL procedure as a representation learning target. We evaluate our approach on DeepMind control tasks with default and distracting backgrounds. By statistically reliable evaluation protocols, our experiments demonstrate our approach is superior to previous metric-based methods in terms of sample efficiency and asymptotic performance in both backgrounds.

Hospital evacuation in large-scale disasters using limited aerial transport resources

Hospitals play a key role in providing medical assistance in a post-disaster phase. However, depending on the nature, scale, and severity of disasters, they may also be affected and may need to evacuate their patients. Current evacuation response plans for hospitals are predominantly based on the use of land vehicles, whereas little methodological advancements have been reported for when aerial hospital evacuations become necessary. To address this problem, this study develops a model that facilitates evacuation planning for moving patients from at-risk hospitals to remote locations using limited aerial transportation resources. A subset of the model parameters is treated as stochastic in order to reflect the uncertainties involved in real-world emergency conditions. The complexities of the model are handled via the use of a metaheuristic approach, the Variable neighbourhood search (VNS) algorithm, to provide a suitable solution within a reasonable amount of time. A relaxed mathematical model is used in the proposed VNS to generate an initial solution, which is then transformed into a feasible solution through a try-and-error procedure. The proposed method applies a reinforcement learning procedure that uses different local search strategies within the loop of the VNS. A hypothetical evacuation scenario of real-life scale, from an island state in Australia to the mainland states, is used to evaluate the ability of the model to generate efficient plans. In this case study, Tasmania is considered as an evacuation point, while hospitals located in other major cities host evacuated patients. Comparing the proposed method with a conventional method shows superior performance based on all relevant metrics. Emergency response agencies across many countries that can adopt the proposed methodology, when facing large-scale emergencies caused by wars, epidemics, or natural hazards such as wildfires and earthquakes.

Graph-Attention-Based Casual Discovery With Trust Region-Navigated Clipping Policy Optimization.

In many domains of empirical sciences, discovering the causal structure within variables remains an indispensable task. Recently, to tackle unoriented edges or latent assumptions violation suffered by conventional methods, researchers formulated a reinforcement learning (RL) procedure for causal discovery and equipped a REINFORCE algorithm to search for the best rewarded directed acyclic graph. The two keys to the overall performance of the procedure are the robustness of RL methods and the efficient encoding of variables. However, on the one hand, REINFORCE is prone to local convergence and unstable performance during training. Neither trust region policy optimization, being computationally expensive, nor proximal policy optimization (PPO), suffering from aggregate constraint deviation, is a decent alternative for combinatory optimization problems with considerable individual subactions. We propose a trust region-navigated clipping policy optimization method for causal discovery that guarantees both better search efficiency and steadiness in policy optimization, in comparison with REINFORCE, PPO, and our prioritized sampling-guided REINFORCE implementation. On the other hand, to boost the efficient encoding of variables, we propose a refined graph attention encoder called SDGAT that can grasp more feature information without priori neighborhood information. With these improvements, the proposed method outperforms the former RL method in both synthetic and benchmark datasets in terms of output results and optimization robustness.

Task sequence planning by intelligent agent with context awareness

The problem of context-aware task sequence planning for independent or weakly related tasks by an intelligent agent has been considered. The principle of matching the task to the context is analyzed. The structure of the context-aware task sequence planning module as part of an intelligent agent and the corresponding algorithm are proposed. In the structure of the planning module, three main groups of blocks are implemented: operations with tasks, operations with the context, determining the relevance of tasks to the context. The article also proposes an algorithm for calculating the dynamic priority of a task, an algorithm for determining the relevance of a task to the context, and an algorithm for adapting a set of rules for matching the task to the context. The value of the dynamic priority depends on the static priority of the task, which is assigned when a new task is added, and the degree of correspondence of the task to the context, taking into account the context vector. Two modes of starting the planning algorithm are allowed: when the intelligent agent requests a new task and when the context vector changes. The dynamic priority calculation algorithm is performed independently for each task. As a result, its software implementation has a large parallelization resource. To adapt the set of rules for matching tasks to the context, a scheme of reinforcement learning in a contextual multi-armed bandit was used. For each matching rule, a separate instance of the reinforcement learning procedure is performed. The reinforcement learning method used in the article is Upper-Confidence-Bound Action Selection. The functional scheme of the reinforcement learning procedure, implemented in the prototype of the context-aware task sequence planning module has been proposed. The functional scheme of the reinforcement learning procedure allows the use of data decomposition and functional decomposition to parallelize the corresponding calculations.

Many‐field packet classification with decomposition and reinforcement learning

Scalable packet classification is a key requirement to support scalable network applications such as firewalls, intrusion detection, and differentiated services. With the ever increasing line-rate in core networks, it becomes a great challenge to design a scalable packet classification solution using hand-tuned heuristic approaches. The authors present a scalable learning-based packet classification engine by building an efficient data structure for different rulesets with many fields. This method consists of the decomposition of fields into subsets and building separate decision trees on those subsets using a deep reinforcement learning procedure. To decompose given fields of a ruleset, the authors consider different grouping metrics such as standard deviation of individual fields and introduce a novel metric called diversity index (DI). The authors examine different decomposition schemes and construct decision trees for each scheme using deep reinforcement learning and compare the results. The results show that the SD decomposition metrics results in 11.5% faster than DI metrics, 25% faster than random 2 and 40% faster than random 1. Furthermore, the authors’ learning-based selection method can be applied to varying rulesets due to its ruleset independence.

Monitoring and controlling real-time bus services: a reinforcement learning procedure for eliminating bus bunching

Bus bunching is one of the main issues encountered in bus service operations. The headway (i.e., time between two successive buses) of high-frequency services operating on congested routes can be subject to significant variability with two or more buses arriving at the same bus stop and at the same time (bus bunching). This affects the reliability of bus services and causes user frustration with the travel experience. Although public transport is encouraged as the most environment-friendly mass transit solution, bus bunching is undesirable given that it creates inefficiencies and can push passengers to use private transport.The advances in information communication technologies (ICTs) offer new opportunities for transit operators to limit this effect, ensuring a more reliable service. The paper, taking advantage of the innovations in ICTs, proposes a machine learning-based procedure addressing the issues causing bus bunching. The procedure is applied to a real test case with encouraging results. It opens the possibility that it may be incorporated in a decision support system to assist operators (drivers) in taking corrective actions throughout the day, improving bus service operations.

AMP

Synthesizing graceful and life-like behaviors for physically simulated characters has been a fundamental challenge in computer animation. Data-driven methods that leverage motion tracking are a prominent class of techniques for producing high fidelity motions for a wide range of behaviors. However, the effectiveness of these tracking-based methods often hinges on carefully designed objective functions, and when applied to large and diverse motion datasets, these methods require significant additional machinery to select the appropriate motion for the character to track in a given scenario. In this work, we propose to obviate the need to manually design imitation objectives and mechanisms for motion selection by utilizing a fully automated approach based on adversarial imitation learning. High-level task objectives that the character should perform can be specified by relatively simple reward functions, while the low-level style of the character's behaviors can be specified by a dataset of unstructured motion clips, without any explicit clip selection or sequencing. For example, a character traversing an obstacle course might utilize a task-reward that only considers forward progress, while the dataset contains clips of relevant behaviors such as running, jumping, and rolling. These motion clips are used to train an adversarial motion prior, which specifies style-rewards for training the character through reinforcement learning (RL). The adversarial RL procedure automatically selects which motion to perform, dynamically interpolating and generalizing from the dataset. Our system produces high-quality motions that are comparable to those achieved by state-of-the-art tracking-based techniques, while also being able to easily accommodate large datasets of unstructured motion clips. Composition of disparate skills emerges automatically from the motion prior, without requiring a high-level motion planner or other task-specific annotations of the motion clips. We demonstrate the effectiveness of our framework on a diverse cast of complex simulated characters and a challenging suite of motor control tasks.

Модели и методы обучения с подкреплением в архитектуре адаптивных веб-ориентированных информационных систем

Widespread use of web-based systems in business, marketing, e-learning, etc. makes it necessary to take into account and analyze the information needs of the user in order to optimize interaction with him. One of the main problems of creating adaptive web-based systems is the task of classifying information resources (pages) of the portal describing the offered product or service, for the subsequent formation of the user profile and personalized recommendations of services. Data mining and machine learning methods can be used to solve this problem. The article presents a new approach to creating adaptive web-based information systems using the reinforcement learning algorithms to classify information resources and to form personalized recommendations to users based on their preferences. An adaptive approach is proposed and justified, based on the use of Reinforcement Learning procedures, which allows you to automatically find the most effective strategies for the correct classification of the site's resources and the formation of user groups with the same type of requests and preferences. The proposed scheme allows you to create procedures for evaluating and ranking information resources of the system based on the analysis of user behavior on the site online. The reinforcement learning algorithms used make it possible to evaluate the relevance of each page of the site to the requests and preferences of the users from different categories in order to optimize the structure and content of the site, as well as to build an effective system of recommendations in accordance with the user's interests to be able to choose the most suitable products or services.

Deep reinforcement learning and parameter transfer based approach for the multi-objective agile earth observation satellite scheduling problem

The agile earth observation satellite scheduling problem (AEOSSP) consists of selecting and scheduling a number of tasks from a set of user requests in order to optimize one or multiple criteria. In this paper, we consider a multi-objective version of AEOSSP (called MO-AEOSSP) where the failure rate and the timeliness of scheduled requests are optimized simultaneously. Due to its NP-hardness, traditional iterative problem-tailored heuristic methods are sensitive to problem instances and require massive computational overhead. We thus propose a deep reinforcement learning and parameter transfer based approach (RLPT) to tackle the MO-AEOSSP in a non-iterative manner. RLPT first decomposes the MO-AEOSSP into a number of scalarized sub-problems by a weight sum approach where each sub-problem can be formulated as a Markov Decision Process (MDP). RLPT then applies an encoder–decoder structure neural network (NN) trained by a deep reinforcement learning procedure to producing a high-quality schedule for each sub-problem. The resulting schedules of all scalarized sub-problems form an approximate pareto front for the MO-AEOSSP. Once a NN of a subproblem is trained, RLPT applies a parameter transfer strategy to reducing the training expenses for its neighboring sub-problems. Experimental results on a large set of randomly generated instances show that RLPT outperforms three classical multi-objective evolutionary algorithms (MOEAs) in terms of solution quality, solution distribution and computational efficiency. Results on various-size instances also show that RLPT is highly general and scalable. To the best of our knowledge, this study is the first attempt that applies deep reinforcement learning to a satellite scheduling problem considering multiple objectives.

Agent-based modeling and simulation of the electricity market with residential demand response

Currently, critical peak load caused by residential customers has attracted utility companies and policymakers to pay more attention to residential demand response (RDR) programs. In typical RDR programs, residential customers react to the price or incentive-based signals, but the actions can fall behind flexible market situations. For those residential customers equipped with smart meters, they may contribute more DR loads if they can participate in DR events in a proactive way. In this paper, we propose a comprehensive market framework in which residential customers can provide proactive RDR actions in a day-ahead market (DAM). We model and evaluate the interactions between generation companies (GenCos), retailers, residential customers, and the independent system operator (ISO) via an agent-based modeling and simulation (ABMS) approach. The simulation framework contains two main procedures—the bottom-up modeling procedure and the reinforcement learning (RL) procedure. The bottom-up modeling procedure models the residential load profiles separately by household types to capture the RDR potential differences in advance so that residential customers may rationally provide automatic DR actions. Retailers and GenCos optimize their bidding strategies via the RL procedure. The modified optimization approach in this procedure can prevent the training results from falling into local optimum solutions. The ISO clears the DAM to maximize social welfare via Karush-Kuhn-Tucker (KKT) conditions. Based on realistic residential data in China, the proposed models and methods are verified and compared in a large multi-scenario test case with 30,000 residential households. Results show that proactive RDR programs and interactions between market entities may yield significant benefits for both the supply and demand sides. The models and methods in this paper may be used by utility companies, electricity retailers, market operators, and policymakers to evaluate the consequences of a proactive RDR and the interactions among multi-entities.

Entanglement classification via neural network quantum states

The task of classifying the entanglement properties of a multipartite quantum state poses a remarkable challenge due to the exponentially increasing number of ways in which quantum systems can share quantum correlations. Tackling such challenge requires a combination of sophisticated theoretical and computational techniques. In this paper we combine machine-learning tools and the theory of quantum entanglement to perform entanglement classification for multipartite qubit systems in pure states. We use a parameterisation of quantum systems using artificial neural networks in a restricted Boltzmann machine architecture, known as Neural Network Quantum States, whose entanglement properties can be deduced via a constrained, reinforcement learning procedure. In this way, Separable Neural Network States can be used to build entanglement witnesses for any target state.

An Interactive Perception Method for Warehouse Automation in Smart Cities

The smart city is an integrated environment that heavily relies on intelligent robots, which provides the basis for the warehouse automation. However, a warehouse is a typical unstructured environment, and robotic grasp and manipulation are extremely important for the package, transfer, search, and so on. Currently, the most usual method is to detect the picking or grasping points for some specific end-effector including suction cup, gripper, or robotic hand. The manipulation performance is, therefore, strongly influenced by the visual detector. To tackle this problem, the affordance map has recently been developed. It characterizes the operation possibilities afforded by the operation scene and has been used for several grasp tasks. Nevertheless, the conventional affordance method often fails in complicated environments due to the mistake calculation results. In this article, we develop a novel framework to integrate the interactive exploration with a composite robotic hand for robotic grasping in a complicated environment. The exploration strategy is obtained by a deep reinforcement learning procedure. The developed new composite hand, which integrates the suction cup and grippers, is used to test the merits of the proposed interactive perception method. Experimental results show the proposed method significantly increases the manipulation efficiency and may bring great economic and social and benefits for smart cities.

A Reinforcement-Learning-Based Distributed Resource Selection Algorithm for Massive IoT

Massive IoT including the large number of resource-constrained IoT devices has gained great attention. IoT devices generate enormous traffic, which causes network congestion. To manage network congestion, multi-channel-based algorithms are proposed. However, most of the existing multi-channel algorithms require strict synchronization, an extra overhead for negotiating channel assignment, which poses significant challenges to resource-constrained IoT devices. In this paper, a distributed channel selection algorithm utilizing the tug-of-war (TOW) dynamics is proposed for improving successful frame delivery of the whole network by letting IoT devices always select suitable channels for communication adaptively. The proposed TOW dynamics-based channel selection algorithm has a simple reinforcement learning procedure that only needs to receive the acknowledgment (ACK) frame for the learning procedure, while simply requiring minimal memory and computation capability. Thus, the proposed TOW dynamics-based algorithm can run on resource-constrained IoT devices. We prototype the proposed algorithm on an extremely resource-constrained single-board computer, which hereafter is called the cognitive-IoT prototype. Moreover, the cognitive-IoT prototype is densely deployed in a frequently-changing radio environment for evaluation experiments. The evaluation results show that the cognitive-IoT prototype accurately and adaptively makes decisions to select the suitable channel when the real environment regularly varies. Accordingly, the successful frame ratio of the network is improved.

Reinforcement learning based schemes to manage client activities in large distributed control systems

Large distributed control systems typically can be modeled by a hierarchical structure with two physical layers: console level computers (CLCs) layer and front end computers (FECs) layer. The control system of the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory (BNL) consists of more than 500 FECs, each acting as a server providing services to a large number of clients. Hence the interactions between the server and its clients become crucial to the overall system performance. There are different scenarios of the interactions. For instance, there are cases where the server has a limited processing ability and is queried by a large number of clients. Such cases can put a bottleneck in the system, as heavy traffic can slow down or even crash a system, making it momentarily unresponsive. Also, there are cases where the server has adequate ability to process all the traffic from its clients. We pursue different goals in those cases. For the first case, we would like to manage clients' activities so that their requests are processed by the server as much as possible and the server remains operational. For the second case, we would like to explore an operation point at which the server's resources get utilized efficiently. Moreover, we consider a real-world time constraint to the above case. The time constraint states that clients expect the responses from the server within a time window. In this work, we analyze those cases from a game theory perspective. We model the underlying interactions as a repeated game between clients, which is carried out in discrete time slots. For clients' activity management, we apply a reinforcement learning procedure as a baseline to regulate clients' behaviors. Then we propose a memory scheme to improve its performance. Next, depending on different scenarios, we design corresponding reward functions to stimulate clients in a proper way so that they can learn to optimize different goals. Through extensive simulations, we show that first, the memory structure improves the learning ability of the baseline procedure significantly. Second, by applying appropriate reward functions, clients' activities can be effectively managed to achieve different optimization goals.

Intelligent Parameter Tuning in Optimization-Based Iterative CT Reconstruction via Deep Reinforcement Learning.

A number of image-processing problems can be formulated as optimization problems. The objective function typically contains several terms specifically designed for different purposes. Parameters in front of these terms are used to control the relative importance among them. It is of critical importance to adjust these parameters, as quality of the solution depends on their values. Tuning parameters are a relatively straight forward task for a human, as one can intuitively determine the direction of parameter adjustment based on the solution quality. Yet manual parameter tuning is not only tedious in many cases, but also becomes impractical when a number of parameters exist in a problem. Aiming at solving this problem, this paper proposes an approach that employs deep reinforcement learning to train a system that can automatically adjust parameters in a human-like manner. We demonstrate our idea in an example problem of optimization-based iterative computed tomography (CT) reconstruction with a pixel-wise total-variation regularization term. We set up a parameter-tuning policy network (PTPN), which maps a CT image patch to an output that specifies the direction and amplitude by which the parameter at the patch center is adjusted. We train the PTPN via an end-to-end reinforcement learning procedure. We demonstrate that under the guidance of the trained PTPN, reconstructed CT images attain quality similar or better than those reconstructed with manually tuned parameters.

Iterative Learning Procedure With Reinforcement for High-Accuracy Force Tracking in Robotized Tasks

The paper focuses on industrial interaction robotics tasks, investigating a control approach involving multiples learning levels for training the manipulator to execute a repetitive (partially) changeable task, accurately controlling the interaction. Based on compliance control, the proposed approach consists of two main control levels: 1) iterative friction learning compensation controller with reinforcement and 2) iterative force-tracking learning controller with reinforcement. The learning algorithms rely on the iterative learning and reinforcement learning procedures to automatize the controllers parameters tuning. The proposed procedure has been applied to an automotive industrial assembly task. A standard industrial UR 10 Universal Robot has been used, equipped by a compliant pneumatic gripper and a force/torque sensor at the robot end-effector.

Adaptive simulations, towards interactive protein-ligand modeling

Modeling the dynamic nature of protein-ligand binding with atomistic simulations is one of the main challenges in computational biophysics, with important implications in the drug design process. Although in the past few years hardware and software advances have significantly revamped the use of molecular simulations, we still lack a fast and accurate ab initio description of the binding mechanism in complex systems, available only for up-to-date techniques and requiring several hours or days of heavy computation. Such delay is one of the main limiting factors for a larger penetration of protein dynamics modeling in the pharmaceutical industry. Here we present a game-changing technology, opening up the way for fast reliable simulations of protein dynamics by combining an adaptive reinforcement learning procedure with Monte Carlo sampling in the frame of modern multi-core computational resources. We show remarkable performance in mapping the protein-ligand energy landscape, being able to reproduce the full binding mechanism in less than half an hour, or the active site induced fit in less than 5 minutes. We exemplify our method by studying diverse complex targets, including nuclear hormone receptors and GPCRs, demonstrating the potential of using the new adaptive technique in screening and lead optimization studies.