Combines Reinforcement Learning Research Articles

Learning to Manipulate Tools by Aligning Simulation to Video Demonstration

A seamless integration of robots into human environments requires robots to learn how to use existing human tools. Current approaches for learning tool manipulation skills mostly rely on expert demonstrations provided in the target robot environment, for example, by manually guiding the robot manipulator or by teleoperation. In this work, we introduce an automated approach that replaces an expert demonstration with a Youtube video for learning a tool manipulation strategy. The main contributions are twofold. First, we design an alignment procedure that aligns the simulated environment with the real-world scene observed in the video. This is formulated as an optimization problem that finds a spatial alignment of the tool trajectory to maximize the sparse goal reward given by the environment. Second, we describe an imitation learning approach that focuses on the trajectory of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tool</i> rather than the motion of the human. For this we combine reinforcement learning with an optimization procedure to find a control policy and the placement of the robot based on the tool motion in the aligned environment. We demonstrate the proposed approach on spade, scythe and hammer tools in simulation, and show the effectiveness of the trained policy for the spade on a real Franka Emika Panda robot demonstration.

A Reinforced Hybrid Genetic Algorithm for the Traveling Salesman Problem

We propose a powerful Reinforced Hybrid Genetic Algorithm (RHGA) for the famous NP-hard Traveling Salesman Problem (TSP). RHGA combines reinforcement learning technique with the well-known Edge Assembly Crossover genetic algorithm (EAX-GA) and the Lin-Kernighan-Helsgaun (LKH) local search heuristic. With the help of the proposed hybrid mechanism, the genetic evolution of EAX-GA and the local search of LKH can boost each other's performance. And the reinforcement learning technique based on Q-learning further promotes the hybrid genetic algorithm. Experimental results on 138 well-known and widely used TSP benchmarks, with the number of cities ranging from 1,000 to 85,900, demonstrate the excellent performance of the proposed method.

Adaptive Routing in Wireless Mesh Networks Using Hybrid Reinforcement Learning Algorithm

Wireless mesh networks are popular due to their adaptability, easy-setup, flexibility, cost, and transmission time-reductions. The routing algorithm plays a vital role in transferring the data between the nodes. The network’s performance is significantly impacted by the route opted by the algorithm. The router takes the decision to send the packet to the next router as per the policy of that algorithm. So even though that decision does not favor the right path selection, the router tends to follow its policy. This can be avoided by having intelligent routers that can make routing decisions on the fly.This paper presents the QL-Feed Forward routing algorithm(QFFR), a new generation of routing algorithms that combines reinforcement learning based on the Q-learning algorithm with a Feed Forward neural network. This algorithm(QFFR) can learn from the network environment and make routing decisions based on the algorithm’s learnings. The AI agent’s ability to select the fastest path, which enhances the efficiency of the routing operation, is demonstrated by the working of the suggested QFFR algorithm. This paper also evaluates the performance of traditional algorithms, namely, Ad-hoc On-Demand Distance-Vector, Optimized-Link-State-routing, Destination-Sequenced Distance-Vector and Distance Source routing. The evaluation parameters include throughput, packet delivery ratio, and delay. The parameters are the outcomes of the time the information takes to reach from source to destination. This analysis highlights the improvement in the routing decision ability of a router. As per analysis, Ad hoc On-Demand Distance Vector Algorithm outperforms with throughput 723.13 Kbps, delay 343.73 ns. Q-learning agent identifies the route and reaches the destination in average of 3.7s in non-grid architecture. The Q-learning agent takes 0.49sec with a grid size ten by ten and 0.53sec in three by four grid size. The suggested QFFR takes 7.62s score-over time with stable, consistent performance.

Warrants Generations using a Language Model and a Multi-Agent System

Each argument begins with a conclusion, which is followed by one or more premises supporting the conclusion. The warrant is a critical component of Toulmin's argument model; it explains why the premises support the claim. Despite its critical role in establishing the claim's veracity, it is frequently omitted or left implicit, leaving readers to infer. We consider the problem of producing more diverse and high-quality warrants in response to a claim and evidence. To begin, we employ BART [1] as a conditional sequence tosequence language model to guide the output generation process. On the ARCT dataset [2], we fine-tune the BART model. Second, we propose the Multi-Agent Network for Warrant Generation as a model for producing more diverse and high-quality warrants by combining Reinforcement Learning (RL) and Generative Adversarial Networks (GAN) with the mechanism of mutual awareness of agents. In terms of warrant generation, our model generates a greater variety of warrants than other baseline models. The experimental results validate the effectiveness of our proposed hybrid model for generating warrants.

Actor-Critic Traction Control Based on Reinforcement Learning with Open-Loop Training

The use of actor-critic algorithms can improve the controllers currently implemented in automotive applications. This method combines reinforcement learning (RL) and neural networks to achieve the possibility of controlling nonlinear systems with real-time capabilities. Actor-critic algorithms were already applied with success in different controllers including autonomous driving, antilock braking system (ABS), and electronic stability control (ESC). However, in the current researches, virtual environments are implemented for the training process instead of using real plants to obtain the datasets. This limitation is given by trial and error methods implemented for the training process, which generates considerable risks in case the controller directly acts on the real plant. In this way, the present research proposes and evaluates an open-loop training process, which permits the data acquisition without the control interaction and an open-loop training of the neural networks. The performance of the trained controllers is evaluated by a design of experiments (DOE) to understand how it is affected by the generated dataset. The results present a successful application of open-loop training architecture. The controller can maintain the slip ratio under adequate levels during maneuvers on different floors, including grounds that are not applied during the training process. The actor neural network is also able to identify the different floors and change the acceleration profile according to the characteristics of each ground.

Multi-UAV Assisted Offloading Optimization: A Game Combined Reinforcement Learning Approach

Although unmanned aerial vehicles (UAVs) have attracted much attention by providing aerial relays to massive ground users (GUs) for tasks offloading, there still exist several issues, such as the unbalance of tasks size and trajectory optimization related to energy efficiency and obstacles avoidance. The letter models the multi-UAV assisted offloading system as two separate problems optimized by a potential game combined reinforcement learning algorithm, i.e., potential game for service assignment, and deep deterministic policy gradient (DDPG) for trajectory planning. The former largely reduces the convergence time, and the latter can search the best action in a continuous domain. The numerical results show that the proposed approach has great advantages in minimizing offloading delay, enhancing energy efficiency and avoiding obstacles.

SOLO: Search Online, Learn Offline for Combinatorial Optimization Problems

We study combinatorial problems with real world applications such as machine scheduling, routing, and assignment. We propose a method that combines Reinforcement Learning (RL) and planning. This method can equally be applied to both the offline, as well as online, variants of the combinatorial problem, in which the problem components (e.g., jobs in scheduling problems) are not known in advance, but rather arrive during the decision-making process. Our solution is quite generic, scalable, and leverages distributional knowledge of the problem parameters. We frame the solution process as an MDP, and take a Deep Q-Learning approach wherein states are represented as graphs, thereby allowing our trained policies to deal with arbitrary changes in a principled manner. Though learned policies work well in expectation, small deviations can have substantial negative effects in combinatorial settings. We mitigate these drawbacks by employing our graph-convolutional policies as non-optimal heuristics in a compatible search algorithm, Monte Carlo Tree Search, to significantly improve overall performance. We demonstrate our method on two problems: Machine Scheduling and Capacitated Vehicle Routing. We show that our method outperforms custom-tailored mathematical solvers, state of the art learning-based algorithms, and common heuristics, both in computation time and performance.

An oppositional-Cauchy based GSK evolutionary algorithm with a novel deep ensemble reinforcement learning strategy for COVID-19 diagnosis

A novel coronavirus (COVID-19) has globally attracted attention as a severe respiratory condition. The epidemic has been first tracked in Wuhan, China, and has progressively been expanded in the entire world. The growing expansion of COVID-19 around the globe has made X-ray images crucial for accelerated diagnostics. Therefore, an effective computerized system must be established as a matter of urgency, to facilitate health care professionals in recognizing X-ray images from COVID-19 patients. In this work, we design a novel artificial intelligent-based automated X-ray image analysis framework based on an ensemble of deep optimized convolutional neural networks (CNNs) in order to distinguish coronavirus patients from non-patients. By developing a modified version of gaining–sharing knowledge (GSK) optimization algorithm using the Opposition-based learning (OBL) and Cauchy mutation operators, the architectures of the deployed deep CNNs are optimized automatically without performing the general trial and error procedures. After obtaining the optimized CNNs, it is also very critical to identify how to decrease the number of ensemble deep CNN classifiers to ensure the classification effectiveness. To this end, a selective ensemble approach is proposed for COVID-19 X-ray based image classification using a deep Q network that combines reinforcement learning (RL) with the optimized CNNs. This approach increases the model performance in particular and therefore decreases the ensemble size of classifiers. The experimental results show that the proposed deep RL optimized ensemble approach has an excellent performance over two popular X-ray image based COVID-19 datasets. Our proposed advanced algorithm can accurately identify the COVID-19 patients from the normal individuals with a significant accuracy of 0.991441, precision of 0.993568, recall (sensitivity) of 0.981445, F-measure of 0.989666 and AUC of 0.990337 for Kaggle dataset as well as an excellent accuracy of 0.987742, precision of 0.984334, recall (sensitivity) of 0.989123, F-measure of 0.984939 and AUC of 0.988466 for Mendely dataset.

Probability learning based tabu search for the budgeted maximum coverage problem

The Budgeted Maximum Coverage Problem (BMCP) is a general model with a number of real-world applications. Given n elements with nonnegative profits, m subsets of elements with nonnegative weights and a total budget, the BMCP aims to select some subsets such that the total weight of the selected subsets does not exceed the budget, while the total profit of the associated elements is maximized. BMCP is NP-hard and thus computationally challenging. We investigate for the first time an effective practical algorithm for solving this problem, which combines reinforcement learning and local search. The algorithm iterates through two distinct phases, namely a tabu search phase and a probability learning based perturbation phase. To assess the effectiveness of the proposed algorithm, we show computational results on a set of 30 benchmark instances introduced in this paper and present comparative studies with respect to the approximation algorithm, the genetic algorithm and the CPLEX solver.

Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization

Combinatorial optimization has found applications in numerous fields, from aerospace to transportation planning and economics. The goal is to find an optimal solution among a finite set of possibilities. The well-known challenge one faces with combinatorial optimization is the state-space explosion problem: the number of possibilities grows exponentially with the problem size, which makes solving intractable for large problems. In the last years, deep reinforcement learning (DRL) has shown its promise for designing good heuristics dedicated to solve NP-hard combinatorial optimization problems. However, current approaches have an important shortcoming: they only provide an approximate solution with no systematic ways to improve it or to prove optimality. In another context, constraint programming (CP) is a generic tool to solve combinatorial optimization problems. Based on a complete search procedure, it will always find the optimal solution if we allow an execution time large enough. A critical design choice, that makes CP non-trivial to use in practice, is the branching decision, directing how the search space is explored. In this work, we propose a general and hybrid approach, based on DRL and CP, for solving combinatorial optimization problems. The core of our approach is based on a dynamic programming formulation, that acts as a bridge between both techniques. We experimentally show that our solver is efficient to solve three challenging problems: the traveling salesman problem with time windows, the 4-moments portfolio optimization problem, and the 0-1 knapsack problem. Results obtained show that the framework introduced outperforms the stand-alone RL and CP solutions, while being competitive with industrial solvers.

Combining Reinforcement Learning with Lin-Kernighan-Helsgaun Algorithm for the Traveling Salesman Problem

We address the Traveling Salesman Problem (TSP), a famous NP-hard combinatorial optimization problem. And we propose a variable strategy reinforced approach, denoted as VSR-LKH, which combines three reinforcement learning methods (Q-learning, Sarsa and Monte Carlo) with the well-known TSP algorithm, called Lin-Kernighan-Helsgaun (LKH). VSR-LKH replaces the inflexible traversal operation in LKH, and lets the program learn to make choice at each search step by reinforcement learning. Experimental results on 111 TSP benchmarks from the TSPLIB with up to 85,900 cities demonstrate the excellent performance of the proposed method.

Hybrid algorithms based on combining reinforcement learning and metaheuristic methods to solve global optimization problems

This paper introduces three hybrid algorithms that help in solving global optimization problems using reinforcement learning along with metaheuristic methods. Using the algorithms presented, the search agents try to find a global optimum avoiding the local optima trap. Compared to the classical metaheuristic approaches, the proposed algorithms display higher success in finding new areas as well as exhibiting a more balanced performance while in the exploration and exploitation phases. The algorithms employ reinforcement agents to select an environment based on predefined actions and tasks. A reward and penalty system is used by the agents to discover the environment, done dynamically without following a predetermined model or method. The study makes use of Q-Learning method in all three metaheuristic algorithms, so-called RLI−GWO, RLEx−GWO, and RLWOA algorithms, so as to check and control exploration and exploitation with Q-Table. The Q-Table values guide the search agents of the metaheuristic algorithms to select between the exploration and exploitation phases. A control mechanism is used to get the reward and penalty values for each action. The algorithms presented in this paper are simulated over 30 benchmark functions from CEC 2014, 2015 and the results obtained are compared with well-known metaheuristic and hybrid algorithms (GWO, RLGWO, I-GWO, Ex-GWO, and WOA). The proposed methods have also been applied to the inverse kinematics of the robot arms problem. The results of the used algorithms demonstrate that RLWOA provides better solutions for relevant problems.

Distributed diffusion fusion cooperative spectrum sensing based on reinforcement learning

The openness and complexity of wireless channels make collaborative spectrum sensing vulnerable to malicious users[1]. Therefore, it is very important to identify the attributes of malicious users before collaborative spectrum awareness networks make data fusion decisions. In this paper, a method combining reinforcement learning and cognitive user credit model is proposed, in which the maximum and minimum eigenvalues of signals are used as the initial information for exchange, and the whole sensing network tends to diffuse and fuse nodes with high credit. Finally, the convergence value of the whole network is compared with the decision threshold to complete collaborative spectrum sensing. By combining with consensus fusion network and traditional collaborative sensing algorithm, the proposed method can effectively improve the convergence speed of fusion network and shorten the sensing time on the premise of effectively identifying malicious users[2], so as to improve the spectrum sensing performance and make the collaborative sensing network more adaptive and stable.

Reinforcement Learning for Bit-Flipping Decoding of Polar Codes

A traditional successive cancellation (SC) decoding algorithm produces error propagation in the decoding process. In order to improve the SC decoding performance, it is important to solve the error propagation. In this paper, we propose a new algorithm combining reinforcement learning and SC flip (SCF) decoding of polar codes, which is called a Q-learning-assisted SCF (QLSCF) decoding algorithm. The proposed QLSCF decoding algorithm uses reinforcement learning technology to select candidate bits for the SC flipping decoding. We establish a reinforcement learning model for selecting candidate bits, and the agent selects candidate bits to decode the information sequence. In our scheme, the decoding delay caused by the metric ordering can be removed during the decoding process. Simulation results demonstrate that the decoding delay of the proposed algorithm is reduced compared with the SCF decoding algorithm, based on critical set without loss of performance.

Software-in-the-Loop Combined Reinforcement Learning Method for Dynamic Response Analysis of FOWTs

Floating offshore wind turbines (FOWTs) still face many challenges on how to better predict the dynamic responses. Artificial intelligence (AI) brings a new solution to overcome these challenges with intelligent strategies. A new AI technology-based method, named SADA, is proposed in this paper for the prediction of dynamic responses of FOWTs. Firstly, the methodology of SADA is introduced with the selection of Key Disciplinary Parameters (KDPs). The AI module in SADA was built in a coupled aero-hydro-servo-elastic in-house programDARwindand the policy decision is provided by the machine learning algorithms deep deterministic policy gradient (DDPG). Secondly, a set of basin experimental results of a Hywind Spar-type FOWT were employed to train the AI module. SADA weights KDPs by DDPG algorithms' actor network and changes their values according to the training feedback of 6DOF motions of Hywind platform through comparing theDARwindsimulation results and that of experimental data. Many other dynamic responses that cannot be measured in basin experiment could be predicted in higher accuracy with this intelligentDARwind. Finally, the case study of SADA method was conducted and the results demonstrated that the mean values of the platform's motions can be predicted by AI-basedDARwindwith higher accuracy, for example the maximum error of surge motion is reduced by 21%. This proposed SADA method takes advantage of numerical-experimental method and the machine learning method, which brings a new and promising solution for overcoming the handicap impeding direct use of traditional basin experimental technology in FOWTs design.

A Kind of Reinforcement Learning to Improve Genetic Algorithm for Multiagent Task Scheduling

It is difficult to coordinate the various processes in the process industry. We built a multiagent distributed hierarchical intelligent control model for manufacturing systems integrating multiple production units based on multiagent system technology. The model organically combines multiple intelligent agent modules and physical entities to form an intelligent control system with certain functions. The model consists of system management agent, workshop control agent, and equipment agent. For the task assignment problem with this model, we combine reinforcement learning to improve the genetic algorithm for multiagent task scheduling and use the standard task scheduling dataset in OR-Library for simulation experiment analysis. Experimental results show that the algorithm is superior.

Simultaneous Optimization of Mechanism and Control in Cooperative Work of Multiple Robot Arms

This paper present a simultaneous optimization of the mechanism and control when two six degree of freedom robot arms in the cooperate button pushing task by combining reinforcement learning and genetic algorithm. The proposed optimization system uses a genetic algorithm to optimize the mechanism and reinforcement learning to optimize the control. The length of each link is changed to the length represented by the gene, and the target is to press six buttons in cooperation with two robot arms. The results are evaluated and optimization is performed by repeating crossovers and generational changes. As a result, it was confirmed that not only the control is optimized, but also the mechanism is optimized to achieve efficient and coordinated task achievement.

Adjust Planning Strategies to Accommodate Reinforcement Learning Agents

In agent control issues, the idea of combining reinforcement learning and planning has attracted much attention. Two methods focus on micro and macro action respectively. Their advantages would show together if there is good cooperation between them. An essential for the cooperation is to find an appropriate boundary, assigning different functions to each method. Such a boundary could be represented by parameters in a planning algorithm. In this paper, we create an optimization strategy for planning parameters, through analysis of the connection of reaction and planning; we also create a non-gradient method for accelerating the optimization. The whole algorithm can find a satisfactory setting of planning parameters, making full use of the reaction capability of specific agents.

Using Deep Q-Networks to Train an Agent to Navigate the Unity ML-Agents Banana Environment

Deep Q-learning is the combination of the Q-learning process with a function approximation technique such as a neural network. According to (Zai & Brown 2020), the main idea behind Q-learning is the use of an algorithm to predict a state-action pair, and to then compare the results generated from this prediction to the observed accumulated rewards at some later time. The parameters of the algorithms are then updated so that it makes better predictions next time. While this technique has some advantages that make it very useful for solving reinforcement learning problems, it also falls short for solving complex problems with large state-space. In fact, (Google DeepMind 2015) supported the above conclusion in its seminal paper entitled “Human-level control through deep reinforcement learning”. In this paper, (Mnih et al 2015) asserted that “to use reinforcement learning successfully in situations approaching real-world complexity, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experiences to new situations”. To achieve this objective they stated further, “we developed a novel agent, a deep Q-network (DQN), which is able to combine reinforcement learning with a class of artificial neural network known as deep neural networks”. While Q-learning as a tool for solving reinforcement learning problems has enjoyed some remarkable successes in the past, it was not until the introduction of DQN that practitioners were able to use it to solve large-scale problems. Prior to that, reinforcement learning was limited to “applications and domains in which useful features could be handcrafted, or to domains with fully observed, low-dimensional state spaces”, (Mnih et al 2015) argued further.

Reinforcement Learning Based on Equivalent Consumption Minimization Strategy for Optimal Control of Hybrid Electric Vehicles

Hybrid electric vehicles, operated by engines and motors, require an energy management strategy to achieve competitive fuel economy performance. The equivalent consumption minimization strategy is a well-known algorithm that can be employed for the energy management of hybrid electric vehicles, based on the concept of the equivalent cost of fossil fuels and electric battery energy. However, in the equivalent consumption minimization strategy approach, a parameter called the equivalent factor should be determined to obtain the optimal control policy. In this study, reinforcement learning based approaches are proposed to determine the equivalent factor. First, we show that the equivalent factor can be indirectly extracted from the reinforcement learning results, using the control action from reinforcement learning for the specific driving cycle. In addition, a novel approach that combines reinforcement learning and the equivalent consumption minimization strategy is proposed, where the equivalent factor is determined based on the interaction between the reinforcement learning agent and driving environment, while the control input is decided by the equivalent consumption minimization strategy based on the determined equivalent factor. A model-based reinforcement learning method is used, and the proposed method is validated for vehicle simulation using a parallel hybrid electric vehicle. The simulation results show that the proposed method can achieve a near-optimal solution, which is close to the global solution obtained with the dynamic programming approach (96.7% compared to dynamic programming result in average), and improved performance of 4.3% in average compared with the existing adaptive equivalent consumption minimization strategy.