Combines Reinforcement Learning Research Articles

A Sequential Sampling Approach to the Integration of Habits and Goals

Habits often conflict with goal-directed behaviors and this phenomenon continues to attract interests from neuroscientists, experimental psychologists, and applied health psychologists. Recent computational models explain habit-goal conflicts as the competitions between two learning systems, arbitrated by a central unit. Based on recent research that combined reinforcement learning and sequential sampling, we show that habit-goal conflicts can be more parsimoniously explained by a dynamic integration of habit and goal values in a sequential sampling model, without any arbitration. A computational model was developed by extending the multialternative decision field theory with the assumptions that habits bias starting points of preference accumulation, and that goal importance and goal relevance determine sampling probabilities of goal-related attributes. Simulation studies demonstrated our approach’s ability to qualitatively reproduce important empirical findings from three paradigms – classic devaluation, devaluation with a concurrent schedule, and reversal learning, and to predict gradual changes in decision times. In addition, a parameter recovery exercise using approximate Bayesian computation showcased the possibility of fitting the model to empirical data in future research. Implications of our work for habit theories and applications are discussed.

Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion

In this paper, a novel reinforcement learning method that enables a humanoid robot to learn bipedal walking using a simple reference motion is proposed. Reinforcement learning has recently emerged as a useful method for robots to learn bipedal walking, but, in many studies, a reference motion is necessary for successful learning, and it is laborious or costly to prepare a reference motion. To overcome this problem, our proposed method uses a simple reference motion consisting of three sine waves and automatically sets the waveform parameters using Bayesian optimization. Thus, the reference motion can easily be prepared with minimal human involvement. Moreover, we introduce two means to facilitate reinforcement learning: (1) we combine reinforcement learning with inverse kinematics (IK), and (2) we use the reference motion as a bias for the action determined via reinforcement learning, rather than as an imitation target. Through numerical experiments, we show that our proposed method enables bipedal walking to be learned based on a small number of samples. Furthermore, we conduct a zero-shot sim-to-real transfer experiment using a domain randomization method and demonstrate that a real humanoid robot, KHR-3HV, can walk with the controller acquired using the proposed method.

Enhancing photovoltaic parameter estimation: integration of non-linear hunting and reinforcement learning strategies with golden jackal optimizer.

The advancement of Photovoltaic (PV) systems hinges on the precise optimization of their parameters. Among the numerous optimization techniques, the effectiveness of each often rests on their inherent parameters. This research introduces a new methodology, the Reinforcement Learning-based Golden Jackal Optimizer (RL-GJO). This approach uniquely combines reinforcement learning with the Golden Jackal Optimizer to enhance its efficiency and adaptability in handling various optimization problems. Furthermore, the research incorporates an advanced non-linear hunting strategy to optimize the algorithm's performance. The proposed algorithm is first validated using 29 CEC2017 benchmark test functions and five engineering-constrained design problems. Secondly, rigorous testing on PV parameter estimationbenchmark datasets, including the single-diode model, double-diode model, three-diode model, and a representative PV module, was carried out to highlight the superiority of RL-GJO. The results were compelling: the root mean square error values achieved by RL-GJO were markedly lower than those of the original algorithm and other prevalent optimization methods. The synergy between reinforcement learning and GJO in this approach facilitates faster convergence and improved solution quality. This integration not only improves the performance metrics but also ensures a more efficient optimization process, especially in complex PV scenarios. With an average Freidman's rank test values of 1.564 for numerical and engineering design problems and 1.742 for parameter estimation problems, the proposed RL-GJO is performing better than the original GJO and other peers. The proposed RL-GJO stands out as a reliable tool for PV parameter estimation. By seamlessly combining reinforcement learning with the golden jackal optimizer, it sets a new benchmark in PV optimization, indicating a promising avenue for future research and applications.

TinyML Algorithms for Big Data Management in Large-Scale IoT Systems

In the context of the Internet of Things (IoT), Tiny Machine Learning (TinyML) and Big Data, enhanced by Edge Artificial Intelligence, are essential for effectively managing the extensive data produced by numerous connected devices. Our study introduces a set of TinyML algorithms designed and developed to improve Big Data management in large-scale IoT systems. These algorithms, named TinyCleanEDF, EdgeClusterML, CompressEdgeML, CacheEdgeML, and TinyHybridSenseQ, operate together to enhance data processing, storage, and quality control in IoT networks, utilizing the capabilities of Edge AI. In particular, TinyCleanEDF applies federated learning for Edge-based data cleaning and anomaly detection. EdgeClusterML combines reinforcement learning with self-organizing maps for effective data clustering. CompressEdgeML uses neural networks for adaptive data compression. CacheEdgeML employs predictive analytics for smart data caching, and TinyHybridSenseQ concentrates on data quality evaluation and hybrid storage strategies. Our experimental evaluation of the proposed techniques includes executing all the algorithms in various numbers of Raspberry Pi devices ranging from one to ten. The experimental results are promising as we outperform similar methods across various evaluation metrics. Ultimately, we anticipate that the proposed algorithms offer a comprehensive and efficient approach to managing the complexities of IoT, Big Data, and Edge AI.

A methodology combining reinforcement learning and simulation to optimize the in silico culture of epithelial sheets

Tissue Engineering (TE) and Regenerative Medicine (RM) aim to replicate and replace tissues for curing disease. However, full tissue integration and homeostasis are still far from reach. Biofabrication is an emerging field that identifies the processes required for generating biologically functional products with the desired structural organization and functionality and can potentially revolutionize the regenerative medicine domain, which aims to use patients’ cells to restore the structure and function of damaged tissues and organs. However, biofabrication still has limitations in the quality of processes and products. Biofabrication processes are often improved empirically, but this is slow, costly, and provides partial results. Computational approaches can tap into biofabrication underused potential, supporting analysis, modeling, design, and optimization of biofabrication processes, speeding up their improvement towards a higher quality of products and subsequent higher clinical relevance. This work proposes a reinforcement learning-based computational design space exploration methodology to generate optimal in-silico protocols for the simulated fabrication of epithelial sheets. The optimization strategy relies on a Deep Reinforcement Learning (DRL) algorithm, the Advantage-Actor Critic, which relies on a neural network model for learning. In contrast, simulations rely on the PalaCell2D simulation framework. Validation demonstrates the proposed approach on two protocol generation targets: maximizing the final number of obtained cells and optimizing the spatial organization of the cell aggregate.

Global and Local Awareness: Combine Reinforcement Learning and Model-Based Control for Collision Avoidance

Global and Local Awareness: Combine Reinforcement Learning and Model-Based Control for Collision Avoidance

Combining Reinforcement Learning and In Situ Current Velocity Measurements for Efficient UUV Guidance

Conventional guidance algorithms for unmanned underwater vehicles commonly operate on geometric principles that do not directly take into account local environmental conditions. We describe here a deep reinforcement learning (RL) framework that partially addresses this issue by generating energy-minimizing guidance systems. Specifically, the proposed framework outputs trained policy networks (agents) capable of mapping histories of local current velocity measurements to heading and depth commands that yield nearly-optimal, minimum-energy transits through spatially varying current fields. Numerical experiments in three spatial dimensions on realistic ocean currents, where direction and speed vary non-monotonically in depth, demonstrate that trained agents reduce the energy costs of transits when compared against dead-reckoning, a common in-service guidance heuristic. In particular, numerical results suggest that trained agents learn an explore-exploit strategy according to which they initially execute an exploratory dive, followed by a return to a depth with optimally favorable currents, which they exploit for the remainder of the transit.

Mapless mobile robot navigation at the edge using self-supervised cognitive map learners.

Navigation of mobile agents in unknown, unmapped environments is a critical task for achieving general autonomy. Recent advancements in combining Reinforcement Learning with Deep Neural Networks have shown promising results in addressing this challenge. However, the inherent complexity of these approaches, characterized by multi-layer networks and intricate reward objectives, limits their autonomy, increases memory footprint, and complicates adaptation to energy-efficient edge hardware. To overcome these challenges, we propose a brain-inspired method that employs a shallow architecture trained by a local learning rule for self-supervised navigation in uncharted environments. Our approach achieves performance comparable to a state-of-the-art Deep Q Network (DQN) method with respect to goal-reaching accuracy and path length, with a similar (slightly lower) number of parameters, operations, and training iterations. Notably, our self-supervised approach combines novelty-based and random walks to alleviate the need for objective reward definition and enhance agent autonomy. At the same time, the shallow architecture and local learning rule do not call for error backpropagation, decreasing the memory overhead and enabling implementation on edge neuromorphic processors. These results contribute to the potential of embodied neuromorphic agents utilizing minimal resources while effectively handling variability.

Combining Reinforcement Learning With Genetic Algorithm for Many-To-Many Route Optimization of Autonomous Vehicles

Combining Reinforcement Learning With Genetic Algorithm for Many-To-Many Route Optimization of Autonomous Vehicles

Deep learning combined with the teaching practice path of computer software application courses under the perspective of “five-education integration”

Abstract As a core course of educational technology, the teaching of application software in colleges and universities faces the problem of insufficient practical ability of learners. This paper constructs a suitable path of computer software application teaching with reinforcement learning by combining reinforcement learning algorithm with students’ cognitive diagnosis theory. Students’ practical level is diagnosed through students’ actual operation results combined with the factors of guessing and mistakes, and appropriate teaching practice paths are recommended based on students’ practical ability combined with Markov decision-making to model the environment of computer software application practice. The TOVE method was used to evaluate the computer application practice course, and the practice paths with individualization were reasoned based on the connection between students’ knowledge points. The results show that among the experimental results of different practice paths, the effectiveness of the deep learning practice path of S1 is the highest at 85%. Compared with the expert-recommended practice path, the deviation value of the path effectiveness of S5 is 1, which indicates that the deep learning practice path is similar to the expert path likewise with a certain degree of specialization. On the Analysis of classroom project practice, the deviation value of the control class is 0.5512, and the deviation value of the experimental class is 4.2662, which is significantly larger than that of the control class, indicating that there is a significant improvement in students’ practical ability under the deep learning teaching practice path.

Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks

Heat integration is important for energy saving in the process industry. It is linked to the persistently challenging task of optimal design of heat exchanger networks (HEN). Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem, it is not easy to find solutions of high quality for large scale problems. Reinforcement learning (RL) method which learns strategies through ongoing exploration and exploitation, reveal advantages in such area. However, due to the complexity of the HEN design problem, the RL method for HEN should be dedicated and designed. A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods. An insightful state representation of HEN structure as well as a customized reward function are introduced. A Q-learning algorithm is applied to update the HEN structure using the ϵ-greedy strategy. Better results are obtained from three literature cases of different scales.

Q-LEARNING, POLICY ITERATION AND ACTOR-CRITIC REINFORCEMENT LEARNING COMBINED WITH METAHEURISTIC ALGORITHMS IN SERVO SYSTEM CONTROL

This paper carries out the performance analysis of three control system structures and approaches, which combine Reinforcement Learning (RL) and Metaheuristic Algorithms (MAs) as representative optimization algorithms. In the first approach, the Gravitational Search Algorithm (GSA) is employed to initialize the parameters (weights and biases) of the Neural Networks (NNs) involved in Deep Q-Learning by replacing the traditional way of initializing the NNs based on random generated values. In the second approach, the Grey Wolf Optimizer (GWO) algorithm is employed to train the policy NN in Policy Iteration RL-based control. In the third approach, the GWO algorithm is employed as a critic in an Actor-Critic framework, and used to evaluate the performance of the actor NN. The goal of this paper is to analyze all three RL-based control approaches, aiming to determine which one represents the best fit for solving the proposed control optimization problem. The performance analysis is based on non-parametric statistical tests conducted on the data obtained from real-time experimental results specific to nonlinear servo system position control.

Active compliance control of robot peg-in-hole assembly based on combined reinforcement learning

Active compliance control of robot peg-in-hole assembly based on combined reinforcement learning

Construction of rehabilitation training simulation model based on energy monitoring and transmission by wireless sensor networks

In today's world, with the advent of information technology, the speed of social development has also been greatly improved, and the national demand is also increasing. Among them, national health issues have always been the focus of countries. In addition to basic medical testing, the rehabilitation training for special groups in the medical industry is also gradually becoming more mature. In traditional wireless sensor networks, the energy consumption of nodes is often high, because each node needs to continuously send and receive data, leading to energy waste. In addition, many nodes remain active even when data transmission is not required, further increasing energy consumption. The coverage range of traditional wireless sensor networks is influenced by the number and distribution of nodes, often resulting in coverage vulnerabilities and blind spots. This makes it impossible for the network to fully cover the target area, reducing the reliability and accuracy of monitoring. Based on the above situation, this paper studies the construction of rehabilitation training simulation model by using wireless sensor network monitoring coverage and transmission technology on the basis of energy acquisition. Firstly, it discusses the development trend of rehabilitation training technology in various countries and the social status of rehabilitation training. Through the use of wireless sensor network monitors to monitor the target data collection and model construction. The target coverage and transmission speed are optimized by combining reinforcement learning algorithm SARSA and multilayer feedforward BP neural network. The sensor coverage and transmission of monitoring targets are also studied. The results show that the optimized wireless sensor system has a good effect in rehabilitation training.

Resource Allocation Method of Edge IoT Agent Based on Deep Reinforcement Learning

Reasonable allocation of resources is an important guarantee for efficient support of power business in edge IoT agents. Facing the above problems of the current power Internet of Things, this paper proposes a resource optimization allocation method based on deep Q-learning. This method first comprehensively considers the communication performance and network security. Involving indicators such as latency and service satisfaction, a complete and reliable mathematical model of the edge Internet of Things proxy network is constructed to achieve efficient and reliable modeling of the power Internet of Things (pIoT), aiming to better fit the practical interaction needs for efficient and secure communication. The Q-learning network model is optimized, and the method combining Reinforcement learning and deep learning is used to solve the model. Used by this network, the optimization and improvement of the deep network model is realized, so that the status, action and other parameters of the network model can be solved in a timely manner, so as to better support the reliable and efficient information interaction of the communication network. The test results prove that the delay of the proposed method can be maintained within 12[Formula: see text]ms in more complex scenarios, and the interaction success rate reaches 0.975, confirming that the proposed method can provide good information interaction guarantee services.

EarlGAN: An enhanced actor–critic reinforcement learning agent-driven GAN for de novo drug design

Deep generative models, such as Generative Adversarial Networks (GANs), have attracted the attention of researchers in the denovo drug design field. However, traditional GANs are typically used for image processing. Therefore, they are unsuitable for Simplified Molecular-Input Line-Entry System (SMILES) strings due to their discrete nature. Previous studies addressed this problem by combining Reinforcement Learning (RL) approaches with Monte Carlo tree search. However, for large chemical datasets, the molecule generation process is time-consuming due to the lengthy atom-by-atom sampling process with cumulative reward, an essence of the Monte Carlo tree search-based RL approaches. To address this problem, we propose an enhanced actor–critic RL agent-driven GAN, called EarlGAN, for denovo drug design. Specifically, EarlGAN’s generator acts as an actor to generate SMILES strings, and the discriminator acts as a critic to perform discrimination. EarlGAN makes autoregressive predictions at the atomic level. While the generator is based on previously generated atoms, the discriminator discriminates using a bidirectional pass over the atoms, including the current atom that is being predicted. We integrate moment, global-level discrimination rewards, and information entropy maximization. The moment rewards reduce the computation time, and the global-level rewards ensure the consistency of the molecule, whereas the information entropy maximization leads to a more diverse sample generation. Experiments and ablation studies verify the effectiveness of EarlGAN for denovo drug design on the QM9 and ZINC datasets. In addition, the visualization analysis provides insight into EarlGAN and supports our conclusion.

Sample-efficient multi-agent reinforcement learning with masked reconstruction.

Deep reinforcement learning (DRL) is a powerful approach that combines reinforcement learning (RL) and deep learning to address complex decision-making problems in high-dimensional environments. Although DRL has been remarkably successful, its low sample efficiency necessitates extensive training times and large amounts of data to learn optimal policies. These limitations are more pronounced in the context of multi-agent reinforcement learning (MARL). To address these limitations, various studies have been conducted to improve DRL. In this study, we propose an approach that combines a masked reconstruction task with QMIX (M-QMIX). By introducing a masked reconstruction task as an auxiliary task, we aim to achieve enhanced sample efficiency-a fundamental limitation of RL in multi-agent systems. Experiments were conducted using the StarCraft II micromanagement benchmark to validate the effectiveness of the proposed method. We used 11 scenarios comprising five easy, three hard, and three very hard scenarios. We particularly focused on using a limited number of time steps for each scenario to demonstrate the improved sample efficiency. Compared to QMIX, the proposed method is superior in eight of the 11 scenarios. These results provide strong evidence that the proposed method is more sample-efficient than QMIX, demonstrating that it effectively addresses the limitations of DRL in multi-agent systems.

Combining reinforcement learning method to enhance LEDBAT++ over diversified network environments

LEDBAT++ is a novel less-than-best-effort congestion control algorithm. However, it still shows aggressiveness when competing with CUBIC in shallow buffer networks or with BBRv2 in low latency networks. In order to maintain its low-priority performance over diversified network environments, we propose PeaceKeeper, which combines reinforcement learning algorithm to dynamically adjust the target based on the network state. Extensive simulations show that, compared to LEDBAT++, the throughput of the primary flow competing with PeaceKeeper improved by 30.76% to 173.63%. Additionally, compared to heuristics adjusting the target, PeaceKeeper increases the link bandwidth utilization by 50.91%.

Hybrid Optimized Verification Methodology using Deep Reinforcement Neural Network

Universal Verification Methodology (UVM) caters to an essential role in verifying the different categories of circuits ranging from small-scale chips to complex system-on-chip architectures. Constrained random simulations are an indispensable part of UVM and are often used for design verification. However, the effort and time spent manually updating and analyzing the design input constraints result in high time complexity, which typically impacts the coverage goal and fault verification ratio. To overcome this problem, this paper proposes a novel hybrid optimized verification framework that combines Reinforcement Learning (RL) and Deep Neural Networks (DNN) for automatically optimizing the input constraints, accelerating faster verification with a high coverage ratio. The proposed algorithm uses reinforcement learning to generate all possible vector sequences needed for testing the target devices and corresponding outputs of the target devices and potential design errors. Furthermore, the framework intends to use high-speed deep-feedforward neural networks to automate and optimize the constraints during runtime. The proposed framework was developed using Python interfaced with the TCL environment. Extensive experimentation was carried out using several circuits, including multi-core designs, and performance parameters such as coverage accuracy, speed, and computational complexity were calculated and analyzed. The experiment demonstrated the proposed framework remarkable results, showing its superior performance in faster coverage and fewer misclassification errors. Furthermore, the proposed framework is compared with existing verification frameworks and other classical learning models. Good results demonstrate that the proposed framework increases the 4.5x speed for verifying multi-core designs and the 99% accuracy of detection and coverage.

Research on the Application of Prompt Learning Pretrained Language Model in Machine Translation Task with Reinforcement Learning

With the continuous advancement of deep learning technology, pretrained language models have emerged as crucial tools for natural language processing tasks. However, optimization of pretrained language models is essential for specific tasks such as machine translation. This paper presents a novel approach that integrates reinforcement learning with prompt learning to enhance the performance of pretrained language models in machine translation tasks. In our methodology, a “prompt” string is incorporated into the input of the pretrained language model, to guide the generation of an output that aligns closely with the target translation. Reinforcement learning is employed to train the model in producing optimal translation results. During this training process, the target translation is utilized as a reward signal to incentivize the model to generate an output that aligns more closely with the desired translation. Experimental results validated the effectiveness of the proposed approach. The pretrained language model trained with prompt learning and reinforcement learning exhibited superior performance compared to traditional pretrained language models in machine translation tasks. Furthermore, we observed that different prompt strategies significantly impacted the model’s performance, underscoring the importance of selecting an optimal prompt strategy tailored to the specific task. The results suggest that using techniques such as prompt learning and reinforcement learning can improve the performance of pretrained language models for tasks such as text generation and machine translation. The method proposed in this paper not only offers a fresh perspective on leveraging pretrained language models in machine translation and other related tasks but also serves as a valuable reference for further research in this domain. By combining reinforcement learning with prompt learning, researchers can explore new avenues for optimizing pretrained language models and improving their efficacy in various natural language processing tasks.