Combines Reinforcement Learning Research Articles

Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations

Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations

Reinforcement learning based blockchain model for revoking unauthorized access in Virtualized Network Functions‐based Internet of Things Mobile Edge Computing

AbstractVNFs boost data processing efficiency in Mobile Edge Computing (MEC)‐driven Internet of Things (IoT) for healthcare, smart cities, and industrial automation. VNF‐based IoT MEC systems encounter a significant security threat due to unauthorized access, posing risks to data privacy and system integrity. Existing approaches struggle to adapt to dynamic environments and lack tamper‐proof enforcement mechanisms. In this work, we propose a novel system combining Reinforcement Learning (RL) and blockchain technology to revoke unauthorized access in VNF‐based IoT MEC. We introduce the Integrated Action‐selection DRL Algorithm for Unauthorized Access Revocation (IASDRL‐UAR), a novel RL approach that excels in dynamic environments by handling both continuous and discrete actions, enabling real‐time optimization of security risk, execution time, and energy consumption. A behavior control contract (BCC) is proposed and integrated into the RL system, automating behavior checks and enforcement, streamlining security management, and reducing manual intervention. RL feedback plays a pivotal role in steering dynamic security adjustments, gaining valuable perspectives from user behavior via trust scores in the behavior contract. The security features of the proposed method are analyzed. Performance comparisons reveal a substantial improvement, with the proposed system outperforming existing methods by 30% in terms of throughput, 21.7% in system stability, and 26% in access revocation latency. Additionally, the system demonstrates a higher security index, energy efficiency, and scalability.

Reinforcement learning for addressing the cold-user problem in recommender systems

Recommender systems are widely used in webshops because of their ability to provide users with personalized recommendations. However, the cold-user problem (i.e., recommending items to new users) is an important issue many webshops face. With the recent General Data Protection Regulation in Europe, the use of additional user information such as demographics is not possible without the user’s explicit consent. Several techniques have been proposed to solve the cold-user problem. Many of these techniques utilize Active Learning (AL) methods, which let cold users rate items to provide better recommendations for them. In this research, we propose two novel approaches that combine reinforcement learning with AL to elicit the users’ preferences and provide them with personalized recommendations. We compare reinforcement learning approaches that are either AL-based or item-based, where the latter predicts users’ ratings of an item by using their ratings of similar items. Differently than many of the existing approaches, this comparison is made based on implicit user information. Using a large real-world dataset, we show that the item-based strategy is more accurate than the AL-based strategy as well as several existing AL strategies.

Model predictive control combined reinforcement learning for automatic vehicles applied in intelligent transportation system

Model predictive control combined reinforcement learning for automatic vehicles applied in intelligent transportation system

Optimization of high-speed fixed-wing UAV penetration strategy based on deep reinforcement learning

The penetration decision of unmanned aerial vehicles (UAVs) is one of the key components for UAVs to detect or strike crucial targets in adversarial environments. Presently, most methods lack research on penetration decisions under unknown interception information. The executing UAV for decision-making is generally a simplified three-degree-of-freedom model. The decision format is often simplified to path point optimization, lacking control over the stability of complex model attitudes and exhibiting relatively poor robustness. This paper proposes a penetration relative motion theory and aircraft control method based on the six-degrees-of-freedom UAV model to reduce response errors between algorithm control decisions and actual flight control. Using a Markov decision process, a UAV penetration decision control method based on an attitude-overload control loop is designed. This method combines reinforcement learning techniques to enable autonomous penetration decision-making and control for UAVs facing active defense scenarios, enhancing the autonomy, accuracy, and generalization of UAV online decision-making. The paper concludes by implementing autonomous decision-making and control for UAV penetration through the integration of interceptor trajectory prediction with proximal policy optimization. This method reduces the dependency of penetration strategies on interceptor information, ensuring both the generalization ability of penetration strategy models under uncertain information and the consistency of decision stability. Through simulations and experiments, we verified that the penetration strategy based on the pre-sampling PPO method improves the penetration effect by 15.44 % on average and the hit rate by 18.48 %. Moreover, the probability of penetration in different scenarios exceeds 70 %, and the penetration effect is improved by more than 10 % in the case of multiple interceptors. This paper also analyzes the impact of interceptor spatial distribution and the quantity of interceptors on the effectiveness of UAV penetration decision-making.

Decoding topological XYZ$^{2}$ codes with reinforcement learning based on attention mechanisms

Abstract Quantum error correction, a technique that relies on the principle of redundancy to encode logical information into additional qubits to better protect the system from noise, is necessary to design a viable quantum computer. For this new topological stabilizer code-XYZ$^{2}$ code defined on the cellular lattice in the paper, it is implemented on a hexagonal lattice of qubits and encodes the logical qubits with the help of stabilizer measurements of weight six and weight two. However topological stabilizer codes in cellular lattice quantum systems suffer from the detrimental effects of noise due to interaction with the environment. Several decoding approaches have been proposed to address this problem.In this paper, we propose the use of a state-attention based reinforcement learning decoder to decode XYZ$^{2}$ codes, which enables the decoder to more accurately focus on the information related to the current decoding position, and the error correction accuracy of our reinforcement learning decoder model under the optimisation conditions, which can reach 83.27% under the depolarising noise model, and we have measured thresholds of 0.18856 and 0.19043 for XYZ$^{2}$ codes at code spacing of 3 to 7 and 7 to 11, respectively. our study provides directions and ideas for the application of decoding schemes combining reinforcement learning attention mechanisms to other topological quantum error-correcting codes.

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

This paper proposes an innovative method for achieving optimal tracking control in nonlinear continuous-time systems with input constraints. The method combines reinforcement learning and hybrid metaheuristics to enhance the controller’s performance. Specifically, a hybrid metaheuristic algorithm is employed to optimize the hyperparameters of a critic neural network, which serves as the system’s controller. The proposed approach is evaluated through extensive simulation studies on a nonlinear system with input constraints. Results demonstrate its superiority over traditional control techniques in terms of accuracy, timeliness, and stability. Notably, the approach effectively eliminates overshoot and steady-state error while providing precise and prompt tracking and showcasing remarkable robustness against model uncertainties. By synergistically integrating reinforcement learning and hybrid metaheuristics, this approach represents a significant advancement in enhancing the control performance of complex nonlinear systems. The simulation studies confirm superiority of the proposed approach over existing techniques, offering a promising solution for achieving optimal tracking control in nonlinear systems with input constraints. This approach holds potential for a wide range of applications, including robotics, aerospace, and manufacturing, where precise and prompt tracking control is critical.

A Sequential Sampling Approach to the Integration of Habits and Goals

AbstractHabits often conflict with goal-directed behaviors and this phenomenon continues to attract interests from neuroscientists, experimental psychologists, and applied health psychologists. Recent computational models explain habit-goal conflicts as the competitions between two learning systems, arbitrated by a central unit. Based on recent research that combined reinforcement learning and sequential sampling, we show that habit-goal conflicts can be more parsimoniously explained by a dynamic integration of habit and goal values in a sequential sampling model, without any arbitration. A computational model was developed by extending the multialternative decision field theory with the assumptions that habits bias starting points of preference accumulation, and that goal importance and goal relevance determine sampling probabilities of goal-related attributes. Simulation studies demonstrated our approach’s ability to qualitatively reproduce important empirical findings from three paradigms – classic devaluation, devaluation with a concurrent schedule, and reversal learning, and to predict gradual changes in decision times. In addition, a parameter recovery exercise using approximate Bayesian computation showcased the possibility of fitting the model to empirical data in future research. Implications of our work for habit theories and applications are discussed.

Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion

In this paper, a novel reinforcement learning method that enables a humanoid robot to learn bipedal walking using a simple reference motion is proposed. Reinforcement learning has recently emerged as a useful method for robots to learn bipedal walking, but, in many studies, a reference motion is necessary for successful learning, and it is laborious or costly to prepare a reference motion. To overcome this problem, our proposed method uses a simple reference motion consisting of three sine waves and automatically sets the waveform parameters using Bayesian optimization. Thus, the reference motion can easily be prepared with minimal human involvement. Moreover, we introduce two means to facilitate reinforcement learning: (1) we combine reinforcement learning with inverse kinematics (IK), and (2) we use the reference motion as a bias for the action determined via reinforcement learning, rather than as an imitation target. Through numerical experiments, we show that our proposed method enables bipedal walking to be learned based on a small number of samples. Furthermore, we conduct a zero-shot sim-to-real transfer experiment using a domain randomization method and demonstrate that a real humanoid robot, KHR-3HV, can walk with the controller acquired using the proposed method.

Enhancing photovoltaic parameter estimation: integration of non-linear hunting and reinforcement learning strategies with golden jackal optimizer.

The advancement of Photovoltaic (PV) systems hinges on the precise optimization of their parameters. Among the numerous optimization techniques, the effectiveness of each often rests on their inherent parameters. This research introduces a new methodology, the Reinforcement Learning-based Golden Jackal Optimizer (RL-GJO). This approach uniquely combines reinforcement learning with the Golden Jackal Optimizer to enhance its efficiency and adaptability in handling various optimization problems. Furthermore, the research incorporates an advanced non-linear hunting strategy to optimize the algorithm's performance. The proposed algorithm is first validated using 29 CEC2017 benchmark test functions and five engineering-constrained design problems. Secondly, rigorous testing on PV parameter estimationbenchmark datasets, including the single-diode model, double-diode model, three-diode model, and a representative PV module, was carried out to highlight the superiority of RL-GJO. The results were compelling: the root mean square error values achieved by RL-GJO were markedly lower than those of the original algorithm and other prevalent optimization methods. The synergy between reinforcement learning and GJO in this approach facilitates faster convergence and improved solution quality. This integration not only improves the performance metrics but also ensures a more efficient optimization process, especially in complex PV scenarios. With an average Freidman's rank test values of 1.564 for numerical and engineering design problems and 1.742 for parameter estimation problems, the proposed RL-GJO is performing better than the original GJO and other peers. The proposed RL-GJO stands out as a reliable tool for PV parameter estimation. By seamlessly combining reinforcement learning with the golden jackal optimizer, it sets a new benchmark in PV optimization, indicating a promising avenue for future research and applications.

TinyML Algorithms for Big Data Management in Large-Scale IoT Systems

In the context of the Internet of Things (IoT), Tiny Machine Learning (TinyML) and Big Data, enhanced by Edge Artificial Intelligence, are essential for effectively managing the extensive data produced by numerous connected devices. Our study introduces a set of TinyML algorithms designed and developed to improve Big Data management in large-scale IoT systems. These algorithms, named TinyCleanEDF, EdgeClusterML, CompressEdgeML, CacheEdgeML, and TinyHybridSenseQ, operate together to enhance data processing, storage, and quality control in IoT networks, utilizing the capabilities of Edge AI. In particular, TinyCleanEDF applies federated learning for Edge-based data cleaning and anomaly detection. EdgeClusterML combines reinforcement learning with self-organizing maps for effective data clustering. CompressEdgeML uses neural networks for adaptive data compression. CacheEdgeML employs predictive analytics for smart data caching, and TinyHybridSenseQ concentrates on data quality evaluation and hybrid storage strategies. Our experimental evaluation of the proposed techniques includes executing all the algorithms in various numbers of Raspberry Pi devices ranging from one to ten. The experimental results are promising as we outperform similar methods across various evaluation metrics. Ultimately, we anticipate that the proposed algorithms offer a comprehensive and efficient approach to managing the complexities of IoT, Big Data, and Edge AI.

A methodology combining reinforcement learning and simulation to optimize the in silico culture of epithelial sheets

Tissue Engineering (TE) and Regenerative Medicine (RM) aim to replicate and replace tissues for curing disease. However, full tissue integration and homeostasis are still far from reach. Biofabrication is an emerging field that identifies the processes required for generating biologically functional products with the desired structural organization and functionality and can potentially revolutionize the regenerative medicine domain, which aims to use patients’ cells to restore the structure and function of damaged tissues and organs. However, biofabrication still has limitations in the quality of processes and products. Biofabrication processes are often improved empirically, but this is slow, costly, and provides partial results. Computational approaches can tap into biofabrication underused potential, supporting analysis, modeling, design, and optimization of biofabrication processes, speeding up their improvement towards a higher quality of products and subsequent higher clinical relevance. This work proposes a reinforcement learning-based computational design space exploration methodology to generate optimal in-silico protocols for the simulated fabrication of epithelial sheets. The optimization strategy relies on a Deep Reinforcement Learning (DRL) algorithm, the Advantage-Actor Critic, which relies on a neural network model for learning. In contrast, simulations rely on the PalaCell2D simulation framework. Validation demonstrates the proposed approach on two protocol generation targets: maximizing the final number of obtained cells and optimizing the spatial organization of the cell aggregate.

Combining Reinforcement Learning With Genetic Algorithm for Many-To-Many Route Optimization of Autonomous Vehicles

Combining Reinforcement Learning With Genetic Algorithm for Many-To-Many Route Optimization of Autonomous Vehicles

Deep learning combined with the teaching practice path of computer software application courses under the perspective of “five-education integration”

Abstract As a core course of educational technology, the teaching of application software in colleges and universities faces the problem of insufficient practical ability of learners. This paper constructs a suitable path of computer software application teaching with reinforcement learning by combining reinforcement learning algorithm with students’ cognitive diagnosis theory. Students’ practical level is diagnosed through students’ actual operation results combined with the factors of guessing and mistakes, and appropriate teaching practice paths are recommended based on students’ practical ability combined with Markov decision-making to model the environment of computer software application practice. The TOVE method was used to evaluate the computer application practice course, and the practice paths with individualization were reasoned based on the connection between students’ knowledge points. The results show that among the experimental results of different practice paths, the effectiveness of the deep learning practice path of S1 is the highest at 85%. Compared with the expert-recommended practice path, the deviation value of the path effectiveness of S5 is 1, which indicates that the deep learning practice path is similar to the expert path likewise with a certain degree of specialization. On the Analysis of classroom project practice, the deviation value of the control class is 0.5512, and the deviation value of the experimental class is 4.2662, which is significantly larger than that of the control class, indicating that there is a significant improvement in students’ practical ability under the deep learning teaching practice path.

Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks

Heat integration is important for energy saving in the process industry. It is linked to the persistently challenging task of optimal design of heat exchanger networks (HEN). Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem, it is not easy to find solutions of high quality for large scale problems. Reinforcement learning (RL) method which learns strategies through ongoing exploration and exploitation, reveal advantages in such area. However, due to the complexity of the HEN design problem, the RL method for HEN should be dedicated and designed. A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods. An insightful state representation of HEN structure as well as a customized reward function are introduced. A Q-learning algorithm is applied to update the HEN structure using the ϵ-greedy strategy. Better results are obtained from three literature cases of different scales.

Q-LEARNING, POLICY ITERATION AND ACTOR-CRITIC REINFORCEMENT LEARNING COMBINED WITH METAHEURISTIC ALGORITHMS IN SERVO SYSTEM CONTROL

This paper carries out the performance analysis of three control system structures and approaches, which combine Reinforcement Learning (RL) and Metaheuristic Algorithms (MAs) as representative optimization algorithms. In the first approach, the Gravitational Search Algorithm (GSA) is employed to initialize the parameters (weights and biases) of the Neural Networks (NNs) involved in Deep Q-Learning by replacing the traditional way of initializing the NNs based on random generated values. In the second approach, the Grey Wolf Optimizer (GWO) algorithm is employed to train the policy NN in Policy Iteration RL-based control. In the third approach, the GWO algorithm is employed as a critic in an Actor-Critic framework, and used to evaluate the performance of the actor NN. The goal of this paper is to analyze all three RL-based control approaches, aiming to determine which one represents the best fit for solving the proposed control optimization problem. The performance analysis is based on non-parametric statistical tests conducted on the data obtained from real-time experimental results specific to nonlinear servo system position control.

Active compliance control of robot peg-in-hole assembly based on combined reinforcement learning

Active compliance control of robot peg-in-hole assembly based on combined reinforcement learning

Construction of rehabilitation training simulation model based on energy monitoring and transmission by wireless sensor networks

In today's world, with the advent of information technology, the speed of social development has also been greatly improved, and the national demand is also increasing. Among them, national health issues have always been the focus of countries. In addition to basic medical testing, the rehabilitation training for special groups in the medical industry is also gradually becoming more mature. In traditional wireless sensor networks, the energy consumption of nodes is often high, because each node needs to continuously send and receive data, leading to energy waste. In addition, many nodes remain active even when data transmission is not required, further increasing energy consumption. The coverage range of traditional wireless sensor networks is influenced by the number and distribution of nodes, often resulting in coverage vulnerabilities and blind spots. This makes it impossible for the network to fully cover the target area, reducing the reliability and accuracy of monitoring. Based on the above situation, this paper studies the construction of rehabilitation training simulation model by using wireless sensor network monitoring coverage and transmission technology on the basis of energy acquisition. Firstly, it discusses the development trend of rehabilitation training technology in various countries and the social status of rehabilitation training. Through the use of wireless sensor network monitors to monitor the target data collection and model construction. The target coverage and transmission speed are optimized by combining reinforcement learning algorithm SARSA and multilayer feedforward BP neural network. The sensor coverage and transmission of monitoring targets are also studied. The results show that the optimized wireless sensor system has a good effect in rehabilitation training.

Resource Allocation Method of Edge IoT Agent Based on Deep Reinforcement Learning

Reasonable allocation of resources is an important guarantee for efficient support of power business in edge IoT agents. Facing the above problems of the current power Internet of Things, this paper proposes a resource optimization allocation method based on deep Q-learning. This method first comprehensively considers the communication performance and network security. Involving indicators such as latency and service satisfaction, a complete and reliable mathematical model of the edge Internet of Things proxy network is constructed to achieve efficient and reliable modeling of the power Internet of Things (pIoT), aiming to better fit the practical interaction needs for efficient and secure communication. The Q-learning network model is optimized, and the method combining Reinforcement learning and deep learning is used to solve the model. Used by this network, the optimization and improvement of the deep network model is realized, so that the status, action and other parameters of the network model can be solved in a timely manner, so as to better support the reliable and efficient information interaction of the communication network. The test results prove that the delay of the proposed method can be maintained within 12[Formula: see text]ms in more complex scenarios, and the interaction success rate reaches 0.975, confirming that the proposed method can provide good information interaction guarantee services.

EarlGAN: An enhanced actor–critic reinforcement learning agent-driven GAN for de novo drug design

Deep generative models, such as Generative Adversarial Networks (GANs), have attracted the attention of researchers in the denovo drug design field. However, traditional GANs are typically used for image processing. Therefore, they are unsuitable for Simplified Molecular-Input Line-Entry System (SMILES) strings due to their discrete nature. Previous studies addressed this problem by combining Reinforcement Learning (RL) approaches with Monte Carlo tree search. However, for large chemical datasets, the molecule generation process is time-consuming due to the lengthy atom-by-atom sampling process with cumulative reward, an essence of the Monte Carlo tree search-based RL approaches. To address this problem, we propose an enhanced actor–critic RL agent-driven GAN, called EarlGAN, for denovo drug design. Specifically, EarlGAN’s generator acts as an actor to generate SMILES strings, and the discriminator acts as a critic to perform discrimination. EarlGAN makes autoregressive predictions at the atomic level. While the generator is based on previously generated atoms, the discriminator discriminates using a bidirectional pass over the atoms, including the current atom that is being predicted. We integrate moment, global-level discrimination rewards, and information entropy maximization. The moment rewards reduce the computation time, and the global-level rewards ensure the consistency of the molecule, whereas the information entropy maximization leads to a more diverse sample generation. Experiments and ablation studies verify the effectiveness of EarlGAN for denovo drug design on the QM9 and ZINC datasets. In addition, the visualization analysis provides insight into EarlGAN and supports our conclusion.