Hybrid Reinforcement Learning Research Articles

Multi-Objective Optimization of Resilient, Sustainable, and Safe Urban Bus Routes for Tourism Promotion Using a Hybrid Reinforcement Learning Algorithm

Urban transportation systems in tourism-centric cities face challenges from rapid urbanization and population growth. Efficient, resilient, and sustainable bus route optimization is essential to ensure reliable service, minimize environmental impact, and maintain safety standards. This study presents a novel Hybrid Reinforcement Learning-Variable Neighborhood Strategy Adaptive Search (H-RL-VaNSAS) algorithm for multi-objective urban bus route optimization. Our mathematical model maximizes resilience, sustainability, tourist satisfaction, and accessibility while minimizing total travel distance. H-RL-VaNSAS is evaluated against leading optimization methods, including the Crested Porcupine Optimizer (CPO), Krill Herd Algorithm (KHA), and Salp Swarm Algorithm (SSA). Using metrics such as Hypervolume and the Average Ratio of Pareto Optimal Solutions, H-RL-VaNSAS demonstrates superior performance. Specifically, H-RL-VaNSAS achieved the highest resilience index (550), sustainability index (370), safety score (480), tourist preferences score (300), and accessibility score (2300), while minimizing total travel distance to 950 km. Compared to other methods, H-RL-VaNSAS improved resilience by 12.24–17.02%, sustainability by 5.71–12.12%, safety by 4.35–9.09%, tourist preferences by 7.14–13.21%, accessibility by 4.55–9.52%, and reduced travel distance by 9.52–17.39%. This research offers a framework for designing efficient, resilient, and sustainable public transit systems that align with urban planning and transportation goals. The integration of reinforcement learning with VaNSAS significantly enhances optimization capabilities, providing a valuable tool for mathematical and urban transportation research communities.

Optimizing EV Battery Management: Advanced Hybrid Reinforcement Learning Models for Efficient Charging and Discharging

This paper investigates the application of hybrid reinforcement learning (RL) models to optimize lithium-ion batteries’ charging and discharging processes in electric vehicles (EVs). By integrating two advanced RL algorithms—deep Q-learning (DQL) and active-critic learning—within the framework of battery management systems (BMSs), this study aims to harness the combined strengths of these techniques to improve battery efficiency, performance, and lifespan. The hybrid models are put through their paces via simulation and experimental validation, demonstrating their capability to devise optimal battery management strategies. These strategies effectively adapt to variations in battery state of health (SOH) and state of charge (SOC) relative error, combat battery voltage aging, and adhere to complex operational constraints, including charging/discharging schedules. The results underscore the potential of RL-based hybrid models to enhance BMSs in EVs, offering tangible contributions towards more sustainable and reliable electric transportation systems.

An autonomous stair climbing method for the crawler robot based on a hybrid reinforcement learning controller

In response to problems in using deep reinforcement learning to perform the stair climbing task for the crawler robot, such as the lack of theoretical values for stair size range, the inaccurate force on the model using wheel to simulate track and difficulty in training, we propose an autonomous stair climbing method for the crawler robot based on a hybrid reinforcement learning controller. We propose a dynamics parameter settings method for fast simulation model by studying the distribution model of the ground pressure of the track, which improves the accuracy of the force on the fast simulation model in the stair climbing task. Our method calculates the climbing ability of the crawler robot by studying the kinematics and dynamics models of the real crawler robot in each stage of the stair climbing task, and combining with the control flow of the hybrid reinforcement learning controller. Further, we develop the training course of the hybrid reinforcement learning controller. The agent is trained in ROS-Gazebo. The course reduces the impact of poor trajectory on training, achieves higher success rates and faster convergence speed compared to random training, and effectively improves the generalization ability of the strategy.

Mild exogenous inflammation blunts neural signatures of bounded evidence accumulation and reward prediction error processing in healthy male participants

BackgroundAltered neural haemodynamic activity during decision making and learning has been linked to the effects of inflammation on mood and motivated behaviours. So far, it has been reported that blunted mesolimbic dopamine reward signals are associated with inflammation-induced anhedonia and apathy. Nonetheless, it is still unclear whether inflammation impacts neural activity underpinning decision dynamics. The process of decision making involves integration of noisy evidence from the environment until a critical threshold of evidence is reached. There is growing empirical evidence that such process, which is usually referred to as bounded accumulation of decision evidence, is affected in the context of mental illness. MethodsIn a randomised, placebo-controlled, crossover study, 19 healthy male participants were allocated to placebo and typhoid vaccination. Three to four hours post-injection, participants performed a probabilistic reversal-learning task during functional magnetic resonance imaging. To capture the hidden neurocognitive operations underpinning decision-making, we devised a hybrid sequential sampling and reinforcement learning computational model. We conducted whole brain analyses informed by the modelling results to investigate the effects of inflammation on the efficiency of decision dynamics and reward learning. ResultsWe found that during the decision phase of the task, typhoid vaccination attenuated neural signatures of bounded evidence accumulation in the dorsomedial prefrontal cortex, only for decisions requiring short integration time. Consistent with prior work, we showed that, in the outcome phase, mild acute inflammation blunted the reward prediction error in the bilateral ventral striatum and amygdala. ConclusionsOur study extends current insights into the effects of inflammation on the neural mechanisms of decision making and shows that exogenous inflammation alters neural activity indexing efficiency of evidence integration, as a function of choice discriminability. Moreover, we replicate previous findings that inflammation blunts striatal reward prediction error signals.

Optimal control for continuous-time Markov jump singularly perturbed systems : A hybrid reinforcement learning scheme

This article discusses the adaptive optimal control problem for continuous-time Markov jump singularly perturbed systems with unknown system dynamics. First, the subsystems transformation technique is introduced to reconstruct the system with stochastic jump characteristics, yielding a set of parallel subsystems. Next, under the framework of reinforcement learning, an offline model-based hybrid iteration algorithm is developed to approximate the solution of the full-order coupled algebraic Riccati equations. Following this, to escape from the system model constraints, an online model-free hybrid iteration algorithm is introduced and relevant convergence proof is given. Compared to traditional value iteration and policy iteration algorithms, the hybrid iteration algorithm has ideal convergence rate and eliminates the requirement of initial stabilizing control policy. At the end, the availability of the online algorithm is demonstrated by means of an operational amplifier circuit model as a simulation example.

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

This paper proposes an innovative method for achieving optimal tracking control in nonlinear continuous-time systems with input constraints. The method combines reinforcement learning and hybrid metaheuristics to enhance the controller’s performance. Specifically, a hybrid metaheuristic algorithm is employed to optimize the hyperparameters of a critic neural network, which serves as the system’s controller. The proposed approach is evaluated through extensive simulation studies on a nonlinear system with input constraints. Results demonstrate its superiority over traditional control techniques in terms of accuracy, timeliness, and stability. Notably, the approach effectively eliminates overshoot and steady-state error while providing precise and prompt tracking and showcasing remarkable robustness against model uncertainties. By synergistically integrating reinforcement learning and hybrid metaheuristics, this approach represents a significant advancement in enhancing the control performance of complex nonlinear systems. The simulation studies confirm superiority of the proposed approach over existing techniques, offering a promising solution for achieving optimal tracking control in nonlinear systems with input constraints. This approach holds potential for a wide range of applications, including robotics, aerospace, and manufacturing, where precise and prompt tracking control is critical.

Data Quality-Aware Mixed-Precision Quantization via Hybrid Reinforcement Learning.

Mixed-precision quantization mostly predetermines the model bit-width settings before actual training due to the non-differential bit-width sampling process, obtaining suboptimal performance. Worse still, the conventional static quality-consistent training setting, i.e., all data is assumed to be of the same quality across training and inference, overlooks data quality changes in real-world applications which may lead to poor robustness of the quantized models. In this article, we propose a novel data quality-aware mixed-precision quantization framework, dubbed DQMQ, to dynamically adapt quantization bit-widths to different data qualities. The adaption is based on a bit-width decision policy that can be learned jointly with the quantization training. Concretely, DQMQ is modeled as a hybrid reinforcement learning (RL) task that combines model-based policy optimization with supervised quantization training. By relaxing the discrete bit-width sampling to a continuous probability distribution that is encoded with few learnable parameters, DQMQ is differentiable and can be directly optimized end-to-end with a hybrid optimization target considering both task performance and quantization benefits. Trained on mixed-quality image datasets, DQMQ can implicitly select the most proper bit-width for each layer when facing uneven input qualities. Extensive experiments on various benchmark datasets and networks demonstrate the superiority of DQMQ against existing fixed/mixed-precision quantization methods.

Hybrid Reinforcement Learning for Optimal Control of Non-Linear Switching System.

Based on the reinforcement learning mechanism, a data-based scheme is proposed to address the optimal control problem of discrete-time non-linear switching systems. In contrast to conventional systems, in the switching systems, the control signal consists of the active mode (discrete) and the control inputs (continuous). First, the Hamilton-Jacobi-Bellman equation of the hybrid action space is derived, and a two-stage value iteration method is proposed to learn the optimal solution. In addition, a neural network structure is designed by decomposing the Q-function into the value function and the normalized advantage value function, which is quadratic with respect to the continuous control of subsystems. In this way, the Q-function and the continuous policy can be simultaneously updated at each iteration step so that the training of hybrid policies is simplified to a one-step manner. Moreover, the convergence analysis of the proposed algorithm with consideration of approximation error is provided. Finally, the algorithm is applied evaluated on three different simulation examples. Compared to the related work, the results demonstrate the potential of our method.

A Hybrid Online Off-Policy Reinforcement Learning Agent Framework Supported by Transformers.

Reinforcement learning (RL) is a powerful technique that allows agents to learn optimal decision-making policies through interactions with an environment. However, traditional RL algorithms suffer from several limitations such as the need for large amounts of data and long-term credit assignment, i.e. the problem of determining which actions actually produce a certain reward. Recently, Transformers have shown their capacity to address these constraints in this area of learning in an offline setting. This paper proposes a framework that uses Transformers to enhance the training of online off-policy RL agents and address the challenges described above through self-attention. The proposal introduces a hybrid agent with a mixed policy that combines an online off-policy agent with an offline Transformer agent using the Decision Transformer architecture. By sequentially exchanging the experience replay buffer between the agents, the agent's learning training efficiency is improved in the first iterations and so is the training of Transformer-based RL agents in situations with limited data availability or unknown environments.

Decentralized Optimization for Fuzzy Singularly Perturbed Interconnected Systems: A Hybrid Reinforcement Learning Method

Decentralized Optimization for Fuzzy Singularly Perturbed Interconnected Systems: A Hybrid Reinforcement Learning Method

Hybrid Reinforcement Learning for Power Transmission Network Self-Healing Considering Wind Power.

Transmission network self-healing considering uncertain wind power becomes crucial with increasing penetration of wind power. A hybrid reinforcement learning (HRL) method combining offline self-learning with online Monte Carlo tree search (MCTS) is designed to deal with the strong uncertainty induced by wind power restoration. The HRL method trains a policy network with offline self-learning based on historical wind and transmission system data. It then applies the policy network to guide MCTS to realize step-by-step transmission network self-healing based on real-time and forecast data in different wind power scenarios. Besides, a model predictive control method for active power dispatch is proposed to improve wind power generation credibility during self-healing. Simulation results of both test and real-life power systems demonstrate that the proposed method can realize online transmission system self-healing reliably. Comparisons among different reinforcement learning methods indicate that the number of scenarios dominated by HRL is more than twice that dominated by MCTS and a dozen times that dominated by deep Q-network. Meanwhile, the online method is more flexible in uncertain wind power scenarios than optimization methods.

A Hybrid Simulation and Reinforcement Learning Algorithm for Enhancing Efficiency in Warehouse Operations

The use of simulation and reinforcement learning can be viewed as a flexible approach to aid managerial decision-making, particularly in the face of growing complexity in manufacturing and logistic systems. Efficient supply chains heavily rely on steamlined warehouse operations, and therefore, having a well-informed storage location assignment policy is crucial for their improvement. The traditional methods found in the literature for tackling the storage location assignment problem have certain drawbacks, including the omission of stochastic process variability or the neglect of interaction between various warehouse workers. In this context, we explore the possibilities of combining simulation with reinforcement learning to develop effective mechanisms that allow for the quick acquisition of information about a complex environment, the processing of that information, and then the decision-making about the best storage location assignment. In order to test these concepts, we will make use of the FlexSim commercial simulator.

Asynchronous Hybrid Reinforcement Learning for Latency and Reliability Optimization in the Metaverse Over Wireless Communications

Technology advancements in wireless communications and high-performance Extended Reality (XR) have empowered the developments of the Metaverse. The demand for the Metaverse applications and hence, real-time digital twinning of real-world scenes is increasing. Nevertheless, the replication of 2D physical world images into 3D virtual objects is computationally intensive and requires computation offloading. The disparity in transmitted object dimension (2D as opposed to 3D) leads to asymmetric data sizes in uplink (UL) and downlink (DL). To ensure the reliability and low latency of the system, we consider an asynchronous joint UL-DL scenario where in the UL stage, the smaller data size of the physical world images captured by multiple extended reality users (XUs) will be uploaded to the Metaverse Console (MC) to be construed and rendered. In the DL stage, the larger-size 3D virtual objects need to be transmitted back to the XUs. We design a novel multi-agent reinforcement learning algorithm structure, namely Asynchronous Actors Hybrid Critic (AAHC), to optimize the decisions pertaining to computation offloading and channel assignment in the UL stage and optimize the DL transmission power in the DL stage. Extensive experiments demonstrate that compared to proposed baselines, AAHC obtains better solutions with satisfactory training time.

Learn with Curiosity: A Hybrid Reinforcement Learning Approach for Resource Allocation for 6G enabled Connected Cars

Learn with Curiosity: A Hybrid Reinforcement Learning Approach for Resource Allocation for 6G enabled Connected Cars

Distributed consensus for nonlinear multi-agent systems with two-time-scales: A hybrid reinforcement learning consensus algorithm

The optimal consensus problem for nonlinear two-time-scales multi-agent systems with completely unknown system dynamics is investigated in this paper. First, the original system is linearly represented based on the Takagi-Sugeno fuzzy model. Then, the optimal consensus problem for multi-agent systems is transformed into solving the game algebraic Riccati equation associated with agents and their neighbors. And individual agent dynamics studied in this paper are replaced with local error dynamics. Moreover, an offline hybrid iteration algorithm with rapid convergence speed and no initial stable control policy is presented for multi-agent systems. Meanwhile, to avoid the utilization of the knowledge of system matrices, an online hybrid reinforcement learning algorithm that only uses the state and control input data of each agent and its neighbors is given to generate the distributed optimal control policy. The convergence of proposed algorithms is also discussed. Finally, the applicability of the presented method is illustrated by an example.

A Hybrid Reinforcement Learning Approach With a Spiking Actor Network for Efficient Robotic Arm Target Reaching

The increasing demand for applications in competitive fields, such as assisted living and aerial robots, drives contemporary research into the development, implementation and integration of power-constrained solutions. Although, deep neural networks (DNNs) have achieved remarkable performances in many robotics applications, energy consumption remains a major limitation. The paper at hand proposes a hybrid variation of the well-established deep deterministic policy gradient (DDPG) reinforcement learning approach to train a 6 degree of freedom robotic arm in the target-reach task <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://www.youtube.com/watch?v=238693OpvD0.</uri> In particular, we introduce a spiking neural network (SNN) for the actor model and a DNN for the critic one, aiming to find an optimal set of actions for the robot. The deep critic network is employed only during training and discarded afterwards, allowing the deployment of the SNN in neuromorphic hardware for inference. The agent is supported by a combination of RGB and laser scan data exploited for collision avoidance and object detection. We compare the hybrid-DDPG model against a classic DDPG one, demonstrating the superiority of our approach.

Hybrid Particle Swarm Optimization Algorithm Based on the Theory of Reinforcement Learning in Psychology

To more effectively solve the complex optimization problems that exist in nonlinear, high-dimensional, large-sample and complex systems, many intelligent optimization methods have been proposed. Among these algorithms, the particle swarm optimization (PSO) algorithm has attracted scholars’ attention. However, the traditional PSO can easily become an individual optimal solution, leading to the transition of the optimization process from global exploration to local development. To solve this problem, in this paper, we propose a Hybrid Reinforcement Learning Particle Swarm Algorithm (HRLPSO) based on the theory of reinforcement learning in psychology. First, we used the reinforcement learning strategy to optimize the initial population in the population initialization stage; then, chaotic adaptive weights and adaptive learning factors were used to balance the global exploration and local development process, and the individual optimal solution and the global optimal solution were obtained using dimension learning. Finally, the improved reinforcement learning strategy and mutation strategy were applied to the traditional PSO to improve the quality of the individual optimal solution and the global optimal solution. The HRLPSO algorithm was tested by optimizing the solution of 12 benchmarks as well as the CEC2013 test suite, and the results show it can balance the individual learning ability and social learning ability, verifying its effectiveness.

TC-Driver: A Trajectory Conditioned Reinforcement Learning Approach to Zero-Shot Autonomous Racing

Autonomous racing is becoming popular for academic and industry researchers as a test for general autonomous driving by pushing perception, planning, and control algorithms to their limits. While traditional control methods such as model predictive control are capable of generating an optimal control sequence at the edge of the vehicles’ physical controllability, these methods are sensitive to the accuracy of the modeling parameters, such as tire modeling coefficients. As model mismatch is inevitable in reality, the heuristic nature of Reinforcement Learning (RL) offers a viable approach to modeling robustness. This paper presents TC-Driver, an RL approach for robust control in autonomous racing. In particular, the TC-Driver agent is conditioned by a trajectory generated by any arbitrary traditional high-level trajectory planner. The proposed TC-Driver architecture addresses the tire parameter modeling inaccuracies by exploiting the learning capabilities of RL while utilizing the reliability of traditional planning methods in a hybrid fashion. We train the agent under varying tire conditions, allowing it to generalize to different model parameters, aiming to increase the racing capabilities of the system in practice. Experimental results demonstrate that the proposed hybrid RL architecture of the TC-Driver improves the generalization robustness of autonomous racing agents when compared to a previous state-of-the-art end-to-end-based architecture. Namely, the proposed controller yields a 29-fold improvement in crash ratio when facing model mismatch and can zero-shot transfer its behavior on unseen tracks which present completely new features, while the end-to-end baseline fails. When deployed on a physical system, the proposed architecture demonstrates zero-shot Sim2Real capabilities that outperform end-to-end agents 10-fold in terms of crash ratio while exhibiting similar driving characteristics in reality as in simulation.

A Hybrid Reinforcement Learning Algorithm for 2D Irregular Packing Problems

Packing problems, also known as nesting problems or bin packing problems, are classic and popular NP-hard problems with high computational complexity. Inspired by classic reinforcement learning (RL), we established a mathematical model for two-dimensional (2D) irregular-piece packing combined with characteristics of 2D irregular pieces. An RL algorithm based on Monte Carlo learning (MC), Q-learning, and Sarsa-learning is proposed in this paper to solve a 2D irregular-piece packing problem. Additionally, mechanisms of reward–return and strategy-update based on piece packing were designed. Finally, the standard test case of irregular pieces was used for experimental testing to analyze the optimization effect of the algorithm. The experimental results show that the proposed algorithm can successfully realize packing of 2D irregular pieces. A similar or better optimization effect can be obtained compared to some classical heuristic algorithms. The proposed algorithm is an early attempt to use machine learning to solve 2D irregular packing problems. On the one hand, our hybrid RL algorithm can provide a basis for subsequent deep reinforcement learning (DRL) to solve packing problems, which has far-reaching theoretical significance. On the other hand, it has practical significance for improving the utilization rate of raw materials and broadening the application field of machine learning.

Stochastic pedestrian avoidance for autonomous vehicles using hybrid reinforcement learning

Stochastic pedestrian avoidance for autonomous vehicles using hybrid reinforcement learning