Q-learning Strategy Research Articles

Neuro-enhanced fractional hysteresis modeling and identification by modified Newton-Raphson optimizer

The modeling and parameter identification of a system with hysteresis remains a difficult task. This paper aims to develop a hysteretic model by combining with a fractional Backlash-like model and cascade forward neural network, and proposed a modified Newton-Raphson-based optimizer parameter identification method to precisely capture the nonlinear behavior of piezoelectric platform. There are three contributions in this paper. Firstly, leveraging the definition of fractional calculus, a fractional Backlash-like model is proposed, whose rate dependence is proved theoretically. Secondly, a hybrid fractional Backlash-like model is established by combining fractional Backlash-like model with cascade forward neural network to enhance the fitting and generalization ability of the model. Thirdly, a modified Newton-Raphson-based optimizer enhances the original Newton-Raphson-based optimizer by incorporating a dynamic reverse learning strategy, a Q-learning strategy, a new adaptive coefficient, and local exploitation based on Levy flight, with theoretical analysis demonstrating improvements in convergence, diversity, and accuracy. The ablation test results show that the accuracy of the proposed modeling method is more than 40% less than that of the root mean square error obtained by the classic Backlash-like model. In addition, compared with the traditional Newton-Raphson-based optimizer, dandelion optimizer, particle swarm optimization and african vultures optimization algorithm, the modified algorithm shows significant advantages in accuracy, convergence speed and robustness under all test frequencies. The root mean square error obtained was reduced by more than 20%. These results show that the proposed method provides a powerful research idea for efficient and accurate hysteresis system modeling and parameter identification.

A Test Report Optimization Method Fusing Reinforcement Learning and Genetic Algorithms

Filtering high-variability and high-severity defect reports from large test report databases is a challenging task in crowdtesting. Traditional optimization algorithms based on clustering and distance techniques have made progress but are limited by initial parameter settings and significantly decrease in efficiency with an increasing number of reports. To address this issue, this paper proposes a method that integrates reinforcement learning with genetic algorithms for crowdsourced testing report optimization, called Reinforcement Learning-based Genetic Algorithm for Crowdsourced Testing Report Optimization (RLGA). Its core goal is to identify distinct, high-severity defect reports from a large set. The method uses genetic algorithms to generate the optimal report selection sequence and adjusts the crossover probability (Pc) and mutation probability (Pm) dynamically with reinforcement learning based on the population’s average fitness, best fitness, and diversity. The reinforcement learning component uses a hybrid SARSA and Q-Learning strategy to update the Q-value table, allowing the algorithm to learn quickly in early iterations and expand the search space later to avoid local optima, thereby improving efficiency. To validate the RLGA method, this paper uses four public datasets and compares RLGA with six classic methods. The results indicate that RLGA outperforms BDDIV in terms of execution time and is less sensitive to the total number of test reports. In terms of optimization objectives, the test reports selected by RLGA have higher levels of defect severity and diversity than those selected by the random choice, BDDIV, and TSE methods. Regarding population diversity, RLGA effectively enhances the uniformity and diversity of individuals compared to random initialization. In terms of convergence speed, RLGA is superior to the GA, GA-SARSA, and GA-Q methods.

An efficient Q-learning integrated multi-objective hyper-heuristic approach for hybrid flow shop scheduling problems with lot streaming

Efficient scheduling in flow shop environments with lot streaming remains a critical challenge in various industrial settings, necessitating innovative approaches to optimize production processes. This study investigates a hybrid flow shop scheduling problem dominant in real-world printed circuit board assembly shops. A novel multi-objective hyper-heuristic combining Q-learning, i.e., two-stage improved spider monkey optimization (TS-ISMO), is tailored to address the complexities of the flow shop scheduling problems. The proposed method aims to simultaneously optimize conflicting objectives such as minimizing makespan, total energy consumption, and total tardiness time while incorporating lot streaming considerations. For multi-objective hyper-heuristic techniques, the algorithm dynamically selects and adapts a diverse set of low-level heuristics to explore the solution space comprehensively and strike a balance among competing objectives. The proposed TS-ISMO algorithm incorporates several significant features aimed at enhancing its performance. These features encompass hybrid heuristics for solution initialization, a contribution value method for comprehensive convergence and diversity assessment, diverse evolutionary state judgments to promote the algorithm’s balance between exploration and exploitation capabilities, and a Q-learning strategy for self-adaptive parameter tuning. The integration of Q-learning facilitates intelligent parameter control, enabling the algorithm to autonomously adjust its behavior based on past experiences and evolution dynamics. This adaptive mechanism enhances convergence speed and solution quality by effectively guiding the search process toward promising regions of the solution space. Extensive computational experiments are conducted on benchmark instances of hybrid flow shop scheduling problems with lot streaming to evaluate the performance of the proposed algorithm. Comparative analyses against state-of-the-art approaches demonstrate its superior solution quality and computational efficiency.

Effect of Q-learning on the evolution of cooperation behavior in collective motion: An improved Vicsek model

There have been numerous studies on collective behavior, among which communication between agents can have a great impact on both the payoff and the cost of making decisions. Research usually focuses on how to improve the collective synchronization rate or accelerate the process of cooperation under given communication cost constraints. In this context, evolutionary game theory (EGT) and reinforcement learning (RL) arise as essential frameworks for tackling this intricate problem. In this study, an adapted Vicsek model is introduced, wherein agents exhibit varying movement patterns contingent on their chosen strategies. Each agent gains a payoff determined by the advantages of collective motion juxtaposed with the cost of communicating with neighboring agents. Individuals choose the objective agents based on the Q-learning strategy and then adapt their strategies following the Fermi rule. The research reveals that the utmost level of cooperation and synchronization can be attained at an optimal communication radius after applying Q-learning. Similar conclusions have been drawn from research on the influence of random noise and relative cost. Different cost functions were considered in the study to demonstrate the robustness of the proposed model and conclusions under a wide range of conditions. (https://github.com/WangchengjieT/VM-EGT-Q)

Multi-objective sustainable flexible job shop scheduling problem: Balancing economic, ecological, and social criteria

Industry 5.0 makes it imperative to reevaluate the manner of using resources in manufacturing systems to ensure sustainability. In this context, scheduling problems are encountering new environmental and human-related challenges, and the concept of sustainable scheduling has gained importance, aiming to balance economic, environmental, and human factors. In this paper, we propose two multi-objective mathematical models to simultaneously address these three factors as objective functions. In the first model, we consider the operator safety while using the Occupational Repetitive Action (OCRA) index to assess ergonomic risks related to task execution. The second model includes workers’ preferences in terms of machines, shifts and task variety. The objective is to improve the general well-being of workers by proposing a schedule that respects as much as possible their preferences. Both models integrate the travel time of workers and products between machines. To solve these NP-hard scheduling problems, we use the Non-dominated Sorting Genetic Algorithms II and III (NSGA-II and NSGA-III), enhanced with a Q-learning strategy for parameter selection and a variable neighborhood search based on reinforcement learning. The obtained results provide a comprehensive analysis of the interactions between these criteria, demonstrating the capability of such approach to achieve a favorable balance between multiple objectives while addressing the new challenges of production scheduling in Industry 5.0 context.

Ensemble meta-heuristics and Q-learning for staff dissatisfaction constrained surgery scheduling and rescheduling

In this study, we investigate the multi-objective surgery scheduling and rescheduling problems with considering medical staff dissatisfaction and fuzzy surgery time. Rescheduling is activated when emergency patients arrive. First, a multi-objective mathematical model is established for maximizing the average patient satisfaction, and minimizing the fuzzy maximum completion time and total medical cost, simultaneously. Second, five meta-heuristics are employed and improved to solve the concerned problems. Five heuristic rules are developed to improve the diversity and quality of initial solutions. For improving the performance of meta-heuristics, six local search operators are designed and two Q-learning-based strategies are developed to select optimal ones intelligently. Finally, 29 instances with different scales are used to verify the performance of the proposed algorithms. Compared with the basic meta-heuristics, the average performance of the algorithms with the second Q-learning-based strategy is improved by 62.5%, 62.1%, 50%, 70.7%, and 70.7%, respectively. Through the Friedman test, the asymptotic significance values of both metrics (0.034 and 0.000) are less than 0.05, indicating that there is a significant performance gap among five algorithms with the second Q-learning-based strategy. The average rank values of the teaching-learning-based optimization with the second Q-learning strategy are 3.7069 and 2.0690 for two metrics, which are better than the compared ones.

Evolutionary Optimization of Fuzzy Reinforcement Learning and Its Application to Time-Varying Tracking Control of Industrial Parallel Robotic Manipulators

This paper contributes to the development of evolutionary optimization of fuzzy reinforcement learning and its application to time-varying tracking control for industrial parallel robotic manipulators. An improved evolutionary social spider optimization (SSO) paradigm with Cauchy mutation is presented to address the optimal fuzzy reinforcement learning problem. The SSO swarm intelligence incorporated with the fuzzy Q-learning strategy, called SSOFQ, is applied to the time-varying tracking control of parallel robotic Stewart manipulators with six degrees of freedom (DOFs). Having the derived kinematics and dynamics of six-DOF Stewart platforms, the SSOFQ is employed to synthesize an intelligent real-time tracking control scheme. More importantly, a custom experimental Stewart platform is constructed using hardware/software codesign and the system-on-a- programmable chip (SoPC) technique. Finally, the simulations, experimental results and comparative works are provided to validate the efficacy and superiority of the presented SSOFQ control method.

An improved spider monkey optimization algorithm for multi-objective planning and scheduling problems of PCB assembly line

In recent years, the demand for printed circuit boards (PCBs) with various electronic components has tremendously increased due to high customization in smart electronic devices. To meet the market competition, efficient and realistic decisions with optimum planning and scheduling are required to enhance the productivity of PCB industries. Processing time of surface mounting technology (SMT) machines defines the pace of PCB production because of time consuming and critical process. In addition, it is observed that various planning and scheduling problems exist in PCB assembly line which are required to emphasize to increase the productivity. Therefore, in this study critical planning and scheduling problems i.e., component allocation problem (CAP) and component placement sequence problem (CPSP) are considered to optimize. Moreover, the periodic maintenance (PM) is also considered to maintain the efficiency of machines and make the problem more realistic. A mathematical model is formulated to minimize the maximum completion time, total energy consumption and the total maintenance time simultaneously. To solve this multi-objective NP-hard problem, an improved spider monkey optimization (ISMO) algorithm is developed. The efficiency and effectiveness of ISMO algorithm is improved by embedding few additional features. a) Two heuristics to ensure better initial solutions; b) evolutionary state judgment based on change in Pareto entropy to ensure the tradeoff between exploration and exploitation; c) an archive-based Q-Learning strategy to ensure the parametric adaptive tuning. Finally, computational experiments and comprehensive performance comparisons with other well-known multi-objective algorithms are performed with several size problem instances. The statistical results and analysis demonstrate that the proposed ISMO algorithm outperforms the other competitors.

Improving the Performance of ALOHA with Internet of Things Using Reinforcement Learning

Intelligent medium access control (MAC) protocols have been a vital solution in enhancing the performance of a variety of wireless networks. ALOHA, as the first MAC approach, inspired the development of several MAC schemes in the network domain, with the primary advantage of simplicity. In this article, we present design, implementation, and performance evaluations of the ALOHA approach, through significant improvements in attaining high channel utilization as the most important performance metric. A critical emphasis is currently focused on removing the burden of packet collisions, while satisfying requirements of energy and delay criteria. We first implement the ALOHA protocol to practically explore its performance behaviors in comparison to analytical models. We then introduce the concept of dynamic payload instead of fixed-length packets, whereby a dynamic selection of the length of each transmitted packet is employed. Another specific contribution of this paper is the integration of the transmission policy of ALOHA with the potential of Internet of Things (IoT) opportunities. The proposed policy utilizes a state-less Q-learning strategy to achieve the maximum performance efficiency. Performance outputs prove that the proposed idea ensures a maximum throughput of approximately 58%, while ALOHA is limited to nearly 18% over a single-hop scenario.

DQN regenerative braking control strategy based on adaptive weight coefficients

Aiming at the problems existing in regenerative braking control strategy based on Q-learning which include the dimensional disaster of state and action variables discretization and the return function weight coefficient determined empirically. This paper proposes deep Q-learning network (DQN) regenerative braking control strategy based on adaptive weight coefficients. Firstly, braking performance evaluation indexes are determined which are braking energy recovery efficiency and braking stability coefficient. Then, the state and action variables and return function are constructed respectively. Therein the braking demand power and state of charge ( SOC) are taken as state variables, braking torque proportional coefficient, and weight coefficients are taken as action variables. And return function is formulated by trading off braking energy recovery efficiency and braking stability. Finally, using the MATLAB/Simulink software, the simulation model of real working condition in Yubei district of Chongqing is established. The simulation results show that braking recovery efficiency of the proposed strategy is 7.4% higher than that of Q-learning strategy, and the average braking stability coefficient is decreased by 0.08. The results indicate the proposed strategy can better balance between braking energy recovery efficiency and braking stability than the conventional strategy.

A novel Q-learning-based hybrid algorithm for the optimal offloading and scheduling in mobile edge computing environments

Mobile Edge Computing (MEC) has arisen as a promising computing paradigm consisting of three tiers: Smart Mobile Devices (SMDs), fog nodes, and the cloud. The MEC enables computational offloading and execution schedules to cope with the problems of insufficient resources for the SMDs and the computational tasks' deadlines. The offloading problem determines in what order and source of the network the tasks should be performed to minimize execution time and power consumption. The main aim of the current paper is to reduce execution time and energy consumption by optimizing tasks' offloading and scheduling in MEC networks. As a result, the task scheduling and offloading are modeled as an optimization problem. Then, an enhanced hybridization of Artificial Ecosystem-based Optimization (AEO) and Arithmetic Optimization Algorithm (AOA), named E-AEO-AOA, is presented to optimize it. In the E-AEO-AOA, the AOA and AEO algorithms are initially discretized. Next, the Q-learning strategy is modified and recruited to hybridize the algorithms in a complementary manner. Subsequently, chaos theory is utilized in a local search procedure to enhance the exploitation capability of the E-AEO-AOA. Eventually, the performance of E-AEO-AOA is examined on fifteen MEC networks. In the experiments, the E-AEO-AOA is compared with AEO, AO, AOA, JS, MRFO, STOA, SCA, and TSA algorithms statistically. Besides, the algorithms' convergence rate and solutions dispersity are visually compared. Moreover, the algorithms are compared by the Wilcoxon signed-rank test. The experimental results indicate that the E-AEO-AOA surpassed competitor algorithms in 90% of cases. Likewise, in 6% of the cases, the E-AEO-AOA produced the same results as AEO, AOA and MRFO.

QSec-RPL: Detection of version number attacks in RPL based mobile IoT using Q-Learning

Internet of Things (IoT) has revolutionized the networking by connecting the real world entities to the Internet. IoT connects the communication devices and has an incredible impact on perspective analytics on the massive volume of data produced every day. An attacker may exploit vulnerabilities of IoT entities and compromise users’ security and privacy. Development of solutions to address security and privacy issues of IoT is in premature stage and considered as challenge. This challenge becomes more critical when the devices in the network are resource-constrained in terms of energy, processing and memory. The IPv6 over Low-Power Wireless Personal Area Networks (6LoWPAN) has emerged in recent years as an adaptation layer to carry IPv6 packets over IEEE 802.15.4. Many IoT applications use Routing Protocol for Low Power and Lossy Networks (RPL) as a network layer protocol developed for routing in 6LoWPAN. Security is challenging in resource constrained environment where encryption may not be a viable solution. Version number attack is one of the most common network layer attacks against RPL based 6LoWPAN. The RPL specification does not address the integrity of the version number and therefore leaves version number mechanism as a weak point in terms of security. This paper investigates the impact of version number attack in RPL networks while considering mobility of the sensor nodes. We propose a solution that utilizes Q-Learning strategy to detect the malicious nodes that are performing version number attack. The proposed approach detects malicious nodes with reasonable accuracy while imposing significantly less overhead on the nodes of low power and lossy networks. There are other approaches too like Message Authentication Codes (MAC) based on symmetric keys but these techniques have memory and communication overhead. So we propose different approach Q-Learning to detect the attacker nodes.

Dual-Layer Q-Learning Strategy for Energy Management of Battery Storage in Grid-Connected Microgrids

Real-time energy management of battery storage in grid-connected microgrids can be very challenging due to the intermittent nature of renewable energy sources (RES), load variations, and variable grid tariffs. Two reinforcement learning (RL)–based energy management systems have been previously used, namely, offline and online methods. In offline RL, the agent learns the optimum policy using forecasted generation and load data. Once the convergence is achieved, battery commands are dispatched in real time. The performance of this strategy highly depends on the accuracy of the forecasted data. An agent in online RL learns the best policy by interacting with the system in real time using real data. Online RL deals better with the forecasted error but can take a longer time to converge. This paper proposes a novel dual layer Q-learning strategy to address this challenge. The first (upper) layer is conducted offline to produce directive commands for the battery system for a 24 h horizon. It uses forecasted data for generation and load. The second (lower) Q-learning-based layer refines these battery commands every 15 min by considering the changes happening in the RES and load demand in real time. This decreases the overall operating cost of the microgrid as compared with online RL by reducing the convergence time. The superiority of the proposed strategy (dual-layer RL) has been verified by simulation results after comparing it with individual offline and online RL algorithms.

Data-Driven Self-Learning Controller Design Approach for Power-Aware IoT Devices based on Double Q-Learning Strategy

Data-Driven Self-Learning Controller Design Approach for Power-Aware IoT Devices based on Double Q-Learning Strategy

A reinforcement learning-based energy management strategy for fuel cell hybrid vehicle considering real-time velocity prediction

The fuel cell vehicle is an ideal new energy vehicle development direction, and its energy management strategy is one of the core technologies to ensure the safe and efficient operation of the vehicle. We proposed a novel reinforcement learning-based energy management method for the fuel cell/lithium battery hybrid system in this paper. In order to improve the reliability of the EMS, the real-time driving profile classification and velocity prediction method based on data driven and statistical analysis is proposed to forecast vehicle velocity in the near future. Then a reinforcement learning method is designed to realize the real-time power allocation. The reward value function which comprehensively considers the system safety, economics and fuel cell durability is creatively put forward. The double Q-learning strategy is applied to update the Q value function. In addition, the real-time reference path of power allocation is designed by taking battery state-of-charge as an indicator. A new dynamic test profile is conducted to verify the proposed method. The multiple groups of comparative simulation experiments show that the proposed EMS can effectively reduce the life decay rate of fuel cell, but also improves fuel economics by up to 6% compared with other commonly used methods.

Advanced deep deterministic policy gradient based energy management strategy design for dual-motor four-wheel-drive electric vehicle

Four-wheel-drive battery electric vehicles (BEV) driven by multiple motors on different axles are getting popular by offering outstanding dynamic and safety performance without sacrificing structure complexity. However, efficiently splitting the power flow between power sources is crucial and difficult. In this study, an intelligent energy management strategy (EMS) is proposed for a specific dual-motor four-wheel-drive (DM-4WD) BEV to reduce energy consumption in unknown traffic conditions. A novel reward factor involved deep deterministic policy gradient (DDPG) algorithm is proposed in EMS design, whose parameters matching are based on particle swarm optimization algorithm to provide a platform to investigate the maximum potential of energy performance improvement for the proposed EMS. The simulation results show that the proposed DDPG-EMS reaches 95.7%, 94.8%, and 95.5% of benchmark dynamic programming-EMS energy performance and outperforms the discontinued-action-based double deep Q-learning strategy in unknow driving cycles. Furthermore, the adaptability of DDPG-EMS is improved by introducing novel rewards setting, which is 3%, 3.8%, and 2.4% better than the traditional State-of-Charge (SOC)-based DDPG-EMS. The simulation results suggest the proposed strategy is efficient and instructive for multi-power BEV EMS design.

Framework for feature selection of predicting the diagnosis and prognosis of necrotizing enterocolitis

Neonatal necrotizing enterocolitis (NEC) occurs worldwide and is a major source of neonatal morbidity and mortality. Researchers have developed many methods for predicting NEC diagnosis and prognosis. However, most people use statistical methods to select features, which may ignore the correlation between features. In addition, because they consider a small dimension of characteristics, they neglect some laboratory parameters such as white blood cell count, lymphocyte percentage, and mean platelet volume, which could be potentially influential factors affecting the diagnosis and prognosis of NEC. To address these issues, we include more perinatal, clinical, and laboratory information, including anemia-red blood cell transfusion and feeding strategies, and propose a ridge regression and Q-learning strategy based bee swarm optimization (RQBSO) metaheuristic algorithm for predicting NEC diagnosis and prognosis. Finally, a linear support vector machine (linear SVM), which specializes in classifying high-dimensional features, is used as a classifier. In the NEC diagnostic prediction experiment, the area under the receiver operating characteristic curve (AUROC) of dataset 1 (feeding intolerance + NEC) reaches 94.23%. In the NEC prognostic prediction experiment, the AUROC of dataset 2 (medical NEC + surgical NEC) reaches 91.88%. Additionally, the classification accuracy of the RQBSO algorithm on the NEC dataset is higher than the other feature selection algorithms. Thus, the proposed approach has the potential to identify predictors that contribute to the diagnosis of NEC and stratification of disease severity in a clinical setting.

Energy Management Strategy for Fuel Cell/Battery/Ultracapacitor Hybrid Electric Vehicles Using Deep Reinforcement Learning With Action Trimming

As for fuel cell hybrid electric vehicle equipped with battery (BAT) and ultracapacitor (UC), its dynamic topology structure is complex and different characteristics of three power sources induce challenges in energy management for fuel economy, power sources lifespan, and dynamic performance of the vehicle. In this paper, an energy management strategy (EMS) based on a hierarchical power splitting structure and deep reinforcement learning (DRL) is proposed. In the higher layer strategy of the proposed EMS, the UC is employed to supply peak power and recover braking energy through the adaptive filter based on fuzzy control. Then, the integrated DRL and equivalent consumption minimization strategy framework is proposed to optimize the power allocation of fuel cell (FC) and BAT in the lower layer, to ensure the highly efficient operation of FC and reduce hydrogen consumption. And the action trimming based on heuristic technique is proposed to further restrain the adverse effect of sudden peak power on FC lifespan. The simulation results show the proposed EMS can make the output of FC smoother, improve its working efficiency to alleviate the stress of BAT, and increase by 14.8% compared with the Q-learning strategy in fuel economy under WLTP driving cycle. Meanwhile, the obtained results under UDDSHDV show fuel economy of the proposed EMS can reach dynamic programming (DP) benchmark level of 89.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> .

An approach of flow compensation incentive based on Q-Learning strategy for IoT user privacy protection

In MCS (mobile crowd sensing), reducing network overhead, protecting IoT user privacy and increasing the participation enthusiasm of perception task are key issues. The QLPPIA (an incentive approach of flow offset based on Q-Learning algorithm for perception user privacy protection) was proposed. A system model that combined MCS with MEC (mobile edge computing) was designed. The edge center uploaded the perception results to the MCS cloud declining its cloud overhead. A privacy protection structure of attribute relevance based on Markov chain Monte Carlo was constructed, which can protect the privacy data. Perception outcomes with higher accuracy of attribute relevance were generated. An incentive pattern of flow offset for user privacy protection based on Q-Learning opportunistic cooperation transfer was designed to cut down MCS cloud flow offset cost and facilitate user enthusiasm. Compared with the existing private protection of high-dimensional attribute data, opportunistic relay perception incentive, and other solutions, the QLPPIA method improves the perception result precision by 27.06%, increases MCS cloud cost of 88.80%, and declines flow compensation expenditure at 19.03% on average.

Energy consumption and battery aging minimization using a Q-learning strategy for a battery/ultracapacitor electric vehicle

Propulsion system electrification revolution has been undergoing in the automotive industry. The electrified propulsion system improves energy efficiency and reduces the dependence on fossil fuel. However, the batteries of electric vehicles experience degradation process during vehicle operation. Research considering both battery degradation and energy consumption in battery/ultracapacitor electric vehicles is still lacking. This study proposes a Q-learning-based strategy to minimize battery degradation and energy consumption. Besides Q-learning, two rule-based energy management methods are also proposed and optimized using Particle Swarm Optimization algorithm. A vehicle propulsion system model is first presented, where the severity factor battery degradation model is considered and experimentally validated with the help of Genetic Algorithm. In the results analysis, Q-learning is first explained with the optimal policy map after learning. Then, the result from a vehicle without ultracapacitor is used as the baseline, which is compared with the results from the vehicle with ultracapacitor using Q-learning, and two rule-based methods as the energy management strategies. At the learning and validation driving cycles, the results indicate that the Q-learning strategy slows down the battery degradation by 13–20% and increases the vehicle range by 1.5–2% compared with the baseline vehicle without ultracapacitor.