RL Model Research Articles

Late acute stress effects on decision-making: The magnified attraction to immediate gains in the iowa gambling task

Previous literature indicates that the later phases of the acute stress response may promote poor decision-making, characterized by riskier choices and a likely inclination towards immediate reward-seeking. However, all studies addressing the effect of this phase have treated decisional capacity as a singular dimension, without analyzing the underlying processes under decision-making. Employing the Value-Plus-Perseveration (VPP) RL model, based on Bayesian logic, this study aims to gain specific insights into how late phase of acute stress impacts the cognitive processes underpinning decision-making in the Iowa Gambling Task (IGT), deciphering whether, as expected, gains are processed in a magnified manner. Seventy-three participants were randomly assigned to two groups, stress (N = 35) and control (N = 38). A virtual version of The Trier Social Stress Test (TSST-VR) was employed as a laboratory stressor. Decision-making was evaluated 35 minutes after the stressor onset, by means of the IGT. Results showed that stressed participants, in comparison to control group, displayed more perseverant and consistent decision-making, enhanced memory, and reinforcement learning capabilities, yet were guided by a greater attraction to decks offering immediate high gains. These results are analyzed with the understanding that in the IGT, short-term decisions focused on instant rewards are seen as counterproductive. This suggests that stress could limit the ability to switch to strategies that are more cautious and offer greater long-term benefits.

Constrained Reinforcement Learning for Fair and Environmentally Efficient Traffic Signal Controllers

Traffic signal controller (TSC) has a crucial role in managing traffic flow in urban areas. Recently, reinforcement learning models have received a great attention for TSC with promising results. However, these RL-TSC models still need to be improved for real-world deployment due to limited exploration of different performance metrics such as fair traffic scheduling or air quality impact. In this work, we introduce a constrained multi-objective RL model that minimizes multiple constrained objectives while achieving a higher expected reward. Furthermore, our proposed RL strategy integrates the peak and average constraint models to the RL problem formulation with maximum entropy off-policy models. We applied this strategy to a single TSC and a network of TSCs. As part of this constrained RL-TSC formulation, we discuss fairness and air quality parameters as constraints for the close-loop control system optimization model at TSCs called FAirLight . Our experimental analysis shows that the proposed FAirLight achieves a good traffic flow performance in terms of average waiting time while being fair and environmentally friendly. Our method outperforms the baseline models and allows a more comprehensive view of RL-TSC regarding its applicability to the real world.

Adaptive real-time energy management strategy using heuristic search for off-road hybrid electric vehicles

The energy management strategies (EMS) of hybrid electric vehicles (HEVs) are put to the test by complex off-road driving conditions. To this end, a novel model-based reinforcement learning (MBRL) algorithm, namely heuristic search, is proposed for EMS. In the MBRL framework, with an online learning Markov Chain (MC) representing the stochastic driving conditions, and a nonlinear state space model describing the deterministic powertrain, an RL model for the HEV is constructed first. Then, heuristic search is introduced to solve the energy management problem, which has two significant advantages: 1) it centers on searching for the optimal action for every current state online; 2) a heuristic function derived from previous experiences is utilized to accelerate the learning. Thus, the optimal actions in each HEV state are learned in real-time, improving the EMS's adaptability to various driving conditions. In the simulation, the proposed EMS is compared with model-free Q-learning (MFQL), model-based Q-learning (MBQL) and dynamic programming (DP) in both off-road driving cycle and standard cycles. Results show that heuristic search only costs about 30 % computing time of MBQL and can maintain better performance than MFQL in various driving conditions.

Synergizing Transfer Learning and Multi-Agent Systems for Thermal Parametrization in Induction Traction Motors

Maintaining optimal temperatures in the critical parts of an induction traction motor is crucial for railway propulsion systems. A reduced-order lumped-parameter thermal network (LPTN) model enables computably inexpensive, accurate temperature estimation; however, it requires empirically based parameter estimation exercises. The calibration process is typically performed in labs in a controlled experimental setting, which is associated with a lot of supervised human efforts. However, the exploration of machine learning (ML) techniques in varied domains has enabled the model parameterization in the drive system outside the laboratory settings. This paper presents an innovative use of a multi-agent reinforcement learning (MARL) approach for the parametrization of an LPTN model. First, a set of reinforcement learning agents are trained to estimate the optimized thermal parameters using the simulated data in several driving cycles (DCs). The selection of a reinforcement learning agent and the level of neurons in the RL model is made based on variability of the driving cycle data. Furthermore, transfer learning is performed on a new driving cycle data collected on the measurement setup. Statistical analysis and clustering techniques are proposed for the selection of an RL agent that has been pre-trained on the historical data. It is established that by synergizing within reinforcement learning techniques, it is possible to refine and adjust the RL learning models to effectively capture the complexities of thermal dynamics. The proposed MARL framework shows its capability to accurately reflect the motor’s thermal behavior under various driving conditions. The transfer learning usage in the proposed approach could yield significant improvement in the accuracy of temperature prediction in the new driving cycles data. This approach is proposed with the aim of developing more adaptive and efficient thermal management strategies for railway propulsion systems.

HumanLight: Incentivizing ridesharing via human-centric deep reinforcement learning in traffic signal control

Single occupancy vehicles are the most attractive transportation alternative for many commuters, leading to increased traffic congestion and air pollution. Advancements in information technologies create opportunities for smart solutions that incentivize ridesharing and mode shift to higher occupancy vehicles (HOVs) to achieve the car lighter vision of cities. In this study, we present HumanLight, a novel decentralized adaptive traffic signal control algorithm designed to optimize people throughput at intersections. Our proposed controller is founded on reinforcement learning with the reward function embedding the transportation-inspired concept of pressure at the person-level. By rewarding HOV commuters with travel time savings for their efforts to merge into a single ride, HumanLight achieves equitable allocation of green times. Apart from adopting FRAP, a state-of-the-art (SOTA) base model, HumanLight introduces the concept of active vehicles, loosely defined as vehicles in proximity to the intersection within the action interval window. The proposed algorithm showcases significant headroom and scalability in different network configurations considering multimodal vehicle splits at various scenarios of HOV adoption. Improvements in person delays and queues range from 15% to over 55% compared to vehicle-level SOTA controllers. We quantify the impact of incorporating active vehicles in the formulation of our RL model for different network structures. HumanLight also enables regulation of the aggressiveness of the HOV prioritization. The impact of parameter setting on the generated phase profile is investigated as a key component of acyclic signal controllers affecting pedestrian waiting times. HumanLight’s scalable, decentralized design can reshape the resolution of traffic management to be more human-centric and empower policies that incentivize ridesharing and public transit systems.

Reinforcement learning for intensive care medicine: actionable clinical insights from novel approaches to reward shaping and off-policy model evaluation

BackgroundReinforcement learning (RL) holds great promise for intensive care medicine given the abundant availability of data and frequent sequential decision-making. But despite the emergence of promising algorithms, RL driven bedside clinical decision support is still far from reality. Major challenges include trust and safety. To help address these issues, we introduce cross off-policy evaluation and policy restriction and show how detailed policy analysis may increase clinical interpretability. As an example, we apply these in the setting of RL to optimise ventilator settings in intubated covid-19 patients.MethodsWith data from the Dutch ICU Data Warehouse and using an exhaustive hyperparameter grid search, we identified an optimal set of Dueling Double-Deep Q Network RL models. The state space comprised ventilator, medication, and clinical data. The action space focused on positive end-expiratory pressure (peep) and fraction of inspired oxygen (FiO2) concentration. We used gas exchange indices as interim rewards, and mortality and state duration as final rewards. We designed a novel evaluation method called cross off-policy evaluation (OPE) to assess the efficacy of models under varying weightings between the interim and terminal reward components. In addition, we implemented policy restriction to prevent potentially hazardous model actions. We introduce delta-Q to compare physician versus policy action quality and in-depth policy inspection using visualisations.ResultsWe created trajectories for 1118 intensive care unit (ICU) admissions and trained 69,120 models using 8 model architectures with 128 hyperparameter combinations. For each model, policy restrictions were applied. In the first evaluation step, 17,182/138,240 policies had good performance, but cross-OPE revealed suboptimal performance for 44% of those by varying the reward function used for evaluation. Clinical policy inspection facilitated assessment of action decisions for individual patients, including identification of action space regions that may benefit most from optimisation.ConclusionCross-OPE can serve as a robust evaluation framework for safe RL model implementation by identifying policies with good generalisability. Policy restriction helps prevent potentially unsafe model recommendations. Finally, the novel delta-Q metric can be used to operationalise RL models in clinical practice. Our findings offer a promising pathway towards application of RL in intensive care medicine and beyond.

TERL: Two-Stage Ensemble Reinforcement Learning Paradigm for Large-Scale Decentralized Decision Making in Transportation Simulation

Transportation simulation is non-trivial due to the co-existence of thousands of heterogeneous decision makers (or vehicles). Such large-scale decision making is intrinsically a complex decentralized problem, the resolution of which is at the forefront of transportation simulation. Despite many physical or mathematical models proposed to date, underlying them is usually a set of universal rules plus some random perturbations to characterize vehicular movements. Inspired by the decision-making mechanism of rational human beings (i.e., learning iteratively from experience), this study proposes a novel two-stage ensemble reinforcement learning (TERL) paradigm for large-scale decentralized decision making in transportation simulation, in order to enhance the computational efficiency and thus scalability of RL to practical applications at scale. After establishing a problem-specific Markov decision process, the first stage utilizes clustering to group heterogeneous vehicles into quasi-homogeneous clusters. A representative RL model (or agent) is employed for each cluster where the included vehicles share and jointly optimize the policy parameters. The second stage develops an ensemble control strategy based on representative RL models for vehicles isolated as noise during clustering. While vehicles are simultaneously simulated, RL models are separately trained with cluster-specific experience replay. Application of TERL to the classical multi-user dynamic route choice problem in a real-world network of the Gusu District in Suzhou, China demonstrates the effectiveness of the proposed approach in deriving desirable simulation results, compared with the classical shortest path model and the dynamic user equilibrium model.

Promoting Green Supply Chain Management With Optimal Selection Of Packaging Materials Using Integrated Fuzzy MCDM and Rl Model

Green supply chain management is highly significant to maintain environmental sustainability. The agglomeration of green components enhances and supports the business activities to practice green supply chain more effectively. Utilizing sustainable packaging materials in logistics is a step towards promoting business eco sustainability. This research work attempts to develop a hybrid decision making model by integrating techniques of fuzzy multi criteria decision making (MCDM) and Reinforcement Learning (RL). This research work proposes a decision-making method of IDOCRIW (Integrated Determination of Objective Criteria Weights) under fuzzy environment with linguistic representations to determine the criterion weights of material selection and applies the RL method of Q learning in ranking the packaging materials for promoting green sustainability. The proposed fuzzy based MCDM method resolves the problems of conflict of uncertainty. The ranking results obtained using this method are compared with the non-integrated MCDM method. The proposed combined model shall be discussed under various other extended fuzzy representations. The decision-making problem on optimal selection of packaging materials addressed in this research work benefits the business decision makers to make right choices. This hybrid model will certainly make the logistic environment more robust and also it will upscale the smart framework of supply chain management.

Real-time adaptive energy management for off-road hybrid electric vehicles based on decision-time planning

Unknown and changeable driving conditions of off-road hybrid electric vehicle (HEV) challenge its energy management strategy (EMS). To tackle this issue, the paper develops a real-time adaptive strategy for off-road HEVs through decision-time planning (DTP), which is a unique method of model-based reinforcement learning (MBRL). First, the MBRL framework for the energy management problem is established, including a RL-oriented model and the DTP algorithm. The RL model consists of a deterministic nonlinear state space model and a stochastic recursive Markov Chain (MC), and the latter is constructed online and updated constantly according to new observations, which can reflect the driving condition precisely. Then, the DTP algorithm is detailed and applied. Instead of learning an overall policy for an entire driving cycle, it seeks to learn the optimal action for each encountered vehicle state, which improves the learning efficiency and realizes the real-time adaptive EMS. In the simulation, assuming that no prior information of the driving conditions is known, the proposed EMS only takes about 1–3% more fuel and 10% more battery life than dynamic programming in both off-road driving conditions and standard road cycles. The EMS significantly outperforms traditional Q-learning and rule-based strategy, verifying its optimization capability and adaptability.

Safe deployment of a reinforcement learning robot using self stabilization

• In RL applications, there is no guarantee that a robot will operate within the same state space in which it was trained. • This paper focuses on ensuring the safe deployment of a robot. • This work defines a condition on the state and action spaces, that if satisfied, guarantees safe recovery. • We also propose a strategy and design that facilitate this recovery within a finite number of steps after perturbation. In toy environments like video games, a reinforcement learning agent is deployed and operates within the same state space in which it was trained. However, in robotics applications such as industrial systems or autonomous vehicles, this cannot be guaranteed. A robot can be pushed out of its training space by some unforeseen perturbation, which may cause it to go into an unknown state from which it has not been trained to move towards its goal. While most prior work in the area of RL safety focuses on ensuring safety in the training phase, this paper focuses on ensuring the safe deployment of a robot that has already been trained to operate within a safe space. This work defines a condition on the state and action spaces, that if satisfied, guarantees the robot’s recovery to safety independently. We also propose a strategy and design that facilitate this recovery within a finite number of steps after perturbation. This is implemented and tested against a standard RL model, and the results indicate a significant improvement in performance.

Empowering Waste Pickers in Brazil: A Case Study of Reverse Logistics

This case study explores the application of Reverse Logistics - RL in the waste pickers cooperatives in Brazil. This study also proposes a novel RL model to integrate society and recycling companies in the effort to clean up the environment and reuse the recycled products as a source of reduction in its production costs. If the government adopts changes to the relationship between the waste pickers cooperatives and manufacturing companies, that will allow municipalities to focus on issues pertinent to the local administration rather than being the "middle man" in the RL chain. With the revised model proposed in this study, waste pickers' cooperatives would negotiate contracts directly with the manufacturers, optimizing the RL chain.

Lithium-ion battery modeling under high-frequency ripple current for co-simulation of high-power DC-DC converters

Studying the output response of lithium-ion batteries under high-frequency ripple current is important for the co-simulation and optimal design of high-power DC-DC converters using IGBTs. However, this influence has not been sufficiently investigated. In this paper, seven equivalent circuit models which include integer and fractional order models are used to simulate the high frequency characteristics of lithium-ion batteries. Thirteen experiments are designed and implemented on three test platforms, including a self-designed DC-DC platform for triangular wave. These experiments consist of average currents of different amplitudes, ripple currents of different types, amplitudes and frequencies. Unlike previous research, the current excitation and voltage response of high frequency ripple are obtained by separating the short and long time scale signals. The equivalent circuit model components are selected based on the frequency domain characteristics of the impedance spectrum. The HMPSO method is employed to implement the parameter identification and simulation analysis of the high frequency model under time domain data. By contrast, the first-order RL model is selected as the preferred choice through modeling and co-simulation. The complexity of the model and co-simulation time is reduced while maintaining voltage accuracy. For the case presented in this paper, the co-simulation time of R(RL) can be reduced by 11.3 % and 30.2 % compared to the RL and R(RL)(RL) models.

Effective deep Q-networks (EDQN) strategy for resource allocation based on optimized reinforcement learning algorithm

The healthcare industry has always been an early adopter of new technology and a big benefactor of it. The use of reinforcement learning in the healthcare system has repeatedly resulted in improved outcomes.. Many challenges exist concerning the architecture of the RL method, measurement metrics, and model choice. More significantly, the validation of RL in authentic clinical settings needs further work. This paper presents a new Effective Resource Allocation Strategy (ERAS) for the Fog environment, which is suitable for Healthcare applications. ERAS tries to achieve effective resource management in the Fog environment via real-time resource allocating as well as prediction algorithms. Comparing the ERAS with the state-of-the-art algorithms, ERAS achieved the minimum Makespan as compared to previous resource allocation algorithms, while maximizing the Average Resource Utilization (ARU) and the Load Balancing Level (LBL). For each application, we further compared and contrasted the architecture of the RL models and the assessment metrics. In critical care, RL has tremendous potential to enhance decision-making. This paper presents two main contributions, (i) Optimization of the RL hyperparameters using PSO, and (ii) Using the optimized RL for the resource allocation and load balancing in the fog environment. Because of its exploitation, exploration, and capacity to get rid of local minima, the PSO has a significant significance when compared to other optimization methodologies.

Scalable Traffic Signal Controls Using Fog-Cloud Based Multiagent Reinforcement Learning

Optimizing traffic signal control (TSC) at intersections continues to pose a challenging problem, particularly for large-scale traffic networks. It has been shown in past research that it is feasible to optimize the operations of individual TSC systems or a small collection of such systems. However, it has been computationally difficult to scale these solution approaches to large networks partly due to the curse of dimensionality that is encountered as the number of intersections increases. Fortunately, recent studies have recognized the potential of exploiting advancements in deep and reinforcement learning to address this problem, and some preliminary successes have been achieved in this regard. However, facilitating such intelligent solution approaches may require large amounts of infrastructure investments such as roadside units (RSUs) and drones, to ensure that connectivity is available across all intersections in the large network. This represents an investment that may be burdensome for the road agency. As such, this study builds on recent work to present a scalable TSC model that may reduce the number of enabling infrastructure that is required. This is achieved using graph attention networks (GATs) to serve as the neural network for deep reinforcement learning. GAT helps to maintain the graph topology of the traffic network while disregarding any irrelevant information. A case study is carried out to demonstrate the effectiveness of the proposed model, and the results show much promise. The overall research outcome suggests that by decomposing large networks using fog nodes, the proposed fog-based graphic RL (FG-RL) model can be easily applied to scale into larger traffic networks.

Energy-Efficient Online Path Planning of Multiple Drones Using Reinforcement Learning

Drones, typically unmanned aerial vehicles (UAVs), have many purposes. However, simultaneous operation of multiple drones is challenging, considering the real-time interactions and the environment; the drone must avoid collision with the other drones or obstacles. The proposed Advanced TD3 model performs energy-efficient path planning at the edge-level drone. We modify the twin-delayed deep deterministic policy gradient (TD3), which is the state-of-the-art policy gradient reinforcement learning. The frame stacking technique considers the continuous action space of the drone to the TD3 model. During the training, we gradually increase the observation range of agents for fast and stable convergence. We train the modified TD3 model through Offline RL to reduce the overhead for the RL model training. Drones mount the converged RL model on their onboard computer. The Advanced TD3 model in the drones selects an energy-efficient path without the overhead of the training process of the RL model in real-time, considering external factors such as wind or another drone. The total energy consumption of drones in flight along with online path planning is approximately 106% of the total energy consumption of drones that follow offline path planning, even though the trained TD3 model does not require complex computations for real-time execution.

New Equivalent Electrical Model of a Fuel Cell and Comparative Study of Several Existing Models with Experimental Data from the PEMFC Nexa 1200 W.

The present work investigates different models of polymer electrolyte membrane fuel cell. More specifically, three models are studied: a nonlinear state-space model, a generic dynamic model integrated into MATLAB/Simulink, and an equivalent RC electrical circuit. A new equivalent electrical RL model is proposed, and the methodology for determining its parameters is also given. An experimental test bench, based on a 1200-W commercial PEMFC, is built to compare the static and dynamic behaviour of the existing models and the proposed RL model with the experimental data. The comparative analysis highlights the advantages and drawbacks of each of these models. The major advantages of the proposed RL model lie in both its simplicity and its ability to provide a similar transitory behaviour compared to the commercially manufactured PEMFC employed in this research.

Routing policy choice prediction in a stochastic network: Recursive model and solution algorithm

We propose a Recursive Logit (STD-RL) model for routing policy choice in a stochastic time-dependent (STD) network, where a routing policy is a mapping from states to actions on which link to take next, and a state is defined by node, time and information. A routing policy encapsulates travelers’ adaptation to revealed traffic conditions when making route choices. The STD-RL model circumvents choice set generation, a procedure with known issues related to estimation and prediction. In a given state, travelers make their link choice maximizing the sum of the utility of the outgoing link and the expected maximum utility until the destination (a.k.a. value function that is a solution to a dynamic programming problem). Existing recursive route choice models and the corresponding solution approaches are based on the assumption that network attributes are deterministic. Hence, they cannot be applied to stochastic networks which are the focus of this paper.We propose an efficient algorithm for solving the value function and its gradient, critical for parameter estimation. It is based on partitioning the state space and decomposing costly matrix operations into a series of simpler ones. We present numerical results using a synthetic network and a network in Stockholm, Sweden. The estimation running time has a 20-30 times speed-up due to matrix decomposition. The estimated model parameters have realistic interpretations. Specifically, travelers are more likely to be adaptive to realized travel times during a longer trip, and more sensitive to travel time when travel time variability is higher. The STD-RL model performs better in predicting route choices than the RL model in a corresponding static and deterministic network.

Reverse logistics network design for waste solar photovoltaic panels: A case study of New South Wales councils in Australia.

Waste solar photovoltaic (PV) panels are considered as one of the fastest-growing future waste streams under the category of large electronic waste (e-waste). The lifespan of solar panels varies from 20 to 30 years, and an appropriate reverse logistics network design is essential to manage the waste stream efficiently once their lifetime expires. Mixed-integer programming-based RL model is proposed in this paper for New South Wales, Australia that minimizes the overall cost by identifying optimal locations and sizing of the collection points while determining optimal capacities for recycling facilities. Using the historical data (2001-2017) on the installed capacity of solar panels in the state, the potential waste generation (at council-level) is estimated and optimized solutions are proposed for the year 2047. The results of the study show that the highest waste solar PV will be generated at Murrumbidgee, Berrigan, Balranald, and Bogan councils. Out of 129 councils in the state, the model identifies 78 optimized-locations of the collection points that would be required in the councils. In the councils of Newcastle, Narrandera and Wagga Wagga, three major recycling facilities would need to be established. This is the first systematic attempt in designing an optimized RL network in Australia focusing on waste solar PV. Policy-makers will find this research highly valuable in decision-making on local recycling infrastructure development.

Estimation of a recursive link-based logit model and link flows in a sensor equipped network

This paper describes a method to estimate the parameters of a Recursive link-based Logit model (RL) using measurements of a set of spatially fixed proximity sensors, with limited hit rates, which can uniquely identify people, such as Wi-Fi-, RFID- or Bluetooth-sensors. The observed ‘route’ of an individual, where we focus on pedestrians in an urban or event context, is modelled as the sequence of sensors that have identified the individual during his or her trip. Obviously, these ‘routes’ contain large gaps, which makes traditional estimation techniques not applicable. Although we do not exactly know what happens within these gaps, we do have some specific insight about the individuals behavior between two identifications; we know with a certain probability which is related to the hit rate of the sensors, that the individual did not cross another sensor location between the two identifications. This paper therefore describes a method to estimate the parameters of an RL model that specifically exploits this knowledge. The framework also allows us to formulate a probabilistic link utilization estimation method, which can be used to estimate link flows in a network based on the sensor observations. The effectiveness of the methodology is demonstrated in simulation using an artificial network, after which the methodology is tested on a real data set, collected at a Dutch music event.

Openness and Integrity in Antitrust

ABSTRACT Reasonable disagreements are pervasive in antitrust, yet the leading antitrust systems function in a broadly effective and consistent manner. How can we explain this paradox? The tentative reply to this question is that the two main antitrust jurisdictions have managed to do so by adopting the features of ‘responsive law’ (RL). Therefore, antitrust institutions could further benefit if they adopt the RL framework to understand and deal with reasonable disagreements. To support this argument, I contend that reasonable disagreements are endogenous in antitrust systems, as they derive from antitrust’s fuzzy mandate, conceptually elastic vocabulary, and rules and standards mode of analysis. In a nutshell, reasonable disagreements are the by-product of two complementary yet antithetical forces of antitrust: openness and integrity. Nonetheless, conventional wisdom has it that such disagreements are temporary indeterminacies that will eventually be eradicated. This view stems from a conceptualization of antitrust as a form of ‘autonomous law’. However, this model of law does not take reasonable disagreements seriously and as a result offers an inadequate modus operandi for dealing with them. The ‘RL’ model, on the contrary, recognizes the endogeneity of reasonable disagreements and the underlying forces that generate them. Instead of attempting to eliminate them, therefore, the RL model suggests that antitrust institutions should seek to tame and exploit them. For this purpose, this model proposes a legal-institutional modus operandi for calibrating the eliciting forces of reasonable disagreements, that is, openness and integrity. The hallmarks of this approach are constructive teleological interpretation, experimentalist network-based enforcement by postbureaucratic enforcers, and courts operating as catalysts.