Reinforcement Learning-based Approach Research Articles

Coordinated Ramp Metering Considering the Dynamics of Mixed-Autonomy Traffic

The introduction of connected autonomous vehicles may bring opportunities and challenges to traditional traffic control instruments, like ramp metering. This paper starts by constructing the fundamental diagram for mixed-autonomy traffic based on the car-following behaviors of both connected autonomous vehicles and human-driven vehicles. Then, analyses are performed on the main factors that influence the critical velocity, critical density, and road capacity under mixed-autonomy traffic. Two methods named COE-HERO and TRLCRM are developed to support the implementations of coordinated ramp metering for freeways with mixed-autonomy traffic. The COE-HERO method enhances the HERO method by incorporating a critical occupancy estimation module. Both COE-HERO and TRLCRM consider dynamic traffic flow parameters of mixed-autonomy traffic. The TRLCRM method is a reinforcement learning-based approach with a two-stage training framework, enabling it to adapt to varying mixed-autonomy demand scenarios. Extensive microscopic simulations show that the learning-based TRLCRM method can effectively alleviate bottleneck congestion and is robust to deal with various traffic scenarios. The COE-HERO method performs better than the HERO method, indicating the necessity of critical occupancy estimation in the implementations of coordinated ramp metering.

A reinforcement learning-based approach for solving multi-agent job shop scheduling problem

Wafer processing is the most expensive, time-consuming, and complex stage in semiconductor manufacturing. It varies significantly based on orders of customers (agents). Optimising the wafer processing flow in a multi-agent scenario can meet customised requirements, speed up delivery, and reduce costs. This work models wafer processing as a multi-agent job shop scheduling problem (MAJSP) with release dates. The objective is to minimise the total weighted makespan of agents. To address both dynamic and static scheduling scenarios in the MAJSP context, two deep reinforcement learning-based (DRL) methods are proposed. In a dynamic scheduling scenario, the statuses of orders and production resources can change at any moment. A DRL method called Graph Transformer Network (GTN) is proposed to rapidly generate high-quality solutions. In a static scheduling scenario, the production plan can be formulated based on predetermined demand and resource conditions. A novel hybrid method (GTN-DABC) that combines GTN with the discrete artificial bee colony algorithm (DABC) is proposed to provide high-quality production plans for manufacturers within an acceptable computation time. Experimental results demonstrate that the proposed GTN outperforms existing heuristics, and the well-designed GTN-DABC is more competitive than other meta-heuristics.

Multi-Agent Reinforcement Learning-Based Maximum Power Point Tracking Approach to Fortify PMSG-Based WECSs

Multi-Agent Reinforcement Learning-Based Maximum Power Point Tracking Approach to Fortify PMSG-Based WECSs

Reproduction and generalization of robot trajectory tracking control method using reinforcement learning and neural network techniques

Abstract. This paper introduces a method for controlling the trajectory of a manipulator robot with uncertainty, using reinforcement learning. The control is designed to work even when there are limitations on the inputs to the system. Reinforcement learning and neural networks are employed alongside standard robust control techniques to enhance the fixed-time convergence of the system state. A novel algorithm is proposed to develop a reinforcement learning-based approach. This approach utilizes radial basis function neural networks and nonsingular fast terminal sliding mode control to ensure error convergence within a predetermined time. This paper presents the task of monitoring the intended path followed by robotic arms in the presence of uncertain and unfamiliar disruptions. The experimental results validate that the suggested approach substantially improves both the stability and accuracy of trajectory tracking, making it more feasible for real-world applications in robotic systems.

Advancing AutoML: A Deep Dive into Neural Architecture Search (NAS) and Its Optimization Techniques

Abstract: Neural Architecture Search (NAS) is a pivotal technique in the field of automated machine learning (AutoML), enabling the automatic design of optimal neural network architectures. As deep learning models grow in complexity, NAS offers a scalable approach to improving model performance by exploring vast search spaces of potential architectures. In our research, we investigate the mathematical foundations and algorithms underpinning NAS, focusing on reinforcement learning-based, evolutionary, and gradient-based approaches. We provide mathematical proofs of convergence and efficiency for each method and analyze real-world applications, such as image classification and natural language processing (NLP). Through a comprehensive exploration of NAS, we aim to highlight its impact on AutoML and its potential to automate neural network design effectively while addressing challenges in computational cost and generalization.

Machine learning-based intelligent localization technique for channel classification in massive MIMO

Multiple-input multiple-output (MIMO) technology has been widely adopted in wireless communications, which enables the simultaneous transmission of multiple data streams via multiple transmitting and receiving antennas. In a MIMO system with non-line-of-sight (NLOS), transmitted signals are reflected by various obstacles along the path, reaching the antenna at different angles and times. In 5G networks, the NLOS problem is a major challenge for massive MIMO localization, significantly reducing positioning accuracy. In this work, an intelligent localization technique based on NLOS identification and mitigation is proposed to address this problem. In this solution, a Convolutional Neural Network (CNN) based hybrid Archimedes-based Salp Swarm Algorithm (HASSA) technique is proposed to detect NLOS or the line of sight (LOS) and estimate the location. The accuracy can be analyzed by considering the angle of arrival of signals, threshold-based time of arrival, and time difference of arrival from different antennas. A novel reinforcement learning-based optimization approach is used for the mitigation of NLOS in the radio wave propagation path, which in turn reduces the computational complexity. We use the Ensemble Deep Deterministic Policy Gradient-Based Approach (EDDPG)-based Honey Badger algorithm (HBA) for the aforementioned process. The simulation of this approach assesses diverse scenarios and considers different parameters, and the approach is compared with various state-of-the-art works. From the simulation results, our proposed approach can be used for the identification and detection of LOS and NLOS components and can precisely enhance the localization compared with other approaches.

Enhancing PV Hosting Capacity of Electricity Distribution Networks Using Deep Reinforcement Learning-Based Coordinated Voltage Control

Coordinated voltage control enables the active management of voltage levels throughout electricity distribution networks by leveraging the voltage support capabilities of existing grid-connected PV inverters. The efficient management of power flows and precise voltage regulation through coordinated voltage control schemes facilitate the increased adoption of rooftop PV systems and enhance the hosting capacity of electricity distribution networks. The research work presented in this paper proposes a coordinated voltage control scheme and evaluates the enhanced hosting capacity utilizing a deep reinforcement learning-based approach. A comparative analysis of the proposed algorithm is presented, and the performance is benchmarked against existing local voltage control schemes. The proposed coordinated voltage control scheme in this paper is evaluated using simulations on a real-world low-voltage electricity distribution network. The evaluation involves quasi-static time series power flow simulations for assessing performance. Furthermore, a discussion is presented that reflects on the strengths and limitations of the proposed scheme based on the results observed from the case study.

Gaussian process-based online sensor selection for source localization

This paper addresses the sensor selection problem for source localization within cyber–physical systems (CPSs). While recent machine learning and reinforcement learning approaches aim to optimize sensor selection and placement within the Area of Interest (AoI), their need for intensive data collection and training precludes online operation. Furthermore, these methods often require prior knowledge of the unknown source’s characteristics and lack adaptability to the dynamic nature of CPSs, leading to inefficiencies in unseen environments. This paper addresses these shortcomings using Gaussian process Optimization coupled with an active sensor selection mechanism to locate the unknown source within the AoI. The proposed approach first builds a probabilistic model of the environment, which is discretized into a grid, without prior training using a Gaussian Process surrogate model. Next, the model iteratively and systematically learns the underlying spatial phenomenon using Gaussian Process optimization. Concurrently, the approach selects a subset of sensors by optimizing a fitness function that advocates selecting informative and energy-efficient sensors. Next, the probabilistic model, having accurately learned the environment, directs the algorithm to the unknown source by identifying the cell with the highest likelihood of containing it. Finally, a peak refinement step is performed, which computes the exact location of the source within the designated cell. The proposed method’s efficacy is validated through experiments in radioactive source localization, validation studies, and adaptability assessments across various environments. In terms of quality of localization (QoL), it outperforms recent localization benchmarks, such as a reinforcement learning-based approach and DANS, by around 18% and 100%, respectively.

Clustering and Cooperative Guidance of Multiple Decoys for Defending a Naval Platform against Salvo Threats

The threat to naval platforms from missile systems is increasing due to recent advancements in radar seeker technology, which have significantly enhanced the accuracy and effectiveness of missile targeting. In scenarios where a naval platform with limited maneuverability faces salvo attacks, the importance of an effective defense strategy becomes crucial to ensuring the protection of the platform. In this study, we present a multi-agent reinforcement learning-based decoy deployment approach that employs six decoys to increase the survival likelihood of a naval platform against salvo missile strikes. Our approach entails separating the decoys into two clusters, each consisting of three decoys. Subsequently, every cluster is allocated to a related missile threat. This is accomplished by training the decoys with the multi-agent deep reinforcement learning algorithm. To compare the proposed approach across different algorithms, we use two distinct algorithms to train the decoys; multi-agent deep deterministic policy gradient (MADDPG) and multi-agent twin-delayed deep deterministic policy gradient (MATD3). Following training, the decoys learn to form groups and establish effective formation configurations within each group to ensure optimal coordination. We assess the proposed decoy deployment strategy using parameters including decoy deployment angle and maximum decoy speed. Our findings indicate that decoys positioned on the same side outperform those positioned on different sides relative to the target platform. In general, MATD3 performs slightly better than MADDPG. Decoys trained with MATD3 succeed in more successful formation configurations than those trained with the MADDPG method, which accounts for this enhancement.

Dynamic soft-kill weapon-target assignment in naval environments

One of the most significant threats faced by ships is anti-ship missiles. Nowadays, these missiles, equipped with diverse guidance systems, can locate their trajectory and attack the ship. Consequently, ships need to utilize their weapons to attempt to neutralize these threats. This article aims to develop dynamic assignment algorithms to assign a ship’s defensive soft-kill weapons to a set of incoming missiles, to minimize the average damage inflicted on the ship. To this end, initially, a binary linear programming model is developed to solve the static weapon-target assignment problem. Subsequently, a simulation–optimization algorithm and a reinforcement learning-based approach, grounded in the value iteration algorithm, are developed to solve the dynamic weapon-target assignment problem. To compare and evaluate the performance of the developed solution methods, we employ a set of randomly generated test instances. Computational results indicate that the reinforcement learning approach, due to its inherent foresight, outperforms the simulation–optimization approach in reducing the inflicted damages. However, in terms of CPU run time, the simulation–optimization approach is more efficient.

A Reinforcement Learning-Based Approach for Promoting Mental Health Using Multimodal Emotion Recognition

This research aims to enhance mental well-being by addressing symptoms of anxiety and depression through a personalized, culturally specific multimodal emotion prediction system. It employs an emotionally aware Reinforcement Learning (RL) agent to suggest tailored Cognitive Behavioral Therapy (CBT) activities. The study focuses on developing precise, individualized emotion prediction models using facial expressions, vocal tones, and text, and integrates these models with the RL agent for emotionally aware CBT recommendations. The mHealth approach combines deep learning models with RL, achieving accuracies of 72% for facial expressions, 73% for vocal tones, and 86% for text, all fine-tuned for the Sri Lankan context. Validation through real-world use and user feedback consistently demonstrated that each model exceeds 70% accuracy, fulfilling the objective of precise emotion prediction. A weighted algorithm was introduced to refine the emotion prediction experience and personalize forecasts across the three modalities to enhance mental well-being. The RL-enabled agent suggests CBT activities approved by mental health professionals, tailored based on predicted emotions, and delivered through the same mHealth application. The effectiveness of these interventions was assessed using the DASS-21 questionnaire, revealing significant reductions in depression scores (from 21.08 to 13.54) and anxiety scores (from 19.85 to 10.46) in the study group compared to the control group. The study concludes that integrating multimodal emotion prediction models with RL-based CBT suggestions positively impacts mental well-being and contributes to personalized mental health interventions.

Blockchain-Assisted Secure Energy Trading in Electricity Markets: A Tiny Deep Reinforcement Learning-Based Stackelberg Game Approach

Blockchain-Assisted Secure Energy Trading in Electricity Markets: A Tiny Deep Reinforcement Learning-Based Stackelberg Game Approach

RL-based mobile edge computing scheme for high reliability low latency services in UAV-aided IIoT networks

The prevailing adoption of Internet of Things paradigm is giving rise to a wide range of use cases in various vertical industries including remote health, industrial automation, and smart agriculture. However, the realization of such use cases is mainly challenged due to their stringent service requirements of high reliability and low latency. This challenge grows further when the service entails processing collected data for informed decision making. In this work, we consider a field of industrial Internet of Things devices that generate computational tasks and are covered by a nearby base station equipped with an edge server. The edge server offers fast processing to the devices’ tasks to help in meeting their latency requirement. Due to statistical wireless variability, the task data may not be correctly delivered in time for processing. To this end, we utilize an unmanned aerial vehicle as a supplemental edge server that tailors its trajectory and flies closer to the IIoT devices to ensure a highly reliable task delivery based on the given task reliability constraints. We formulate the problem as a Markov Decision Process, and propose a deep reinforcement learning-based approach using proximal policy optimization to optimize the unmanned aerial vehicle trajectory and scheduling devices to offload their data for processing. We present simulation results for various system scenarios to illustrate the effectiveness of the proposed solution as compared to several baseline approaches.

Electric Vehicle Charging Guidance Strategy Considering “Traffic Network-Charging Station-Driver” Modeling: A Multiagent Deep Reinforcement Learning-Based Approach

Electric Vehicle Charging Guidance Strategy Considering “Traffic Network-Charging Station-Driver” Modeling: A Multiagent Deep Reinforcement Learning-Based Approach

Reinforcement learning-based multi-impulse rendezvous approach for satellite constellation reconfiguration

This paper presents a satellite constellation orbital reconfiguration method based on the intelligent multi-impulse rendezvous considering the impulse-magnitude limit. This problem is solved by two steps: the single-target rendezvous trajectory optimization and the multi-target rendezvous mission planning. For single target, the multi-impulse rendezvous problem is transformed into a Markov decision process and solved by a reinforcement learning algorithm. The rewards of the critic networks are used as estimation of fuel consumption, while the actions of the actor networks are used as initial values for further optimization. For multiple targets, an efficient Monte-Carlo tree search method is used to realize the global optimal solution of the satellite constellation reconfiguration problem. Finally, the numerical examples show that the proposed method can rapidly optimize the multi-impulse constellation reconfiguration problem.

DRL-Tomo: a deep reinforcement learning-based approach to augmented data generation for network tomography

Abstract Accurate and current comprehension of network status is crucial for efficient network management. Nevertheless, direct network measurement strategies entail substantial traffic overhead and demand intricate coordination among network entities, making them impractical. Network tomography, an indirect measurement approach, utilizes insights garnered from measured parts to deduce characteristics of the entire network. Past studies frequently depend on acquiring challenging-to-access information, such as the complete network topology or support from specialized protocols. Unfortunately, these constraints pose challenges in non-cooperative scenarios where obtaining such information is difficult. Recent endeavors pursue emancipating tomography from dependence on copious information, striving to predict unmeasured path performance using limited data. Nevertheless, the disparity between the measured data and actual performance has hindered the accuracy. In response, we introduce an innovative tomography framework named DRL-Tomo, designed to alleviate potential biases. DRL-Tomo initiates by generating augmented data through deep reinforcement learning, gradually approximating the genuine performance of unmeasured paths. Subsequently, a neural network model is trained using this augmented data, enabling precise inferences. Our experiments, encompassing both real-world and synthetic datasets, vividly demonstrate DRL-Tomo’s remarkable enhancement. Specifically, it achieves a substantial 10%–67% improvement in path delay prediction and an impressive 30%–98% enhancement in path loss rate prediction.

VR-HandNet: A Visually and Physically Plausible Hand Manipulation System in Virtual Reality.

This study aims to allow users to perform dexterous hand manipulation of objects in virtual environments with hand-held VR controllers. To this end, the VR controller is mapped to the virtual hand and the hand motions are dynamically synthesized when the virtual hand approaches an object. At each frame, given the information about the virtual hand, VR controller input, and hand-object spatial relations, the deep neural network determines the desired joint orientations of the virtual hand model in the next frame. The desired orientations are then converted into a set of torques acting on hand joints and applied to a physics simulation to determine the hand pose at the next frame. The deep neural network, named VR-HandNet, is trained with a reinforcement learning-based approach. Therefore, it can produce physically plausible hand motion since the trial-and-error training process can learn how the interaction between hand and object is performed under the environment that is simulated by a physics engine. Furthermore, we adopted an imitation learning paradigm to increase visual plausibility by mimicking the reference motion datasets. Through the ablation studies, we validated the proposed method is effectively constructed and successfully serves our design goal. A live demo is demonstrated in the supplementary video.

A Deep Reinforcement Learning Algorithm for Trajectory Planning of Swarm UAV Fulfilling Wildfire Reconnaissance

Wildfires have long been one of the critical environmental disasters that require a careful monitoring system. An intelligent system has the potential to both prevent/extinguish the fire and deliver urgent requirements postfire. In recent years, unmanned aerial vehicles (UAVs), with the ability to detect missions in high-risk areas, have been gaining increasing interest, particularly in forest fire monitoring. Taking a large-scale area involved in a fire into consideration, a single UAV is often insufficient to accomplish the task of covering the whole disaster zone. This poses the challenge of multi-UAVs optimum path planning with a key focus on limitations such as energy constraints and connectivity. To narrow down this issue, this paper proposes a deep reinforcement learning-based trajectory planning approach for multi-UAVs that permits UAVs to extract the required information within the disaster area on time. A target area is partitioned into several identical subareas in terms of size to enable UAVs to perform their patrol duties over the subareas. This subarea-based arrangement converts the issue of trajectory planning into allowing UAVs to frequently visit each subarea. Each subarea is initiated with a risk level by creating a fire risk map optimizing the UAV patrol route more precisely. Through a set of simulations conducted with a real trace of the dataset, the performance outcomes confirmed the superiority of the proposed idea.

Deep Reinforcement Learning-Based Approach With Varying-Scale Generalization for the Earth Observation Satellite Scheduling Problem Considering Resource Consumptions and Supplements

Deep Reinforcement Learning-Based Approach With Varying-Scale Generalization for the Earth Observation Satellite Scheduling Problem Considering Resource Consumptions and Supplements

Fall prediction, control, and recovery of quadruped robots

When legged robots perform complex tasks in unstructured environments, falls are inevitable due to unknown external disturbances. However, current research mainly focuses on the locomotion control of legged robots without falling. This paper proposes a comprehensive decision-making and control framework to address the falling over of quadruped robots. First, a capturability-based fall prediction algorithm is derived for planar single-contact and 3D multi-contact locomotion with a predefined gait sequence. For safe fall control, a novel contact-implicit trajectory optimization method is proposed to generate both state and input trajectories and contact mode sequences. Specifically, incorporating uncertainty into the system and terrain models enables mitigating the non-smoothness of contact dynamics while improving the robustness of the resulting trajectories. Furthermore, a model-free deep reinforcement learning-based approach is presented to achieve fall recovery after the robot completes a fall. Experimental results demonstrate that the proposed fall prediction algorithm accurately predicts robot falls with up to 95% accuracy approximately 395ms in advance. Compared to classical locomotion controllers, which often struggle to maintain balance under significant pushes or terrain perturbations, the presented framework can autonomously switch to the fall controller approximately 0.06s after the perturbation, effectively preventing falls or achieving recovery with a threefold reduction in touchdown impact velocity. These findings highlight the effectiveness of the proposed framework in enhancing the stability and safety of legged robots in unstructured environments.