Reinforcement Learning-based Approach Research Articles

Advancing AutoML: A Deep Dive into Neural Architecture Search (NAS) and Its Optimization Techniques

Abstract: Neural Architecture Search (NAS) is a pivotal technique in the field of automated machine learning (AutoML), enabling the automatic design of optimal neural network architectures. As deep learning models grow in complexity, NAS offers a scalable approach to improving model performance by exploring vast search spaces of potential architectures. In our research, we investigate the mathematical foundations and algorithms underpinning NAS, focusing on reinforcement learning-based, evolutionary, and gradient-based approaches. We provide mathematical proofs of convergence and efficiency for each method and analyze real-world applications, such as image classification and natural language processing (NLP). Through a comprehensive exploration of NAS, we aim to highlight its impact on AutoML and its potential to automate neural network design effectively while addressing challenges in computational cost and generalization.

Machine learning-based intelligent localization technique for channel classification in massive MIMO

Multiple-input multiple-output (MIMO) technology has been widely adopted in wireless communications, which enables the simultaneous transmission of multiple data streams via multiple transmitting and receiving antennas. In a MIMO system with non-line-of-sight (NLOS), transmitted signals are reflected by various obstacles along the path, reaching the antenna at different angles and times. In 5G networks, the NLOS problem is a major challenge for massive MIMO localization, significantly reducing positioning accuracy. In this work, an intelligent localization technique based on NLOS identification and mitigation is proposed to address this problem. In this solution, a Convolutional Neural Network (CNN) based hybrid Archimedes-based Salp Swarm Algorithm (HASSA) technique is proposed to detect NLOS or the line of sight (LOS) and estimate the location. The accuracy can be analyzed by considering the angle of arrival of signals, threshold-based time of arrival, and time difference of arrival from different antennas. A novel reinforcement learning-based optimization approach is used for the mitigation of NLOS in the radio wave propagation path, which in turn reduces the computational complexity. We use the Ensemble Deep Deterministic Policy Gradient-Based Approach (EDDPG)-based Honey Badger algorithm (HBA) for the aforementioned process. The simulation of this approach assesses diverse scenarios and considers different parameters, and the approach is compared with various state-of-the-art works. From the simulation results, our proposed approach can be used for the identification and detection of LOS and NLOS components and can precisely enhance the localization compared with other approaches.

Enhancing PV Hosting Capacity of Electricity Distribution Networks Using Deep Reinforcement Learning-Based Coordinated Voltage Control

Coordinated voltage control enables the active management of voltage levels throughout electricity distribution networks by leveraging the voltage support capabilities of existing grid-connected PV inverters. The efficient management of power flows and precise voltage regulation through coordinated voltage control schemes facilitate the increased adoption of rooftop PV systems and enhance the hosting capacity of electricity distribution networks. The research work presented in this paper proposes a coordinated voltage control scheme and evaluates the enhanced hosting capacity utilizing a deep reinforcement learning-based approach. A comparative analysis of the proposed algorithm is presented, and the performance is benchmarked against existing local voltage control schemes. The proposed coordinated voltage control scheme in this paper is evaluated using simulations on a real-world low-voltage electricity distribution network. The evaluation involves quasi-static time series power flow simulations for assessing performance. Furthermore, a discussion is presented that reflects on the strengths and limitations of the proposed scheme based on the results observed from the case study.

Gaussian process-based online sensor selection for source localization

This paper addresses the sensor selection problem for source localization within cyber–physical systems (CPSs). While recent machine learning and reinforcement learning approaches aim to optimize sensor selection and placement within the Area of Interest (AoI), their need for intensive data collection and training precludes online operation. Furthermore, these methods often require prior knowledge of the unknown source’s characteristics and lack adaptability to the dynamic nature of CPSs, leading to inefficiencies in unseen environments. This paper addresses these shortcomings using Gaussian process Optimization coupled with an active sensor selection mechanism to locate the unknown source within the AoI. The proposed approach first builds a probabilistic model of the environment, which is discretized into a grid, without prior training using a Gaussian Process surrogate model. Next, the model iteratively and systematically learns the underlying spatial phenomenon using Gaussian Process optimization. Concurrently, the approach selects a subset of sensors by optimizing a fitness function that advocates selecting informative and energy-efficient sensors. Next, the probabilistic model, having accurately learned the environment, directs the algorithm to the unknown source by identifying the cell with the highest likelihood of containing it. Finally, a peak refinement step is performed, which computes the exact location of the source within the designated cell. The proposed method’s efficacy is validated through experiments in radioactive source localization, validation studies, and adaptability assessments across various environments. In terms of quality of localization (QoL), it outperforms recent localization benchmarks, such as a reinforcement learning-based approach and DANS, by around 18% and 100%, respectively.

Clustering and Cooperative Guidance of Multiple Decoys for Defending a Naval Platform against Salvo Threats

The threat to naval platforms from missile systems is increasing due to recent advancements in radar seeker technology, which have significantly enhanced the accuracy and effectiveness of missile targeting. In scenarios where a naval platform with limited maneuverability faces salvo attacks, the importance of an effective defense strategy becomes crucial to ensuring the protection of the platform. In this study, we present a multi-agent reinforcement learning-based decoy deployment approach that employs six decoys to increase the survival likelihood of a naval platform against salvo missile strikes. Our approach entails separating the decoys into two clusters, each consisting of three decoys. Subsequently, every cluster is allocated to a related missile threat. This is accomplished by training the decoys with the multi-agent deep reinforcement learning algorithm. To compare the proposed approach across different algorithms, we use two distinct algorithms to train the decoys; multi-agent deep deterministic policy gradient (MADDPG) and multi-agent twin-delayed deep deterministic policy gradient (MATD3). Following training, the decoys learn to form groups and establish effective formation configurations within each group to ensure optimal coordination. We assess the proposed decoy deployment strategy using parameters including decoy deployment angle and maximum decoy speed. Our findings indicate that decoys positioned on the same side outperform those positioned on different sides relative to the target platform. In general, MATD3 performs slightly better than MADDPG. Decoys trained with MATD3 succeed in more successful formation configurations than those trained with the MADDPG method, which accounts for this enhancement.

Dynamic soft-kill weapon-target assignment in naval environments

One of the most significant threats faced by ships is anti-ship missiles. Nowadays, these missiles, equipped with diverse guidance systems, can locate their trajectory and attack the ship. Consequently, ships need to utilize their weapons to attempt to neutralize these threats. This article aims to develop dynamic assignment algorithms to assign a ship’s defensive soft-kill weapons to a set of incoming missiles, to minimize the average damage inflicted on the ship. To this end, initially, a binary linear programming model is developed to solve the static weapon-target assignment problem. Subsequently, a simulation–optimization algorithm and a reinforcement learning-based approach, grounded in the value iteration algorithm, are developed to solve the dynamic weapon-target assignment problem. To compare and evaluate the performance of the developed solution methods, we employ a set of randomly generated test instances. Computational results indicate that the reinforcement learning approach, due to its inherent foresight, outperforms the simulation–optimization approach in reducing the inflicted damages. However, in terms of CPU run time, the simulation–optimization approach is more efficient.

A Reinforcement Learning-Based Approach for Promoting Mental Health Using Multimodal Emotion Recognition

This research aims to enhance mental well-being by addressing symptoms of anxiety and depression through a personalized, culturally specific multimodal emotion prediction system. It employs an emotionally aware Reinforcement Learning (RL) agent to suggest tailored Cognitive Behavioral Therapy (CBT) activities. The study focuses on developing precise, individualized emotion prediction models using facial expressions, vocal tones, and text, and integrates these models with the RL agent for emotionally aware CBT recommendations. The mHealth approach combines deep learning models with RL, achieving accuracies of 72% for facial expressions, 73% for vocal tones, and 86% for text, all fine-tuned for the Sri Lankan context. Validation through real-world use and user feedback consistently demonstrated that each model exceeds 70% accuracy, fulfilling the objective of precise emotion prediction. A weighted algorithm was introduced to refine the emotion prediction experience and personalize forecasts across the three modalities to enhance mental well-being. The RL-enabled agent suggests CBT activities approved by mental health professionals, tailored based on predicted emotions, and delivered through the same mHealth application. The effectiveness of these interventions was assessed using the DASS-21 questionnaire, revealing significant reductions in depression scores (from 21.08 to 13.54) and anxiety scores (from 19.85 to 10.46) in the study group compared to the control group. The study concludes that integrating multimodal emotion prediction models with RL-based CBT suggestions positively impacts mental well-being and contributes to personalized mental health interventions.

Blockchain-Assisted Secure Energy Trading in Electricity Markets: A Tiny Deep Reinforcement Learning-Based Stackelberg Game Approach

Blockchain-Assisted Secure Energy Trading in Electricity Markets: A Tiny Deep Reinforcement Learning-Based Stackelberg Game Approach

RL-based mobile edge computing scheme for high reliability low latency services in UAV-aided IIoT networks

The prevailing adoption of Internet of Things paradigm is giving rise to a wide range of use cases in various vertical industries including remote health, industrial automation, and smart agriculture. However, the realization of such use cases is mainly challenged due to their stringent service requirements of high reliability and low latency. This challenge grows further when the service entails processing collected data for informed decision making. In this work, we consider a field of industrial Internet of Things devices that generate computational tasks and are covered by a nearby base station equipped with an edge server. The edge server offers fast processing to the devices’ tasks to help in meeting their latency requirement. Due to statistical wireless variability, the task data may not be correctly delivered in time for processing. To this end, we utilize an unmanned aerial vehicle as a supplemental edge server that tailors its trajectory and flies closer to the IIoT devices to ensure a highly reliable task delivery based on the given task reliability constraints. We formulate the problem as a Markov Decision Process, and propose a deep reinforcement learning-based approach using proximal policy optimization to optimize the unmanned aerial vehicle trajectory and scheduling devices to offload their data for processing. We present simulation results for various system scenarios to illustrate the effectiveness of the proposed solution as compared to several baseline approaches.

Electric Vehicle Charging Guidance Strategy Considering “Traffic Network-Charging Station-Driver” Modeling: A Multiagent Deep Reinforcement Learning-Based Approach

Electric Vehicle Charging Guidance Strategy Considering “Traffic Network-Charging Station-Driver” Modeling: A Multiagent Deep Reinforcement Learning-Based Approach

Reinforcement learning-based multi-impulse rendezvous approach for satellite constellation reconfiguration

This paper presents a satellite constellation orbital reconfiguration method based on the intelligent multi-impulse rendezvous considering the impulse-magnitude limit. This problem is solved by two steps: the single-target rendezvous trajectory optimization and the multi-target rendezvous mission planning. For single target, the multi-impulse rendezvous problem is transformed into a Markov decision process and solved by a reinforcement learning algorithm. The rewards of the critic networks are used as estimation of fuel consumption, while the actions of the actor networks are used as initial values for further optimization. For multiple targets, an efficient Monte-Carlo tree search method is used to realize the global optimal solution of the satellite constellation reconfiguration problem. Finally, the numerical examples show that the proposed method can rapidly optimize the multi-impulse constellation reconfiguration problem.

DRL-Tomo: a deep reinforcement learning-based approach to augmented data generation for network tomography

Abstract Accurate and current comprehension of network status is crucial for efficient network management. Nevertheless, direct network measurement strategies entail substantial traffic overhead and demand intricate coordination among network entities, making them impractical. Network tomography, an indirect measurement approach, utilizes insights garnered from measured parts to deduce characteristics of the entire network. Past studies frequently depend on acquiring challenging-to-access information, such as the complete network topology or support from specialized protocols. Unfortunately, these constraints pose challenges in non-cooperative scenarios where obtaining such information is difficult. Recent endeavors pursue emancipating tomography from dependence on copious information, striving to predict unmeasured path performance using limited data. Nevertheless, the disparity between the measured data and actual performance has hindered the accuracy. In response, we introduce an innovative tomography framework named DRL-Tomo, designed to alleviate potential biases. DRL-Tomo initiates by generating augmented data through deep reinforcement learning, gradually approximating the genuine performance of unmeasured paths. Subsequently, a neural network model is trained using this augmented data, enabling precise inferences. Our experiments, encompassing both real-world and synthetic datasets, vividly demonstrate DRL-Tomo’s remarkable enhancement. Specifically, it achieves a substantial 10%–67% improvement in path delay prediction and an impressive 30%–98% enhancement in path loss rate prediction.

VR-HandNet: A Visually and Physically Plausible Hand Manipulation System in Virtual Reality.

This study aims to allow users to perform dexterous hand manipulation of objects in virtual environments with hand-held VR controllers. To this end, the VR controller is mapped to the virtual hand and the hand motions are dynamically synthesized when the virtual hand approaches an object. At each frame, given the information about the virtual hand, VR controller input, and hand-object spatial relations, the deep neural network determines the desired joint orientations of the virtual hand model in the next frame. The desired orientations are then converted into a set of torques acting on hand joints and applied to a physics simulation to determine the hand pose at the next frame. The deep neural network, named VR-HandNet, is trained with a reinforcement learning-based approach. Therefore, it can produce physically plausible hand motion since the trial-and-error training process can learn how the interaction between hand and object is performed under the environment that is simulated by a physics engine. Furthermore, we adopted an imitation learning paradigm to increase visual plausibility by mimicking the reference motion datasets. Through the ablation studies, we validated the proposed method is effectively constructed and successfully serves our design goal. A live demo is demonstrated in the supplementary video.

A Deep Reinforcement Learning Algorithm for Trajectory Planning of Swarm UAV Fulfilling Wildfire Reconnaissance

Wildfires have long been one of the critical environmental disasters that require a careful monitoring system. An intelligent system has the potential to both prevent/extinguish the fire and deliver urgent requirements postfire. In recent years, unmanned aerial vehicles (UAVs), with the ability to detect missions in high-risk areas, have been gaining increasing interest, particularly in forest fire monitoring. Taking a large-scale area involved in a fire into consideration, a single UAV is often insufficient to accomplish the task of covering the whole disaster zone. This poses the challenge of multi-UAVs optimum path planning with a key focus on limitations such as energy constraints and connectivity. To narrow down this issue, this paper proposes a deep reinforcement learning-based trajectory planning approach for multi-UAVs that permits UAVs to extract the required information within the disaster area on time. A target area is partitioned into several identical subareas in terms of size to enable UAVs to perform their patrol duties over the subareas. This subarea-based arrangement converts the issue of trajectory planning into allowing UAVs to frequently visit each subarea. Each subarea is initiated with a risk level by creating a fire risk map optimizing the UAV patrol route more precisely. Through a set of simulations conducted with a real trace of the dataset, the performance outcomes confirmed the superiority of the proposed idea.

Deep Reinforcement Learning-Based Approach With Varying-Scale Generalization for the Earth Observation Satellite Scheduling Problem Considering Resource Consumptions and Supplements

Deep Reinforcement Learning-Based Approach With Varying-Scale Generalization for the Earth Observation Satellite Scheduling Problem Considering Resource Consumptions and Supplements

Fall prediction, control, and recovery of quadruped robots

When legged robots perform complex tasks in unstructured environments, falls are inevitable due to unknown external disturbances. However, current research mainly focuses on the locomotion control of legged robots without falling. This paper proposes a comprehensive decision-making and control framework to address the falling over of quadruped robots. First, a capturability-based fall prediction algorithm is derived for planar single-contact and 3D multi-contact locomotion with a predefined gait sequence. For safe fall control, a novel contact-implicit trajectory optimization method is proposed to generate both state and input trajectories and contact mode sequences. Specifically, incorporating uncertainty into the system and terrain models enables mitigating the non-smoothness of contact dynamics while improving the robustness of the resulting trajectories. Furthermore, a model-free deep reinforcement learning-based approach is presented to achieve fall recovery after the robot completes a fall. Experimental results demonstrate that the proposed fall prediction algorithm accurately predicts robot falls with up to 95% accuracy approximately 395ms in advance. Compared to classical locomotion controllers, which often struggle to maintain balance under significant pushes or terrain perturbations, the presented framework can autonomously switch to the fall controller approximately 0.06s after the perturbation, effectively preventing falls or achieving recovery with a threefold reduction in touchdown impact velocity. These findings highlight the effectiveness of the proposed framework in enhancing the stability and safety of legged robots in unstructured environments.

Joint Drone Access and LEO Satellite Backhaul for a Space–Air–Ground Integrated Network: A Multi-Agent Deep Reinforcement Learning-Based Approach

The space–air–ground integrated network can provide services to ground users in remote areas by utilizing high-altitude platform (HAP) drones to support stable user access and using low earth orbit (LEO) satellites to provide large-scale traffic backhaul. However, the rapid movement of LEO satellites requires dynamic maintenance of the matching relationship between LEO satellites and HAP drones. Additionally, different traffic types generated at HAP drones hold varying levels of values. Therefore, a tripartite matching problem among LEO satellites, HAP drones, and traffic types jointly considering multi-dimensional characteristics such as remaining visible time, channel condition, handover latency, and traffic storage capacity is formulated as mixed integer nonlinear programming to maximize the average transmitted traffic value. The traffic generation state for HAP drones is modeled as a mixture of stochasticity and determinism, which aligns with real-world scenarios, posing challenges for traditional optimization solvers. Thus, the original problem is decoupled into two independent sub-problems: traffic–drone matching and LEO–drone matching, which are addressed by mathematical simplification and multi-agent deep reinforcement learning with centralized training and decentralized execution, respectively. Simulation results verify the effectiveness and superiority of the proposed tripartite matching approach.

Online EVs Vehicle-to-Grid Scheduling Coordinated with Multi-Energy Microgrids: A Deep Reinforcement Learning-Based Approach

The integration of electric vehicles (EVs) into vehicle-to-grid (V2G) scheduling offers a promising opportunity to enhance the profitability of multi-energy microgrid operators (MMOs). MMOs aim to maximize their total profits by coordinating V2G scheduling and multi-energy flexible loads of end-users while adhering to operational constraints. However, scheduling V2G strategies online poses challenges due to uncertainties such as electricity prices and EV arrival/departure patterns. To address this, we propose an online V2G scheduling framework based on deep reinforcement learning (DRL) to optimize EV battery utilization in microgrids with different energy sources. Firstly, our approach proposes an online scheduling model that integrates the management of V2G and multi-energy flexible demands, modeled as a Markov Decision Process (MDP) with an unknown transition. Secondly, a DRL-based Soft Actor-Critic (SAC) algorithm is utilized to efficiently train neural networks and dynamically schedule EV charging and discharging activities in response to real-time grid conditions and energy demand patterns. Extensive simulations are conducted in case studies to testify to the effectiveness of our proposed approach. The overall results validate the efficacy of the DRL-based online V2G scheduling framework, highlighting its potential to drive profitability and sustainability in multi-energy microgrid operations.

Multi-objective reinforcement learning-based approach for pressurized water reactor optimization

A novel method, the Pareto Envelope Augmented with Reinforcement Learning (PEARL), has been developed to address the challenges posed by multi-objective problems, particularly in the field of engineering where the evaluation of candidate solutions can be time-consuming. PEARL distinguishes itself from traditional policy-based multi-objective Reinforcement Learning methods by learning a single policy, eliminating the need for multiple neural networks to independently solve simpler sub-problems. Several versions inspired from deep learning and evolutionary techniques have been crafted, catering to both unconstrained and constrained problem domains. Curriculum Learning (CL) is harnessed to effectively manage constraints in these versions. PEARL’s performance is first evaluated on classical multi-objective benchmarks. Additionally, it is tested on two practical PWR core Loading Pattern (LP) optimization problems to showcase its real-world applicability. The first problem involves optimizing the Cycle length (LC) and the rod-integrated peaking factor (FΔh) as the primary objectives, while the second problem incorporates the mean average enrichment as an additional objective. Furthermore, PEARL addresses three types of constraints related to boron concentration (Cb), peak pin burnup (Bumax), and peak pin power (Fq). The results are systematically compared against conventional approaches from stochastic optimization. Notably, PEARL, specifically the PEAL-NdS ( ▪ ) variant, efficiently uncovers a Pareto front without necessitating additional efforts from the algorithm designer, as opposed to a single optimization with scaled objectives. It also outperforms the classical approach across multiple performance metrics, including the Hyper-volume. Future works will encompass a sensitivity analysis of hyper-parameters with statistical analysis to optimize the application of PEARL and extend it to more intricate problems.

A Deep Reinforcement Learning-Based Preemptive Approach for Cost-Aware Cloud Job Scheduling

A Deep Reinforcement Learning-Based Preemptive Approach for Cost-Aware Cloud Job Scheduling