Reward Reinforcement Learning Research Articles

Application of Reinforcement Learning in Controlling Quadrotor UAV Flight Actions

Most literature has extensively discussed reinforcement learning (RL) for controlling rotorcraft drones during flight for traversal tasks. However, most studies lack adequate details regarding the design of reward and punishment mechanisms, and there is a limited exploration of the feasibility of applying reinforcement learning in actual flight control following simulation experiments. Consequently, this study focuses on the exploration of reward and punishment design and state input for RL. The simulation environment is constructed using AirSim and Unreal Engine, with onboard camera footage serving as the state input for reinforcement learning. The research investigates three RL algorithms suitable for discrete action training. The Deep Q Network (DQN), Advantage Actor–Critic (A2C), and Proximal Policy Optimization (PPO) were combined with three different reward and punishment design mechanisms for training and testing. The results indicate that employing the PPO algorithm along with a continuous return method as the reward mechanism allows for effective convergence during the training process, achieving a target traversal rate of 71% in the testing environment. Furthermore, this study proposes integrating the YOLOv7-tiny object detection (OD) system to assess the applicability of reinforcement learning in real-world settings. Unifying the state inputs of simulated and OD environments and replacing the original simulated image inputs with a maximum dual-target approach, the experimental simulation achieved a target traversal rate of 52% ultimately. In summary, this research formulates a set of logical frameworks for an RL reward and punishment design deployed with real-time Yolo’s OD implementation synergized as a useful aid for related RL studies.

Reinforcement learning-based method for type B aortic dissection localization

In the segmentation of aortic dissection, there are issues such as low contrast between the aortic dissection and surrounding organs and vessels, significant differences in dissection morphology, and high background noise. To address these issues, this paper proposed a reinforcement learning-based method for type B aortic dissection localization. With the assistance of a two-stage segmentation model, the deep reinforcement learning was utilized to perform the first-stage aortic dissection localization task, ensuring the integrity of the localization target. In the second stage, the coarse segmentation results from the first stage were used as input to obtain refined segmentation results. To improve the recall rate of the first-stage segmentation results and include the segmentation target more completely in the localization results, this paper designed a reinforcement learning reward function based on the direction of recall changes. Additionally, the localization window was separated from the field of view window to reduce the occurrence of segmentation target loss. Unet, TransUnet, SwinUnet, and MT-Unet were selected as benchmark segmentation models. Through experiments, it was verified that the majority of the metrics in the two-stage segmentation process of this paper performed better than the benchmark results. Specifically, the Dice index improved by 1.34%, 0.89%, 27.66%, and 7.37% for each respective model. In conclusion, by incorporating the type B aortic dissection localization method proposed in this paper into the segmentation process, the overall segmentation accuracy is improved compared to the benchmark models. The improvement is particularly significant for models with poorer segmentation performance.

Reinforcement Learning-Based Optimal Hull Form Design with Variations in Fore and Aft Parts

Abstract With recent advancements in artificial intelligence technology, various studies are being conducted in the shipbuilding industry. Traditionally, hull form variation methods have relied on the intuition and expertise of designers, leading to inconsistent results and unintended changes in the ship's main dimensions depending on the designer's competence. Moreover, the iterative process of design variation and analysis to derive the optimal hull form is both costly and time-consuming. To address these issues, this study proposes an optimal hull design technique utilizing reinforcement learning, a type of unsupervised learning in machine learning. Reinforcement learning allows the model to learn from past policies by recording and accumulating the rewards associated with various actions taken by an agent in a specific environment. In this study, after calculating the main parameters of the ship, the agent defines a state representing hull information and performs local transformations of the bow and stern. The reward of reinforcement learning is defined as the change in total resistance due to the hull deformation, constrained by limiting the tolerance of the ship's prismatic coefficient (CP) and longitudinal center of buoyancy (LCB). In this study, the problem is solved by comparing the proximal policy optimization (PPO) algorithm and the deep deterministic policy gradient (DDPG) algorithm to find the best deep reinforcement learning (DRL) model for the hull optimization problem. The results were compared with the genetic algorithm and speed-constrained multi-objective particle swarm optimization (SMPSO), and the optimal hull resistance values were less different, but the time of the reinforcement learning model was five times shorter.

Bayesian optimized deep Q-network for diagnosing mine ventilation systems windage alteration fault targeting imbalanced data

Fault diagnosis of mine ventilation system is of great significance for mine safety production. Traditional machine learning algorithms have been widely applied in the field of mine ventilation systems windage alteration faults (WAFs) diagnosis, but these algorithms have poor intelligence and weak generalization ability. Therefore, this research constructs a WAFs diagnosis model (WAFs-BODQN) based on Bayesian optimized (BO) deep Q-network (DQN). Meanwhile, in order to solve the problem of poor performance of intelligent models when the dataset is imbalanced, a reinforcement learning reward function (KMeans-SDF) was designed, which combines the KMeans++ algorithm and spatial distance function (SDF). The WAFs-BODQN model with KMeans-SDF reward function has fault location diagnosis accuracies of 98.75 %, 97.50 %, and 95.00 % for mild, moderate, and extreme imbalanced datasets UB1, UB2, and UB3, respectively. This effectively avoids the "eccentricity" phenomenon of the intelligent model on majority class and achieves accurate prediction of fault locations. The accuracy of fault location diagnosis using the WAFs-BODQN model in simulation experiments and on-site empirical applications is higher than 95 %, demonstrating the intelligence, generalization, and feasibility of WAFs-BODQN model in the field of WAFs diagnosis in mine ventilation systems. By comparing the results with traditional machine learning and deep learning fault diagnosis models, it has been proven that the WAFs-BODQN model has advantages in intelligence and generalization ability. It is expected to establish a universal architecture for diagnosing WAFs, providing theoretical guidance and technical support for achieving intelligent mine ventilation.

Optimization framework of laser oscillation welding based on a deep predictive reward reinforcement learning net

Optimization framework of laser oscillation welding based on a deep predictive reward reinforcement learning net

College students with depression symptom are more sensitive to task difficulty in reinforcement learning

BackgroundDepression is usually characterized by impairments in reward function, and shows altered motivation to reward in reinforcement learning. This study further explored whether task difficulty affects reinforcement learning in college students with and without depression symptom. MethodsThe depression symptom group (20) and the no depression symptom group (26) completed a probabilistic reward learning task with low, medium, and high difficulty levels, in which task the response bias to reward and the discriminability of reward were analyzed. Additionally, electrophysiological responses to reward and loss feedback were recorded and analyzed while they performed a simple gambling task. ResultsThe depression symptom group showed more response bias to reward than the no depression symptom group when the task was easy and then exhibited more quickly decrease in response bias to reward as task difficulty increased. The no depression symptom group showed a decrease in response bias only in the high-difficulty condition. Further regression analyses showed that, the Feedback-related negativity (FRN) and theta oscillation could predict response bias change in the low-difficulty condition, the FRN and oscillations of theta and delta could predict response bias change in the medium and high-difficulty conditions. LimitationsThe electrophysiological responses to loss and reward were not recorded in the same task as the reinforcement learning behaviors. ConclusionsCollege students with depression symptom are more sensitive to task difficulty during reinforcement learning. The FRN, and oscillations of theta and delta could predict reward leaning behavior.

Real-time stochastic flexible flow shop scheduling in a credit factory with model-based reinforcement learning

There has been a significant increase in consumer credit worldwide in recent years. The scheduling of jobs in credit factories is essential for speeding up the loan application process, which can improve the efficiency of credit factories. In this study, we propose a reinforcement learning approach for addressing the scheduling problem in credit factories, which is a stochastic flexible flow shop scheduling problem (SFFSP). First, we propose a mathematical model for the credit factory stochastic flexible flow shop scheduling problem, which abstracts the decision-making process as a semi-Markov process. Then, a reinforcement learning reward mechanism is designed based on the proposed mathematical model. After that, a self-attention neural network is used to extract state information from global and local multidimensional data, enabling each decision to consider the state of the entire process and make a decision that aligns with the global goal. Meanwhile, Monte Carlo Tree Search (MCTS) is utilised to enhance the training effect and sample utilisation of reinforcement learning. Finally, we conduct extensive experiments and demonstrate that our method achieves better performance for SFFSP in credit factories compared to other approaches.

An online hyper‐volume action bounding approach for accelerating the process of deep reinforcement learning from multiple controllers

AbstractThis paper fuses ideas from reinforcement learning (RL), Learning from Demonstration (LfD), and Ensemble Learning into a single paradigm. Knowledge from a mixture of control algorithms (experts) are used to constrain the action space of the agent, enabling faster RL refining of a control policy, by avoiding unnecessary explorative actions. Domain‐specific knowledge of each expert is exploited. However, the resulting policy is robust against errors of individual experts, since it is refined by a RL reward function without copying any particular demonstration. Our method has the potential to supplement existing RLfD methods when multiple algorithmic approaches are available to function as experts, specifically in tasks involving continuous action spaces. We illustrate our method in the context of a visual servoing (VS) task, in which a 7‐DoF robot arm is controlled to maintain a desired pose relative to a target object. We explore four methods for bounding the actions of the RL agent during training. These methods include using a hypercube and convex hull with modified loss functions, ignoring actions outside the convex hull, and projecting actions onto the convex hull. We compare the training progress of each method using expert demonstrators, employing one expert demonstrator with the DAgger algorithm, and without using any demonstrators. Our experiments show that using the convex hull with a modified loss function not only accelerates learning but also provides the most optimal solution compared with other approaches. Furthermore, we demonstrate faster VS error convergence while maintaining higher manipulability of the arm, compared with classical image‐based VS, position‐based VS, and hybrid‐decoupled VS.

Multi-visual modality micro drone-based structural damage detection

Accurate detection and resilience of object detectors in structural damage detection are important in ensuring the continuous use of civil infrastructure. However, achieving robustness in object detectors remains a persistent challenge, impacting their ability to generalize effectively. This study proposes DetectorX, a robust framework for structural damage detection coupled with a micro drone. DetectorX addresses the challenges of object detector robustness by incorporating two innovative modules: a stem block and a spiral pooling technique. The stem block introduces a dynamic visual modality by leveraging the outputs of two Deep Convolutional Neural Network (DCNN) models. The framework employs the proposed event-based reward reinforcement learning to constrain the actions of a parent and child DCNN model leading to a reward. This results in the induction of two dynamic visual modalities alongside the Red, Green, and Blue (RGB) data. This enhancement significantly augments DetectorX's perception and adaptability in diverse environmental situations. Further, a spiral pooling technique, an online image augmentation method, strengthens the framework by increasing feature representations by concatenating spiraled and average/max pooled features. In three extensive experiments: (1) comparative and (2) robustness, which use the Pacific Earthquake Engineering Research Hub ImageNet dataset, and (3) field-experiment, DetectorX performed satisfactorily across varying metrics, including precision (0.88), recall (0.84), average precision (0.91), mean average precision (0.76), and mean average recall (0.73), compared to the competing detectors including You Only Look Once X-medium (YOLOX-m) and others. The study's findings indicate that DetectorX can provide satisfactory results and demonstrate resilience in challenging environments.

Hierarchical medical image report adversarial generation with hybrid discriminator

Background and objectivesGenerating coherent reports from medical images is an important task for reducing doctors' workload. Unlike traditional image captioning tasks, the task of medical image report generation faces more challenges. Current models for generating reports from medical images often fail to characterize some abnormal findings, and some models generate reports with low quality. In this study, we propose a model to generate high-quality reports from medical images. MethodsIn this paper, we propose a model called Hybrid Discriminator Generative Adversarial Network (HDGAN), which combines Generative Adversarial Network (GAN) with Reinforcement Learning (RL). The HDGAN model consists of a generator, a one-sentence discriminator, and a one-word discriminator. Specifically, the RL reward signals are judged on the one-sentence discriminator and one-word discriminator separately. The one-sentence discriminator can better learn sentence-level structural information, while the one-word discriminator can learn word diversity information effectively. ResultsOur approach performs better on the IU-X-ray and COV-CTR datasets than the baseline models. For the ROUGE metric, our method outperforms the state-of-the-art model by 0.36 on the IU-X-ray, 0.06 on the MIMIC-CXR and 0.156 on the COV-CTR. ConclusionsThe compositional framework we proposed can generate more accurate medical image reports at different levels.

Deep Curious Feature Selection: A Recurrent, Intrinsic-Reward Reinforcement Learning Approach to Feature Selection

Feature selection (FS) is an important step in the process of building machine learning-based models. The goal of the feature selection step is to find a small subset of features that will provide good prediction results by removing noisy, irrelevant, or repetitive features. Commonly used Wrapper methods use a machine learning model as a black box and its performance as the goal function for evaluating different features' subsets and selecting the best one. To avoid examining all possible subsets (an NP-hard problem), search algorithms are used to find the subsets to be examined, in a heuristic way. As exhaustive search methods are computationally complicated, most methods use simple and greedy search methods that yield only locally optimal results and are not sensitive to possible features interactions, which means that a feature may be chosen at the expense of two others that are more informative together. We analyze the problem of searching the features subset space with reference to two dimensions, namely, memory of past selected features subset and future selected features. We propose a new wrapper feature selection method based on the deep artificial curiosity framework, which implements intrinsic reward reinforcement learning, with long short-term memory unit (LSTM). This novel algorithm integrates these two elements of memory and future step. We show that our method, called the Deep Curious Feature Selection (DCFS) algorithm, deals with feature interactions, and provides a feature subset which improves the accuracy of learning models on artificial and real datasets.

Realization of a Human-like Gait for a Bipedal Robot Based on Gait Analysis

There are many studies analyzing human motion. However, we do not yet fully understand the mechanisms of our own bodies. We believe that mimicking human motion and function using a robot will help us to deepen our understanding of humans. Therefore, we focus on the characteristics of the human gait, and the goal is to realize a human-like bipedal gait that lands on its heels and takes off from its toes. In this study, we focus on kinematic synergy (planar covariation) in the lower limbs as a characteristic gait seen in humans. Planar covariation is that elevation angles at the thigh, shank, and foot in the sagittal plane are plotted on one plane when the angular data are plotted on the three axes. We propose this feature as a reward for reinforcement learning. By introducing this reward, the bipedal robot achieved a human-like bipedal gait in which the robot lands on its heels and takes off from its toes. We also compared the learning results with those obtained when this feature was not used. The results suggest that planar covariation is one factor that characterizes a human-like gait.

Path planning of substation inspection robot based on high-precision positioning and navigation technology

Abstract Outdoor substation is an important part of power system. Substation inspection robot based on intelligent autonomous inspection system has become the research focus of substation unmanned inspection. In order to improve the positioning accuracy and speed of the system, a high-precision positioning algorithm of transformer detection robot is proposed in this paper. Tikhonov regularization is used to correct the pathological problem of the localization algorithm model. The observation amount of the receiver is increased by using four signals of a single base station with double frequency and double antenna, and the position is solved by using single difference carrier phase observation and the integer ambiguity is fixed. The input–output mapping of the neural network is designed according to the information acquisition and two-wheel angular velocity control of the detection robot. Using the hyperbolic tangent function as the activation function of MLP neural network, the MLP neural network with 32 neurons in each of the three hidden layers is determined. By optimizing reinforcement learning reward function, adding scoring rules, and reward parameters, this paper carries out the following simulation exploration work. The high-precision positioning algorithm of transformer inspection robot is compared with the existing algorithm, and the superiority of the algorithm is verified. The basic motion ability of the robot installed with the system was tested.

An adaptive intelligent routing algorithm based on deep reinforcement learning

A mobile ad hoc network (MANET) is a self-organizing network composed of multiple mobile nodes. These nodes achieve distributed communication and wireless resource sharing through autonomous collaboration. However, typical routing protocols for MANETs suffer from uneven network load, low network throughput, poor anti-destruction performance, and low transmission reliability. In order to meet the routing protocol characteristics of distributed adaptability, reliability and anti-destruction, balance of load and energy consumption and guarantee of high QoS in MANETs, this paper proposes an adaptive intelligent routing algorithm based on the PER-D3QN model of deep reinforcement learning. To achieve better routing strategy selection, the routing problem is first modeled, a distributed multi-agent reinforcement learning routing framework is designed, and a deep reinforcement learning reward function considering multiple factors is used. Next, the implementation process of the adaptive intelligent routing algorithm using the PER-D3QN model is described in detail. Finally, a mobile self-organizing network environment was constructed based on Networkx, and corresponding data was collected to simulate the algorithm. The simulation results show that the adaptive intelligent routing algorithm proposed in this paper can balance the energy consumption of the network and the load of nodes, its average remaining energy has increased by more than 10% and PL is reduced by at least 10% compared to other algorithms, while guaranteeing high QoS performance of routing, that is, reducing the average end-to-end delay of data packet transmission by over 22% compared to other algorithms and maintaining a high packet delivery ratio.

Longitudinal data and a semantic similarity reward for chest X-ray report generation

Radiologists face high burnout rates, partially due to the increasing volume of Chest X-rays (CXRs) requiring interpretation and reporting. Automated CXR report generation holds promise for reducing this burden and improving patient care. While current models show potential, their diagnostic accuracy is limited. Our proposed CXR report generator integrates elements of the radiologist workflow and introduces a novel reward for reinforcement learning. Our approach leverages longitudinal data from a patient’s prior CXR study and effectively handles cases where no prior study exists, thus mirroring the radiologist’s workflow. In contrast, existing models typically lack this flexibility, often requiring prior studies for the model to function optimally. Our approach also incorporates all CXRs from a patient’s study and distinguishes between report sections through section embeddings. Our reward for reinforcement learning leverages CXR-BERT, which forces our model to learn the clinical semantics of radiology reporting. We conduct experiments on publicly available datasets — MIMIC-CXR and Open-i IU X-ray — with metrics shown to more closely correlate with radiologists’ assessment of reporting. Results from our study demonstrate that the proposed model generates reports that are more aligned with radiologists’ reports than state-of-the-art models, such as those utilising large language models, reinforcement learning, and multi-task learning. The proposed model improves the diagnostic accuracy of CXR report generation, which could one day reduce radiologists’ workload and enhance patient care. Our Hugging Face checkpoint (https://huggingface.co/aehrc/cxrmate) and code (https://github.com/aehrc/cxrmate) are publicly available.

Sliding principal component and dynamic reward reinforcement learning based IIoT attack detection

The Internet of Things (IoT) involves the gathering of all those devices that connect to the Internet with the purpose of collecting and sharing data. The application of IoT in the different sectors, including health, industry has also picked up the threads to augment over the past few years. The IoT and, by integrity, the IIoT, are found to be highly susceptible to different types of threats and attacks owing to the networks nature that in turn leads to even poor outcomes (i.e., increasing error rate). Hence, it is critical to design attack detection systems that can provide the security of IIoT networks. To overcome this research work of IIoT attack detection in large amount of evolutions is failed to determine the certain attacks resulting in a minimum detection performance, reinforcement learning-based attack detection method called sliding principal component and dynamic reward reinforcement learning (SPC–DRRL) for detecting various IIoT network attacks is introduced. In the first stage of this research methodology, preprocessing of raw TON_IoT dataset is performed by employing min–max normalization scaling function to obtain normalized values with same scale. Next, with the processed sample data as output, to extract data from multi-sources (i.e., different service profiles from the dataset), a robust log likelihood sliding principal component-based feature extraction algorithm is applied with an arbitrary size sliding window to extract computationally-efficient features. Finally, dynamic reward reinforcement learning-based IIoT attack detection model is presented to control the error rate involved in the design. Here, with the design of dynamic reward function and introducing incident repository that not only generates the reward function in an arbitrary fashion but also stores the action results in the incident repository for the next training, therefore reducing the attack detection error rate. Moreover, an IIoT attack detection system based on SPC–DRRL is constructed. Finally, we verify the algorithm on the ToN_IoT dataset of University of New South Wales Australia. The experimental results show that the IIoT attack detection time and overhead along with the error rate are reduced considerably with higher accuracy than that of traditional reinforcement learning methods.

A Novel Energy Management Method for Networked Multi-Energy Microgrids Based on Improved DQN

The networked Multi-Energy Microgrids (MEMGs) can significantly improve the economic efficiency of operation through energy cooperation and complementarity. However, the complex energy trading mechanism and multi-energy coupling decision process present various challenges for the optimal operation of MEMGs, such as the large decision space and the difficulty of converging to the global optimal solution. This study presents a novel energy management method for MEMG based on an improved Deep Q-network (IDQN) to solve these problems. Firstly, the equivalent model of an external interactive environment is constructed for each MEMG by using a novel Kriging surrogate enhanced Gate Recurrent Unit-Temporal Convolutional Network (GRU-TCN), which requires only the accessible energy exchange data and the public and independent information as input. The computational complexity of the reinforcement learning reward function is reduced accordingly by using this equivalent model. Subsequently, an improved k-crossover sampling strategy with reduced access frequency of the low reward actions is proposed to replace the traditional.-greedy strategy to further improve the original DQN. This effectively addresses the problem of inefficient exploration in a large-scale action space. Lastly, the proposed method is verified using a test system with three MEMGs. The result demonstrates that the proposed IDQN method presents better convergence and stability than the traditional DQN, and presents better energy management and operational efficiency.

Automated vision-based structural health inspection and assessment for post-construction civil infrastructure

In the search to enhance the effectiveness and efficiency of structural health monitoring for preserving safety and integrity in civil infrastructure, the search for innovative technologies is of critical importance. This paper introduces ExpoDet, a comprehensive framework designed for autonomous health inspection and infrastructure assessment. ExpoDet features a multi-detection detector, autonomous navigation for micro aerial vehicles facilitated through secondary reward reinforcement learning, and a damage aggregation scheme for autonomous health assessment following detections. Moreover, it presents an attention module called EEAM+ that introduces dynamic feature orientation and significantly enhances the capabilities of ExpoDet. ExpoDet is extensively tested and evaluated in both offsite and field test experiments. Comparisons with several state-of-the-art object detectors coupled with attention modules show an average improvement of approximately 3% across various evaluation metrics.

Local Path Planning via Improved Fuzzy and Q(λ)-learning Algorithms for the Mobile Robot

<p>With the complexity of the robot operating environment increases, there becoming higher demands on the optimal path planning for robots. Most of the path planning is performed in known environments and static models. However, there are still challenges for robots to perform path planning in complex unknown or dynamic environments, which will suffer from deadlock problems and obstacle avoidance failures. Reinforcement learning (RL) can help fuzzy algorithm to optimize the strategy. However, the difficulty of designing the rewards in RL makes the algorithm require a large number of samples to learn the strategy, resulting in computational complexity. To solve these problems, a new local path planning based on the improved fuzzy and Q(λ)-learning algorithms is proposed, aiming to plan the shortest path and avoid obstacles. For solving the problems of breaking through and avoiding obstacles, a fuzzy controller is designed. The distance of nearest obstacle in front of the mobile robot and the distance between the obstacles in the two breakout directions are regarded as the two inputs for this controller. And the two fuzzy quantities of the mobile robot’s running angle and the safe step length are outputted. In the path planning, the Q(λ)-learning algorithm are used to optimize the weights of the running angle and the safe step, obtaining a more accurate robot position and speeding up path planning efficiency. Furthermore, to solve the overlap problems among the starting point, end point, and obstacles, a safer running environment is designed considering radiuses of these objects. Besides, the mobile robot breakout scheme and sustainable obstacle avoidance scheme are designed to solve the deadlock problem and “large obstacle” avoidance problem, respectively. Simulation results in the sparse and complex operating environment show that our proposed algorithm can plan a relatively optimal and safe path, improving the success rate of path planning.</p> <p> </p>

Smart City Distributed Trade Economy Dynamic Scheduling of E-Commerce Platform Based on Reinforcement Learning Algorithm

Abstract In order to improve the reliability and economy of decentralized trade economy dynamic scheduling on e-Commerce platforms and shorten the running time of decentralized trade economy dynamic scheduling on e-Commerce platforms, a decentralized trade economy dynamic scheduling method based on the reinforcement learning algorithm is proposed. In this paper, we analyze the basic theory of the reinforcement learning algorithm, study the Q-learning algorithm, build a neural network to fit the value model, and initialize the reinforcement learning algorithm. With Markov decision process as the framework model, the optimal state behavior value function is updated by using the modeless discounted reward reinforcement learning algorithm Q-learning as the value iteration method. Gibbs distribution is used to construct exploratory random strategies to select behaviors with probability. Using the reinforcement learning algorithm and the three-layer feedforward neural network as the approximator of the state behavior value function, this paper studies the generalization of the value function faced by the decentralized trade economy dynamic scheduling of e-Commerce platforms and realizes the decentralized trade economy dynamic scheduling of e-Commerce platforms. The experimental results show that the proposed method can effectively improve the reliability and economy of the decentralized trade economy dynamic scheduling of e-Commerce platforms.