Reward Reinforcement Learning Research Articles

Optimization framework of laser oscillation welding based on a deep predictive reward reinforcement learning net

Optimization framework of laser oscillation welding based on a deep predictive reward reinforcement learning net

College students with depression symptom are more sensitive to task difficulty in reinforcement learning

Backgrounds: Depression is usually characterized by impairments in reward function, and shows altered motivation to reward in reinforcement learning. This study further explored whether task difficulty affects reinforcement learning in college students with and without depression symptom. MethodsThe depression symptom group (20) and the no depression symptom group (26) completed a probabilistic reward learning task with low, medium, and high difficulty levels, in which task the response bias to reward and the discriminability of reward were analyzed. Additionally, electrophysiological responses to reward and loss feedback were recorded and analyzed while they performed a simple gambling task. ResultsThe depression symptom group showed more response bias to reward than the no depression symptom group when the task was easy and then exhibited more quickly decrease in response bias to reward as task difficulty increased. The no depression symptom group showed a decrease in response bias only in the high-difficulty condition. Further regression analyses showed that, the Feedback-related negativity (FRN) and theta oscillation could predict response bias change in the low-difficulty condition, the FRN and oscillations of theta and delta could predict response bias change in the medium and high-difficulty conditions. LimitationsThe electrophysiological responses to loss and reward were not recorded in the same task as the reinforcement learning behaviors. ConclusionsCollege students with depression symptom are more sensitive to task difficulty during reinforcement learning. The FRN, and oscillations of theta and delta could predict reward leaning behavior.

Real-time stochastic flexible flow shop scheduling in a credit factory with model-based reinforcement learning

There has been a significant increase in consumer credit worldwide in recent years. The scheduling of jobs in credit factories is essential for speeding up the loan application process, which can improve the efficiency of credit factories. In this study, we propose a reinforcement learning approach for addressing the scheduling problem in credit factories, which is a stochastic flexible flow shop scheduling problem (SFFSP). First, we propose a mathematical model for the credit factory stochastic flexible flow shop scheduling problem, which abstracts the decision-making process as a semi-Markov process. Then, a reinforcement learning reward mechanism is designed based on the proposed mathematical model. After that, a self-attention neural network is used to extract state information from global and local multidimensional data, enabling each decision to consider the state of the entire process and make a decision that aligns with the global goal. Meanwhile, Monte Carlo Tree Search (MCTS) is utilised to enhance the training effect and sample utilisation of reinforcement learning. Finally, we conduct extensive experiments and demonstrate that our method achieves better performance for SFFSP in credit factories compared to other approaches.

An online hyper‐volume action bounding approach for accelerating the process of deep reinforcement learning from multiple controllers

AbstractThis paper fuses ideas from reinforcement learning (RL), Learning from Demonstration (LfD), and Ensemble Learning into a single paradigm. Knowledge from a mixture of control algorithms (experts) are used to constrain the action space of the agent, enabling faster RL refining of a control policy, by avoiding unnecessary explorative actions. Domain‐specific knowledge of each expert is exploited. However, the resulting policy is robust against errors of individual experts, since it is refined by a RL reward function without copying any particular demonstration. Our method has the potential to supplement existing RLfD methods when multiple algorithmic approaches are available to function as experts, specifically in tasks involving continuous action spaces. We illustrate our method in the context of a visual servoing (VS) task, in which a 7‐DoF robot arm is controlled to maintain a desired pose relative to a target object. We explore four methods for bounding the actions of the RL agent during training. These methods include using a hypercube and convex hull with modified loss functions, ignoring actions outside the convex hull, and projecting actions onto the convex hull. We compare the training progress of each method using expert demonstrators, employing one expert demonstrator with the DAgger algorithm, and without using any demonstrators. Our experiments show that using the convex hull with a modified loss function not only accelerates learning but also provides the most optimal solution compared with other approaches. Furthermore, we demonstrate faster VS error convergence while maintaining higher manipulability of the arm, compared with classical image‐based VS, position‐based VS, and hybrid‐decoupled VS.

Multi-visual modality micro drone-based structural damage detection

Accurate detection and resilience of object detectors in structural damage detection are important in ensuring the continuous use of civil infrastructure. However, achieving robustness in object detectors remains a persistent challenge, impacting their ability to generalize effectively. This study proposes DetectorX, a robust framework for structural damage detection coupled with a micro drone. DetectorX addresses the challenges of object detector robustness by incorporating two innovative modules: a stem block and a spiral pooling technique. The stem block introduces a dynamic visual modality by leveraging the outputs of two Deep Convolutional Neural Network (DCNN) models. The framework employs the proposed event-based reward reinforcement learning to constrain the actions of a parent and child DCNN model leading to a reward. This results in the induction of two dynamic visual modalities alongside the Red, Green, and Blue (RGB) data. This enhancement significantly augments DetectorX's perception and adaptability in diverse environmental situations. Further, a spiral pooling technique, an online image augmentation method, strengthens the framework by increasing feature representations by concatenating spiraled and average/max pooled features. In three extensive experiments: (1) comparative and (2) robustness, which use the Pacific Earthquake Engineering Research Hub ImageNet dataset, and (3) field-experiment, DetectorX performed satisfactorily across varying metrics, including precision (0.88), recall (0.84), average precision (0.91), mean average precision (0.76), and mean average recall (0.73), compared to the competing detectors including You Only Look Once X-medium (YOLOX-m) and others. The study's findings indicate that DetectorX can provide satisfactory results and demonstrate resilience in challenging environments.

Hierarchical medical image report adversarial generation with hybrid discriminator

Background and objectivesGenerating coherent reports from medical images is an important task for reducing doctors' workload. Unlike traditional image captioning tasks, the task of medical image report generation faces more challenges. Current models for generating reports from medical images often fail to characterize some abnormal findings, and some models generate reports with low quality. In this study, we propose a model to generate high-quality reports from medical images. MethodsIn this paper, we propose a model called Hybrid Discriminator Generative Adversarial Network (HDGAN), which combines Generative Adversarial Network (GAN) with Reinforcement Learning (RL). The HDGAN model consists of a generator, a one-sentence discriminator, and a one-word discriminator. Specifically, the RL reward signals are judged on the one-sentence discriminator and one-word discriminator separately. The one-sentence discriminator can better learn sentence-level structural information, while the one-word discriminator can learn word diversity information effectively. ResultsOur approach performs better on the IU-X-ray and COV-CTR datasets than the baseline models. For the ROUGE metric, our method outperforms the state-of-the-art model by 0.36 on the IU-X-ray, 0.06 on the MIMIC-CXR and 0.156 on the COV-CTR. ConclusionsThe compositional framework we proposed can generate more accurate medical image reports at different levels.

Deep Curious Feature Selection: A Recurrent, Intrinsic-Reward Reinforcement Learning Approach to Feature Selection

Feature selection (FS) is an important step in the process of building machine learning-based models. The goal of the feature selection step is to find a small subset of features that will provide good prediction results by removing noisy, irrelevant, or repetitive features. Commonly used Wrapper methods use a machine learning model as a black box and its performance as the goal function for evaluating different features' subsets and selecting the best one. To avoid examining all possible subsets (an NP-hard problem), search algorithms are used to find the subsets to be examined, in a heuristic way. As exhaustive search methods are computationally complicated, most methods use simple and greedy search methods that yield only locally optimal results and are not sensitive to possible features interactions, which means that a feature may be chosen at the expense of two others that are more informative together. We analyze the problem of searching the features subset space with reference to two dimensions, namely, memory of past selected features subset and future selected features. We propose a new wrapper feature selection method based on the deep artificial curiosity framework, which implements intrinsic reward reinforcement learning, with long short-term memory unit (LSTM). This novel algorithm integrates these two elements of memory and future step. We show that our method, called the Deep Curious Feature Selection (DCFS) algorithm, deals with feature interactions, and provides a feature subset which improves the accuracy of learning models on artificial and real datasets.

Realization of a Human-like Gait for a Bipedal Robot Based on Gait Analysis

There are many studies analyzing human motion. However, we do not yet fully understand the mechanisms of our own bodies. We believe that mimicking human motion and function using a robot will help us to deepen our understanding of humans. Therefore, we focus on the characteristics of the human gait, and the goal is to realize a human-like bipedal gait that lands on its heels and takes off from its toes. In this study, we focus on kinematic synergy (planar covariation) in the lower limbs as a characteristic gait seen in humans. Planar covariation is that elevation angles at the thigh, shank, and foot in the sagittal plane are plotted on one plane when the angular data are plotted on the three axes. We propose this feature as a reward for reinforcement learning. By introducing this reward, the bipedal robot achieved a human-like bipedal gait in which the robot lands on its heels and takes off from its toes. We also compared the learning results with those obtained when this feature was not used. The results suggest that planar covariation is one factor that characterizes a human-like gait.

An adaptive intelligent routing algorithm based on deep reinforcement learning

A mobile ad hoc network (MANET) is a self-organizing network composed of multiple mobile nodes. These nodes achieve distributed communication and wireless resource sharing through autonomous collaboration. However, typical routing protocols for MANETs suffer from uneven network load, low network throughput, poor anti-destruction performance, and low transmission reliability. In order to meet the routing protocol characteristics of distributed adaptability, reliability and anti-destruction, balance of load and energy consumption and guarantee of high QoS in MANETs, this paper proposes an adaptive intelligent routing algorithm based on the PER-D3QN model of deep reinforcement learning. To achieve better routing strategy selection, the routing problem is first modeled, a distributed multi-agent reinforcement learning routing framework is designed, and a deep reinforcement learning reward function considering multiple factors is used. Next, the implementation process of the adaptive intelligent routing algorithm using the PER-D3QN model is described in detail. Finally, a mobile self-organizing network environment was constructed based on Networkx, and corresponding data was collected to simulate the algorithm. The simulation results show that the adaptive intelligent routing algorithm proposed in this paper can balance the energy consumption of the network and the load of nodes, its average remaining energy has increased by more than 10% and PL is reduced by at least 10% compared to other algorithms, while guaranteeing high QoS performance of routing, that is, reducing the average end-to-end delay of data packet transmission by over 22% compared to other algorithms and maintaining a high packet delivery ratio.

Sliding principal component and dynamic reward reinforcement learning based IIoT attack detection

The Internet of Things (IoT) involves the gathering of all those devices that connect to the Internet with the purpose of collecting and sharing data. The application of IoT in the different sectors, including health, industry has also picked up the threads to augment over the past few years. The IoT and, by integrity, the IIoT, are found to be highly susceptible to different types of threats and attacks owing to the networks nature that in turn leads to even poor outcomes (i.e., increasing error rate). Hence, it is critical to design attack detection systems that can provide the security of IIoT networks. To overcome this research work of IIoT attack detection in large amount of evolutions is failed to determine the certain attacks resulting in a minimum detection performance, reinforcement learning-based attack detection method called sliding principal component and dynamic reward reinforcement learning (SPC–DRRL) for detecting various IIoT network attacks is introduced. In the first stage of this research methodology, preprocessing of raw TON_IoT dataset is performed by employing min–max normalization scaling function to obtain normalized values with same scale. Next, with the processed sample data as output, to extract data from multi-sources (i.e., different service profiles from the dataset), a robust log likelihood sliding principal component-based feature extraction algorithm is applied with an arbitrary size sliding window to extract computationally-efficient features. Finally, dynamic reward reinforcement learning-based IIoT attack detection model is presented to control the error rate involved in the design. Here, with the design of dynamic reward function and introducing incident repository that not only generates the reward function in an arbitrary fashion but also stores the action results in the incident repository for the next training, therefore reducing the attack detection error rate. Moreover, an IIoT attack detection system based on SPC–DRRL is constructed. Finally, we verify the algorithm on the ToN_IoT dataset of University of New South Wales Australia. The experimental results show that the IIoT attack detection time and overhead along with the error rate are reduced considerably with higher accuracy than that of traditional reinforcement learning methods.

A Novel Energy Management Method for Networked Multi-Energy Microgrids Based on Improved DQN

The networked Multi-Energy Microgrids (MEMGs) can significantly improve the economic efficiency of operation through energy cooperation and complementarity. However, the complex energy trading mechanism and multi-energy coupling decision process present various challenges for the optimal operation of MEMGs, such as the large decision space and the difficulty of converging to the global optimal solution. This study presents a novel energy management method for MEMG based on an improved Deep Q-network (IDQN) to solve these problems. Firstly, the equivalent model of an external interactive environment is constructed for each MEMG by using a novel Kriging surrogate enhanced Gate Recurrent Unit-Temporal Convolutional Network (GRU-TCN), which requires only the accessible energy exchange data and the public and independent information as input. The computational complexity of the reinforcement learning reward function is reduced accordingly by using this equivalent model. Subsequently, an improved k-crossover sampling strategy with reduced access frequency of the low reward actions is proposed to replace the traditional.-greedy strategy to further improve the original DQN. This effectively addresses the problem of inefficient exploration in a large-scale action space. Lastly, the proposed method is verified using a test system with three MEMGs. The result demonstrates that the proposed IDQN method presents better convergence and stability than the traditional DQN, and presents better energy management and operational efficiency.

Automated vision-based structural health inspection and assessment for post-construction civil infrastructure

In the search to enhance the effectiveness and efficiency of structural health monitoring for preserving safety and integrity in civil infrastructure, the search for innovative technologies is of critical importance. This paper introduces ExpoDet, a comprehensive framework designed for autonomous health inspection and infrastructure assessment. ExpoDet features a multi-detection detector, autonomous navigation for micro aerial vehicles facilitated through secondary reward reinforcement learning, and a damage aggregation scheme for autonomous health assessment following detections. Moreover, it presents an attention module called EEAM+ that introduces dynamic feature orientation and significantly enhances the capabilities of ExpoDet. ExpoDet is extensively tested and evaluated in both offsite and field test experiments. Comparisons with several state-of-the-art object detectors coupled with attention modules show an average improvement of approximately 3% across various evaluation metrics.

Local Path Planning via Improved Fuzzy and Q(λ)-learning Algorithms for the Mobile Robot

<p>With the complexity of the robot operating environment increases, there becoming higher demands on the optimal path planning for robots. Most of the path planning is performed in known environments and static models. However, there are still challenges for robots to perform path planning in complex unknown or dynamic environments, which will suffer from deadlock problems and obstacle avoidance failures. Reinforcement learning (RL) can help fuzzy algorithm to optimize the strategy. However, the difficulty of designing the rewards in RL makes the algorithm require a large number of samples to learn the strategy, resulting in computational complexity. To solve these problems, a new local path planning based on the improved fuzzy and Q(λ)-learning algorithms is proposed, aiming to plan the shortest path and avoid obstacles. For solving the problems of breaking through and avoiding obstacles, a fuzzy controller is designed. The distance of nearest obstacle in front of the mobile robot and the distance between the obstacles in the two breakout directions are regarded as the two inputs for this controller. And the two fuzzy quantities of the mobile robot’s running angle and the safe step length are outputted. In the path planning, the Q(λ)-learning algorithm are used to optimize the weights of the running angle and the safe step, obtaining a more accurate robot position and speeding up path planning efficiency. Furthermore, to solve the overlap problems among the starting point, end point, and obstacles, a safer running environment is designed considering radiuses of these objects. Besides, the mobile robot breakout scheme and sustainable obstacle avoidance scheme are designed to solve the deadlock problem and “large obstacle” avoidance problem, respectively. Simulation results in the sparse and complex operating environment show that our proposed algorithm can plan a relatively optimal and safe path, improving the success rate of path planning.</p> <p> </p>

Smart City Distributed Trade Economy Dynamic Scheduling of E-Commerce Platform Based on Reinforcement Learning Algorithm

Abstract In order to improve the reliability and economy of decentralized trade economy dynamic scheduling on e-Commerce platforms and shorten the running time of decentralized trade economy dynamic scheduling on e-Commerce platforms, a decentralized trade economy dynamic scheduling method based on the reinforcement learning algorithm is proposed. In this paper, we analyze the basic theory of the reinforcement learning algorithm, study the Q-learning algorithm, build a neural network to fit the value model, and initialize the reinforcement learning algorithm. With Markov decision process as the framework model, the optimal state behavior value function is updated by using the modeless discounted reward reinforcement learning algorithm Q-learning as the value iteration method. Gibbs distribution is used to construct exploratory random strategies to select behaviors with probability. Using the reinforcement learning algorithm and the three-layer feedforward neural network as the approximator of the state behavior value function, this paper studies the generalization of the value function faced by the decentralized trade economy dynamic scheduling of e-Commerce platforms and realizes the decentralized trade economy dynamic scheduling of e-Commerce platforms. The experimental results show that the proposed method can effectively improve the reliability and economy of the decentralized trade economy dynamic scheduling of e-Commerce platforms.

Enhancing construction robot learning for collaborative and long-horizon tasks using generative adversarial imitation learning

The development and deployment of robots on construction sites are integral to the industrialization of construction, known as Construction 4.0. Tele-operated and pre-programmed robots have enhanced construction efficiency and safety. However, their utilization on-site remains limited due to the need for expert remote control and the lack of adaptability in dynamic environments. Reinforcement learning (RL) has emerged as a promising solution, as RL-controlled robots possess inherent self-learning abilities to adapt to diverse situations. Nevertheless, manual design of RL reward functions for complex tasks poses challenges. To address this issue, inverse reinforcement learning (IRL) methods, such as Generative Adversarial Imitation Learning (GAIL), have been proposed to learn optimal actions through expert demonstration and self-exploration, without explicitly defined reward functions. In this study, we propose an innovative approach integrating GAIL and virtual reality (VR) integrated robot control approach to control robots for long-horizon collaborative construction tasks involving multiple sub-tasks. We employ VR expert demonstrations as input for GAIL training, enabling a team of robots, including an Unmanned Ground Vehicle (UGV) and two robot arms, to interact with the designed RL environment and perform tasks such as transporting, picking, and installing window panels. Handle long-horizon collaborative construction tasks (i.e., a long sequence of several sub-tasks performed by multiple robots). For evaluation, we compare the performance of our VR-GAIL model with a prevalent and robust RL baseline model, Proximal Policy Optimization (PPO). The results demonstrate that our reward-free VR-GAIL model achieves, on average, a 4.5% higher success rate than the PPO counterpart equipped with carefully designed reward functions across all three sub-tasks and their randomized variations. Furthermore, the performance gap between GAIL and PPO widens as the task difficulty increases. These findings indicate that our approach effectively enhances RL agent performance in tackling complex construction tasks while expediting development by eliminating reward function design requirements.

Reinforcement Learning system to capture value from Brazilian post-harvest offers

This study assesses the value capture of a result-oriented Product-Service System offer that constitutes a post-harvest solution. Applying the reinforcement learning reward system and general linear models, we identified the Brazilian farmer's propensities to choose different products and services from the proposed system. Reinforcement learning enables one to understand the choice process by rewarding the attributes selected and applying penalties to those not chosen. Regarding product options, farmers' most valued attributes were extended capacity, fixed installation, automatic dryer, and CO2 emission control, considering the investigated system. Regarding service options, the farmers opted for maintenance plans, performance reports, no photovoltaic energy, and purchase over the rental modality. These results assist managers through a reward learning system that constantly updates the value assigned by farmers to product and service attributes. They allow real-time visualization of changes in farmers' preferences regarding the product-service system configurations.

A Combined Reinforcement Learning and Model Predictive Control for Car-Following Maneuver of Autonomous Vehicles

Model predictive control is widely used in the design of autonomous driving algorithms. However, its parameters are sensitive to dynamically varying driving conditions, making it difficult to be implemented into practice. As a result, this study presents a self-learning algorithm based on reinforcement learning to tune a model predictive controller. Specifically, the proposed algorithm is used to extract features of dynamic traffic scenes and adjust the weight coefficients of the model predictive controller. In this method, a risk threshold model is proposed to classify the risk level of the scenes based on the scene features, and aid in the design of the reinforcement learning reward function and ultimately improve the adaptability of the model predictive controller to real-world scenarios. The proposed algorithm is compared to a pure model predictive controller in car-following case. According to the results, the proposed method enables autonomous vehicles to adjust the priority of performance indices reasonably in different scenarios according to risk variations, showing a good scenario adaptability with safety guaranteed.

Multi-objective deep reinforcement learning for computation offloading in UAV-assisted multi-access edge computing

Unmanned aerial vehicle-assisted multi-access edge computing (UAV-MEC) plays an important role in some complex environments such as mountainous and disaster areas. Computation offloading problem (COP) is one of the key issues of UAV-MEC, which mainly aims to minimize the conflict goals between energy consumption and delay. Due to the time-varying and uncertain nature of the UAV-MEC system, deep reinforcement learning is an effective method for solving the COP. Different from the existing works, in this paper, the COP in UAV-MEC system is modeled as a multi-objective Markov decision process, and a multi-objective deep reinforcement learning method is proposed to solve it. In the proposed algorithm, the scalar reward of reinforcement learning is expanded into a vector reward, and the weights are dynamically adjusted to meet different user preferences. The most important preferences are selected by non-dominated sorting, which can better maintain the previously learned strategy. In addition, the Q network structure combines Double Deep Q Network (Double DQN) with Dueling Deep Q Network (Dueling DQN) to improve the optimization efficiency. Simulation results show that the algorithm achieves a good balance between energy consumption and delay, and can obtain a better computation offloading scheme.

Deep Reinforcement Learning Reward Function Design for Autonomous Driving in Lane-Free Traffic

Lane-free traffic is a novel research domain, in which vehicles no longer adhere to the notion of lanes, and consider the whole lateral space within the road boundaries. This constitutes an entirely different problem domain for autonomous driving compared to lane-based traffic, as there is no leader vehicle or lane-changing operation. Therefore, the observations of the vehicles need to properly accommodate the lane-free environment without carrying over bias from lane-based approaches. The recent successes of deep reinforcement learning (DRL) for lane-based approaches, along with emerging work for lane-free traffic environments, render DRL for lane-free traffic an interesting endeavor to investigate. In this paper, we provide an extensive look at the DRL formulation, focusing on the reward function of a lane-free autonomous driving agent. Our main interest is designing an effective reward function, as the reward model is crucial in determining the overall efficiency of the resulting policy. Specifically, we construct different components of reward functions tied to the environment at various levels of information. Then, we combine and collate the aforementioned components, and focus on attaining a reward function that results in a policy that manages to both reduce the collisions among vehicles and address their requirement of maintaining a desired speed. Additionally, we employ two popular DRL algorithms—namely, deep Q-networks (enhanced with some commonly used extensions), and deep deterministic policy gradient (DDPG), which results in better policies. Our experiments provide a thorough investigative study on the effectiveness of different combinations among the various reward components we propose, and confirm that our DRL-employing autonomous vehicle is able to gradually learn effective policies in environments with varying levels of difficulty, especially when all of the proposed rewards components are properly combined.

Solving Robotic Manipulation With Sparse Reward Reinforcement Learning Via Graph-Based Diversity and Proximity

In multigoal reinforcement learning (RL), algorithms usually suffer from inefficiency in the collection of successful experiences in tasks with sparse rewards. By utilizing the ideas of relabeling hindsight experience and curriculum learning, some prior works have greatly improved the sample efficiency in robotic manipulation tasks, such as hindsight experience replay (HER), hindsight goal generation (HGG), graph-based HGG (G-HGG), and curriculum-guided HER (CHER). However, none of these can learn efficiently to solve challenging manipulation tasks with distant goals and obstacles, since they rely either on heuristic or simple distance-guided exploration. In this article, we introduce graph-curriculum-guided HGG (GC-HGG), an extension of CHER and G-HGG, which works by selecting hindsight goals on the basis of graph-based proximity and diversity. We evaluated GC-HGG in four challenging manipulation tasks involving obstacles in both simulations and real-world experiments, in which significant enhancements in both sample efficiency and overall success rates over prior works were demonstrated. Videos and codes can be viewed at this link: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://videoviewsite.wixsite.com/gc-hgg</uri> .