Reinforcement Learning Mechanism Research Articles

A Linear Programming-Based Reinforcement Learning Mechanism for Incomplete-Information Games

A Linear Programming-Based Reinforcement Learning Mechanism for Incomplete-Information Games

Reinforcement Learning for Synchronization of Heterogeneous Multiagent Systems by Improved Q-Functions.

This article dedicates to investigating a methodology for enhancing adaptability to environmental changes of reinforcement learning (RL) techniques with data efficiency, by which a joint control protocol is learned using only data for multiagent systems (MASs). Thus, all followers are able to synchronize themselves with the leader and minimize their individual performance. To this end, an optimal synchronization problem of heterogeneous MASs is first formulated, and then an arbitration RL mechanism is developed for well addressing key challenges faced by the current RL techniques, that is, insufficient data and environmental changes. In the developed mechanism, an improved Q-function with an arbitration factor is designed for accommodating the fact that control protocols tend to be made by historic experiences and instinctive decision-making, such that the degree of control over agents' behaviors can be adaptively allocated by on-policy and off-policy RL techniques for the optimal multiagent synchronization problem. Finally, an arbitration RL algorithm with critic-only neural networks is proposed, and theoretical analysis and proofs of synchronization and performance optimality are provided. Simulation results verify the effectiveness of the proposed method.

Modeling and simulation of Fuzzy-rule based WBAN using multi-level fuzzy colored petri-nets and reinforcement learning

By developing IoTs (Internet of Things), a new concept is arising in WBANs (Wireless Body Area Networks), called IoBs (Internet of Bodies). WBANs are the safety-critical devices that use rule-based knowledge with fuzzy variables for monitoring patients. There are two issues with the knowledgebase: (1) appropriate knowledge representation and inference and (2) handling the situations for which there is no rule in the knowledgebase. In this paper, for the first issue we present MFCPN (Multi-level Fuzzy Colored Petri-nets) and for the second one we present a RL (Reinforcement Learning) mechanism for decision making based on the past device’s feedback to the environment. Through a few scenarios and two datasets, the proposed approach has been applied to pacemakers to show the approach's effectiveness. The comparison between decision making by our approach and two real datasets indicates the proposed approach has 79.3 % accuracy.

Multi-UAVs task allocation method based on MPSO-SA-DQN

Multi-UAVs play an important role in the battlefield. Although many methods are proposed to solve the Multi-UAV task allocation, there still existing the problems of complex time constraints and uncertain solution space. The reason is that multi-UAVs usually face changing environmental factors. Aiming at solving such problem, this paper proposes a multi-UAV task assignment method based on Deep Q-based evolutionary reinforcement learning algorithms (MPSO-SA-DQN). Specifically, this method builds a multi-agent training framework based on the deep evolutionary reinforcement learning mechanism and SA-DQN. Its aim is to improve the global exploration and optimization capabilities of multi-agents. At the same time, the multi-dimensional particle swarm optimization algorithm is introduced to optimize the state space. Based on task priority mapping, the MPSO-SA-DQN algorithm framework is proposed. As a result, multi-agents can optimize the execution state in real time in the environment interaction. Besides, it also has the ability to reach optimal state and maximum reward. According to the characteristics of multi-UAV global task assignment, this paper designs a priority state space autoencoder strategy and global task feature. A multi-UAVs tasks allocation and iterative optimization method based on MPSO-SA-DQN algorithm is proposed, so as to continuously optimize the task allocation scheme. The simulation results show that the multi-UAV task allocation method based on MPSO-SA-DQN can effectively solve the problem of uncertainty in the optimal solution space of task allocation. At the same time, the algorithm achieves faster convergence result, and a good prospect of promotion in the field of UAV swarm cooperative task planning.

Optimized dead-zone inverse control using reinforcement learning and sliding-mode mechanism for a class of high-order nonlinear systems

An optimized control method is developed for a class of high-order nonlinear dynamic systems having controller dead-zone phenomenon. Dead-zone refers to the controller with zero behavior within a certain range, so it will inevitably affect system performance. In order to make the optimized control eliminate the effect of dead zone, the adaptive dead-zone inverse and reinforcement learning (RL) techniques are combined. The main idea is to find the desired optimized control using RL as the input of dead-zone inverse function and then to design the adaptive algorithm to estimate the unknown parameters of dead-zone inverse function, so that the competent system control can be yielded from the dead-zone function. However, most existing RL algorithms are difficult to apply in the dead zone inverse methods because of the algorithm complexity. The proposed RL greatly simplifies the algorithm because it derives the training rules from the negative gradient of a simple positive function yielded by the partial derivative of Hamilton–Jacobi-Bellman (HJB) equation. Meanwhile, the proposed dead-zone inverse method also requires fewer adaptive parameters. Finally, the proposed control is attested through theoretical proofs and simulation examples.

Enhanced reinforcement learning-based two-way transmit-receive directional antennas neighbor discovery in wireless ad hoc networks

The utilization of directional antennas for neighbor discovery in wireless ad hoc networks brings notable benefits, such as extended transmission range, reduced transmission interference, and enhanced antenna gain. However, when nodes use directional antennas for neighbor discovery, the communication range is limited, resulting in a lack of knowledge of potential neighbors. Hence, it is necessary to design a special antenna direction switching strategy for neighbor discovery based on directional antennas. Traditional methods of switching antenna directions are often random or follow predefined sequences, overlooking the historical knowledge of sector exploration for antenna directions. In contrast, existing machine learning approaches aim to leverage observed historical knowledge to adjust antenna directions for faster neighbor discovery. Nonetheless, the latency of neighbor discovery is still high because the node cannot fully utilize the observed historical knowledge (i.e., only using the knowledge observed by the node in transmission mode, ignoring the knowledge observed by the node in reception mode). Meanwhile, the corresponding reward and penalty mechanisms are still not detailed enough (i.e., these reward and penalty mechanisms only consider the sectors of discovered and undiscovered neighboring nodes, ignoring the scenario of sectors that have been rewarded). In this paper, the neighbor discovery process is modeled as a reinforcement learning-based learning automaton. We propose an enhanced reinforcement learning-based two-way transmit-receive directional antennas neighbor discovery algorithm, called ERTTND. The algorithm consists of a two-way transmit-receive reinforcement learning mechanism (TTRL) and an enhanced reward-and-penalty mechanism (ERAP). This algorithm leverages insights from nodes in transmission and reception modes to refine their tactical decisions. Then, through an enriched reward-and-penalty framework, nodes optimize their strategies, thus expediting neighbor discovery based on directional antennas in wireless ad hoc networks. Simulation results demonstrate that compared to existing representative algorithms, the proposed ERTTND algorithm can achieve over 30% savings in terms of average discovery delay and energy consumption.

Disentangling negative reinforcement, working memory, and deductive reasoning deficits in elevated BMI

Neuropsychological data suggest that being overweight or obese is associated with a tendency to perseverate behavior despite negative feedback. This deficit might be observed due to other cognitive factors, such as working memory (WM) deficits or decreased ability to deduce model-based strategies when learning by trial-and-error. In the present study, a group of subjects with overweight or obesity (Ow/Ob, n = 30) was compared to normal-weight individuals (n = 42) in a modified Reinforcement Learning (RL) task. The task was designed to control WM effects on learning by manipulating cognitive load and to foster model-based learning via deductive reasoning. Computational modelling and analysis were conducted to isolate parameters related to RL mechanisms, WM use, and model-based learning (deduction parameter). Results showed that subjects with Ow/Ob had a higher number of perseverative errors and used a weaker deduction mechanism in their performance than control individuals, indicating impairments in negative reinforcement and model-based learning, whereas WM impairments were not responsible for deficits in RL. The present data suggests that obesity is associated with impairments in negative reinforcement and model-based learning.

A novel energy-efficiency framework for UAV-assisted networks using adaptive deep reinforcement learning

In the air-to-ground transmissions, the lifespan of the network is based on the “unmanned aerial vehicle's (UAV)” life span because of the limited battery capacity. Thus, the enhancement of energy efficiency and the outage of the ground candidate’s minimization are significant factors of the network functionality. UAV-aided transmission can highly enhance the spectrum efficacy and coverage. Because of their flexible deployment and the high maneuverability, the UAVs can be the best alternative for the situations where the “Internet of Things (IoT)” systems utilize more energy to attain the essential information rate, when they are far away from the terrestrial base station. Therefore, it is significant to win over the few troubles in the conventional UAV-aided efficiency approaches. Thus, this proposed work is aimed to design an innovative energy efficiency framework in the UAV-assisted network using a reinforcement learning mechanism. The energy efficiency optimization in the UAV offers better wireless coverage to the static and mobile ground user. Presently, reinforcement learning techniques effectively optimize the energy efficiency rate of the system by employing the 2D trajectory mechanism, which effectively removes the interference rate attained in the nearby UAV cells. The main objective of the recommended framework is to maximize the energy efficiency rate of the UAV network by performing the joint optimization using UAV 3D trajectory, with the energy utilized during interference accounting, and connected user counts. Hence, an efficient Adaptive Deep Reinforcement Learning with Novel Loss Function (ADRL-NLF) framework is designed to provide a better energy efficiency rate to the UAV network. Moreover, the parameter of ADRL is tuned using the Hybrid Energy Valley and Hermit Crab (HEVHC) algorithm. Various experimental observations are performed to observe the effectualness rate of the recommended energy efficiency model for UAV-based networks over the classical energy efficiency framework in UAV Networks.

How to improve “construct, merge, solve and adapt"? Use reinforcement learning!

AbstractIn this work, we propose a new variant of construct, merge, solve, and adapt (CMSA), which is a recently introduced hybrid metaheuristic for combinatorial optimization. Our newly proposed variant, named reinforcement learning CMSA (RL-CMSA), makes use of a reinforcement learning (RL) mechanism trained online with data gathered during the search process. In addition to generally outperforming standard CMSA, this new variant proves to be more flexible as it does not require a greedy function for the evaluation of solution components at each solution construction step. We present RL-CMSA as a general framework for enhancing CMSA by leveraging a simple RL learning process. Moreover, we study a range of specific designs for the employed learning mechanism. The advantages of the introduced CMSA variant are demonstrated in the context of the far from most string and minimum dominating set problems, showing the improvement in performance and simplicity with respect to standard CMSA. In particular, the best performing RL-CMSA variant proposed is statistically significantly better than the standard algorithm for both problems, obtaining 1.28% and 0.69% better results on average respectively.

Policy complexity suppresses dopamine responses.

Limits on information processing capacity impose limits on task performance. We show that animals achieve performance on a perceptual decision task that is near-optimal given their capacity limits, as measured by policy complexity (the mutual information between states and actions). This behavioral profile could be achieved by reinforcement learning with a penalty on high complexity policies, realized through modulation of dopaminergic learning signals. In support of this hypothesis, we find that policy complexity suppresses midbrain dopamine responses to reward outcomes, thereby reducing behavioral sensitivity to these outcomes. Our results suggest that policy compression shapes basic mechanisms of reinforcement learning in the brain.

A Multiproject and Multilevel Plan Management Model Based on a Hybrid Program Evaluation and Review Technique and Reinforcement Learning Mechanism

It is very difficult for manufacturing enterprises to achieve automatic coordination of multiproject and multilevel planning when they are unable to make large-scale resource adjustments. In addition, planning and coordination work mostly relies on human experience, and inaccurate planning often occurs. This article innovatively proposes the PERT-RP-DDPGAO algorithm, which effectively combines the program evaluation and review technique (PERT) and deep deterministic policy gradient (DDPG) technology. Innovatively using matrix computing, the resource plan (RP) itself is used for the first time as an intelligent agent for reinforcement learning, achieving automatic coordination of multilevel plans. Through experiments, this algorithm can achieve automatic planning and has interpretability in management theory. To solve the problem of continuous control, the second half of the new algorithm adopts the DDPG algorithm, which has advantages in convergence and response speed compared to traditional reinforcement learning algorithms and heuristic algorithms. The response time of this algorithm is 3.0% lower than the traditional deep Q-network (DQN) algorithm and more than 8.4% shorter than the heuristic algorithm.

The Impact of Educational Informatics on School Management Decision-Making in the Context of Big Data

In the domain of private education, particularly vocational education, school management encounters a complex and variable environment with diverse demands. As the advancement of educational informatics progresses, schools have accumulated a substantial amount of data, including student learning behavior, teaching data, and classroom utilization statistics. However, the efficient utilization of these data for scientific management decisions to enhance the quality of education and resource efficiency presents an urgent challenge. The study of the impacts of educational informatics on school management decision-making is of significant importance. It aids administrators in comprehensively understanding and analyzing the allocation and usage of educational resources, thereby enhancing management efficiency and teaching quality and further promoting the development of educational informatics. Despite some progress in data analysis and decision-making methods within educational informatics, deficiencies remain. Traditional methods often overlook comprehensive considerations of spatiotemporal data, failing to accurately reflect the dynamic changes in educational resources. Existing decision-making methods, predominantly based on single-agent models, lack studies on multi-agent collaborative decision-making, resulting in suboptimal decision outcomes. This study comprises three main components: first, the data modeling of educational informatics based on a spatiotemporal data model; second, an intelligent decision-making framework for school management under a reinforcement learning mechanism; and third, the implementation of an intelligent school management decision-making method based on multi-agent reinforcement learning. Through these investigations, the scientific nature and efficiency of school management are enhanced, providing new ideas and methods for management decision-making in other educational domains.

Cognitive mechanisms of learning in sequential decision-making under uncertainty: an experimental and theoretical approach.

Learning to make adaptive decisions involves making choices, assessing their consequence, and leveraging this assessment to attain higher rewarding states. Despite vast literature on value-based decision-making, relatively little is known about the cognitive processes underlying decisions in highly uncertain contexts. Real world decisions are rarely accompanied by immediate feedback, explicit rewards, or complete knowledge of the environment. Being able to make informed decisions in such contexts requires significant knowledge about the environment, which can only be gained via exploration. Here we aim at understanding and formalizing the brain mechanisms underlying these processes. To this end, we first designed and performed an experimental task. Human participants had to learn to maximize reward while making sequences of decisions with only basic knowledge of the environment, and in the absence of explicit performance cues. Participants had to rely on their own internal assessment of performance to reveal a covert relationship between their choices and their subsequent consequences to find a strategy leading to the highest cumulative reward. Our results show that the participants' reaction times were longer whenever the decision involved a future consequence, suggesting greater introspection whenever a delayed value had to be considered. The learning time varied significantly across participants. Second, we formalized the neurocognitive processes underlying decision-making within this task, combining mean-field representations of competing neural populations with a reinforcement learning mechanism. This model provided a plausible characterization of the brain dynamics underlying these processes, and reproduced each aspect of the participants' behavior, from their reaction times and choices to their learning rates. In summary, both the experimental results and the model provide a principled explanation to how delayed value may be computed and incorporated into the neural dynamics of decision-making, and to how learning occurs in these uncertain scenarios.

Exploring the effectiveness of reward-based learning strategies for second-language speech sounds.

Adults struggle to learn non-native speech categories in many experimental settings (Goto, Neuropsychologia, 9(3), 317-323 1971), but learn efficiently in a video game paradigm where non-native speech sounds have functional significance (Lim & Holt, Cognitive Science, 35(7), 1390-1405 2011). Behavioral and neural evidence from this and other paradigms point toward the involvement of reinforcement learning mechanisms in speech category learning (Harmon, Idemaru, & Kapatsinski, Cognition, 189, 76-88 2019; Lim, Fiez, & Holt, Proceedings of the National Academy of Sciences, 116, 201811992 2019). We formalize this hypothesis computationally and implement a deep reinforcement learning network to map between environmental input and actions. Comparing to a supervised model of learning, we show that the reinforcement network closely matches aspects of human behavior in two experiments - learning of synthesized auditory noise tokens and improvement in speech sound discrimination. Both models perform comparably and the similarity in the output of each model leads us to believe that there is little inherent computational benefit to a reward-based learning mechanism. We suggest that the specific neural circuitry engaged by the paradigm and links between striatum and superior temporal areas play a critical role in effective learning.

Eligibility traces in an autonomous soccer robot with obstacle avoidance and navigation policy

As the thematic literature of machine learning suggests, reinforcement learning falls between the two methods of supervised learning and unsupervised learning. In this method, the learning agent receives a reward or punishment from the environment according to its action. Therefore, the learning agent interacts with the environment through trial and error and learns to choose the optimal action to achieve its goal. In the meantime, the eligibility traces are considered as one of the main mechanisms of reinforcement learning in receiving delayed rewards. In the use of conventional reinforcement learning methods, when the learning agent achieves a goal, only the value function of the last state-action pair changes, but all the trace states are affected based on the eligibility traces. In other words, the delayed rewards are distributed throughout the trace. Like the ant pheromone effect, this method can increase learning speed empirically to some extent. A soccer robot on the field encounters moving obstacles such as balls, rival robots, and home robots, and fixed obstacles such as gates and flags, so its environment is in a very dynamic state. Therefore, due to the dynamic environment, it is an important issue for autonomous soccer robots to avoid obstacles and should be considered in real play. The main idea of this study is to determine the appropriate traces for the robot to move towards the ball and ultimately to score with the approach of avoiding obstacles in simulating a real soccer match. The desired results obtained from a game played indicate a high level of online experience and decision-making power in the face of new situations.

An overview: Attention mechanisms in multi-agent reinforcement learning

In recent years, in the field of Multi-Agent Systems (MAS), significant progress has been made in the research of algorithms that combine Reinforcement Learning (RL) with Attention Mechanism (AM). However, there is a lack of comprehensive reviews in this field. Based on this, this paper does the following work. Firstly, it reviews the classical algorithms of RL and AM; Secondly, it systematically introduces the combination of RL and AM; Thirdly, it sorts out their application in the field of single-agent and multi-agent, and pays attention to and looks forward to the challenges and future research prospects in this field; The last part offers a comprehensive analysis of the challenges encountered by research in this field and anticipates future research paths. The research offers a conceptual understanding and theoretical foundation for future applications of RL using AM and facilitates further in-depth study in this field for researchers.

Chinese image captioning with fusion encoder and visual keyword search

AbstractAutomatic generation of image captions is essentially a cross‐modal conversion from image to text. Owing to the differences in linguistic characteristics between Chinese and English, quite a few Chinese image captioning methods have recently been proposed. Nevertheless, the existing Chinese image captioning models usually lack attention to local details of images or tend to produce general descriptions. To address these challenges, a Chinese image captioning method is proposed that incorporates fusion encoder, visual keyword search, and reinforcement learning. The fusion encoder can simultaneously extract local and global features of the input image to enrich the semantic information in the decoding stage, visual keyword search can pursue potential visual words associated with the image content, and the reinforcement learning mechanism can optimize the evaluation metric CIDEr at sentence level to promote the lexical diversity of image description. The results of extensive experiments demonstrate that the proposed model outperforms the state‐of‐the‐art models and delivers expressive and informative Chinese image captions.

Synergies and Challenges in the Integration of Cloud Computing and Deep Learning: Current Status, Interconnectedness, and Future Directions

This article reviews the status and recent developments in the integration of cloud computing and deep learning, as well as the interrelationship between these two technologies. The paper explores the intersection of cloud computing and deep learning in addressing cybersecurity challenges. Amidst the rapid expansion of the worldwide public cloud services market, the vulnerability to cyber-attacks and breaches in data management is on the rise. Different intrusion detection systems use different deep learning techniques to improve the effectiveness of intrusion detection in cloud computing environments. Additionally, the use of encryption technology and the corresponding deep learning retrieval technology further improves the security of cloud data. Moreover, the paper deeply studies how the scheduling mechanism of deep reinforcement learning can optimize the performance of cloud services by efficiently allocating resources and solving the problem of slow cloud service speed. It also derives the optimal energy strategy through deep neural networks to address the energy consumption challenges in cloud computing data centers. This article also reviews the five emerging architectures of cloud computing and explores the role of deep learning within these frameworks. Finally, it analyzes some of the challenges facing the future of cloud computing and deep learning, including the security and confidentiality of cloud computing, as well as low latency and high throughput optimization in the field of deep learning. In summary, this article provides insight into current trends, challenges, and future prospects for the evolving integration between cloud computing and deep learning.

Evolving malware detection through instant dynamic graph inverse reinforcement learning

The rapid development and increasing evolution of malware necessitate novel defensive techniques with high accuracy and minimal false positives to safeguard information systems against potential threats. Unfortunately, current malware detection methods, primarily relying on deep learning to identify malicious fingerprints from extensive training sets, prove ineffective against few-shot and evolving malware variants. To address these challenges, this paper introduces MalIRL, which designs a model-free inverse reinforcement learning (IRL) mechanism to automatically capture the evolving attack intent of malware. Specifically, MalIRL explores six representative categories of malware actions and employs sliding windows to organically divide the massive malware execution event stream into multiple attack stages, achieving a reduced action space and state space. To model dynamic malicious environments, MalIRL proposes an instant dynamic heterogeneous graph representation learning technique. This technique learns state representations of different malware attack stages, enhancing detection accuracy and efficiency by incrementally capturing the newly added contextual semantics of diverse malware entities and relations. In experiments with three real-world malware datasets, MalIRL surpasses existing state-of-the-art methods, particularly in few-shot malware detection scenarios, MalIRL exhibits performance benefits of up to 18.9%.

Toward Cooperatively Caching in Multi-UAV-Assisted Network: A Queue-Aware CDS-Based Reinforcement Learning Mechanism With Energy-Efficiency Maximization

Toward Cooperatively Caching in Multi-UAV-Assisted Network: A Queue-Aware CDS-Based Reinforcement Learning Mechanism With Energy-Efficiency Maximization