Reinforcement Learning Mechanism Research Articles

Evolving malware detection through instant dynamic graph inverse reinforcement learning

The rapid development and increasing evolution of malware necessitate novel defensive techniques with high accuracy and minimal false positives to safeguard information systems against potential threats. Unfortunately, current malware detection methods, primarily relying on deep learning to identify malicious fingerprints from extensive training sets, prove ineffective against few-shot and evolving malware variants. To address these challenges, this paper introduces MalIRL, which designs a model-free inverse reinforcement learning (IRL) mechanism to automatically capture the evolving attack intent of malware. Specifically, MalIRL explores six representative categories of malware actions and employs sliding windows to organically divide the massive malware execution event stream into multiple attack stages, achieving a reduced action space and state space. To model dynamic malicious environments, MalIRL proposes an instant dynamic heterogeneous graph representation learning technique. This technique learns state representations of different malware attack stages, enhancing detection accuracy and efficiency by incrementally capturing the newly added contextual semantics of diverse malware entities and relations. In experiments with three real-world malware datasets, MalIRL surpasses existing state-of-the-art methods, particularly in few-shot malware detection scenarios, MalIRL exhibits performance benefits of up to 18.9%.

Toward Cooperatively Caching in Multi-UAV-Assisted Network: A Queue-Aware CDS-Based Reinforcement Learning Mechanism With Energy-Efficiency Maximization

Toward Cooperatively Caching in Multi-UAV-Assisted Network: A Queue-Aware CDS-Based Reinforcement Learning Mechanism With Energy-Efficiency Maximization

SMITS: Social and Mobility aware Intelligent Task Scheduling in Vehicular Fog Computing — A Federated DRL Approach

Vehicular fog computing (VFC) has become an enticing research hot spot to provide resources for the Internet of Vehicles (IoV) application requests. Moreover, with the massive expansion of services in vehicular fog networks such as augmented reality, 3-D gaming, and autonomous driving, the allocation of fog computing resources to IoV applications constraining the deadline and energy consumption of the vehicles is becoming challenging. In this paper, we investigate a task scheduling problem in vehicular fog networks to jointly optimize the success rate of vehicular tasks and energy consumption of vehicles by considering the moving paths of the vehicles and the social relationships among the vehicles. In this problem, we model the social relationship among the vehicles based on their communication patterns and social characteristics to improve the success rate of the tasks. We also incorporate the mobility of vehicles using the Markov renewal process (MRP) technique. Initially, we formulate a mixed integer nonlinear programming (MINLP) problem for the proposed problem and further prove it to be an NP-hard problem. To solve the proposed problem, we design a federated deep reinforcement learning (FDRL) mechanism by converting the problem into a Markov decision process (MDP) problem. We compare our proposed algorithm with existing scheduling approaches, and simulation results show that the proposed algorithm performs efficiently for task scheduling problems. The proposed algorithm achieved a substantial improvement of 12% in the maximization of the success rate and 36% in the minimization of the energy consumption compared to the existing algorithms.

5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility: behavioural and computational evidence.

Cognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive-compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown. We investigated the effects of 5-HT2A receptor (5-HT2AR) and 5-HT2C receptor (5-HT2CR) antagonism on flexibility and underlying RL mechanisms. Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT2AR and 5-HT2CR antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task. 5-HT2AR antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT2AR antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT2CR antagonism also decreased beta, but reduced side stickiness. These data indicate that 5-HT2A and 5-HT2CRs both modulate different aspects of flexibility, with 5-HT2ARs modulating learning from negative feedback as measured using RL parameters and 5-HT2CRs for learning from positive feedback assessed through conventional measures.

Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning

Safety in goal directed Reinforcement Learning (RL) settings has typically been handled through constraints over trajectories and have demonstrated good performance in primarily short horizon tasks. In this paper, we are specifically interested in the problem of solving temporally extended decision making problems such as robots cleaning different areas in a house while avoiding slippery and unsafe areas (e.g., stairs) and retaining enough charge to move to a charging dock; in the presence of complex safety constraints. Our key contribution is a (safety) Constrained Search with Hierarchical Reinforcement Learning (CoSHRL) mechanism that combines an upper level constrained search agent (which computes a reward maximizing policy from a given start to a far away goal state while satisfying cost constraints) with a low-level goal conditioned RL agent (which estimates cost and reward values to move between nearby states). A major advantage of CoSHRL is that it can handle constraints on the cost value distribution (e.g., on Conditional Value at Risk, CVaR) and can adjust to flexible constraint thresholds without retraining. We perform extensive experiments with different types of safety constraints to demonstrate the utility of our approach over leading approaches in constrained and hierarchical RL.

OntoFact: Unveiling Fantastic Fact-Skeleton of LLMs via Ontology-Driven Reinforcement Learning

Large language models (LLMs) have demonstrated impressive proficiency in information retrieval, while they are prone to generating incorrect responses that conflict with reality, a phenomenon known as intrinsic hallucination. The critical challenge lies in the unclear and unreliable fact distribution within LLMs trained on vast amounts of data. The prevalent approach frames the factual detection task as a question-answering paradigm, where the LLMs are asked about factual knowledge and examined for correctness. However, existing studies primarily focused on deriving test cases only from several specific domains, such as movies and sports, limiting the comprehensive observation of missing knowledge and the analysis of unexpected hallucinations. To address this issue, we propose OntoFact, an adaptive framework for detecting unknown facts of LLMs, devoted to mining the ontology-level skeleton of the missing knowledge. Specifically, we argue that LLMs could expose the ontology-based similarity among missing facts and introduce five representative knowledge graphs (KGs) as benchmarks. We further devise a sophisticated ontology-driven reinforcement learning (ORL) mechanism to produce error-prone test cases with specific entities and relations automatically. The ORL mechanism rewards the KGs for navigating toward a feasible direction for unveiling factual errors. Moreover, empirical efforts demonstrate that dominant LLMs are biased towards answering Yes rather than No, regardless of whether this knowledge is included. To mitigate the overconfidence of LLMs, we leverage a hallucination-free detection (HFD) strategy to tackle unfair comparisons between baselines, thereby boosting the result robustness. Experimental results on 5 datasets, using 32 representative LLMs, reveal a general lack of fact in current LLMs. Notably, ChatGPT exhibits fact error rates of 51.6% on DBpedia and 64.7% on YAGO, respectively. Additionally, the ORL mechanism demonstrates promising error prediction scores, with F1 scores ranging from 70% to 90% across most LLMs. Compared to the exhaustive testing, ORL achieves an average recall of 80% while reducing evaluation time by 35.29% to 63.12%.

Interactions between dynamic team composition and coordination: an agent-based modeling approach

AbstractThis paper examines the interactions between selected coordination modes and dynamic team composition, and their joint effects on task performance under different task complexity and individual learning conditions. Prior research often treats dynamic team composition as a consequence of suboptimal organizational design choices. The emergence of new organizational forms that consciously employ teams that change their composition periodically challenges this perspective. In this paper, we follow the contingency theory and characterize dynamic team composition as a design choice that interacts with other choices such as the coordination mode, and with additional contextual factors such as individual learning and task complexity. We employ an agent-based modeling approach based on the NK framework, which includes a reinforcement learning mechanism, a recurring team formation mechanism based on signaling, and three different coordination modes. Our results suggest that by implementing lateral communication or sequential decision-making, teams may exploit the benefits of dynamic composition more than if decision-making is fully autonomous. The choice of a proper coordination mode, however, is partly moderated by the task complexity and individual learning. Additionally, we show that only a coordination mode based on lateral communication may prevent the negative effects of individual learning.

Special Issue on Cutting Edge of Reinforcement Learning and its Hybrid Methods

Since deep Q-networks and AlphaGO by Google DeepMind were proposed, not only a reinforcement learning integrated with deep learning but also its applications have attracted much attention on. ChatGPT, which learns by Proximal Policy Optimization as one of reinforcement learning mechanisms, is an excellent example. With this background, we propose the special issue titled “Cutting Edge of Reinforcement Learning and its Hybrid Methods,” which is the same of the special issue in 2017, to cover the latest progress and trends in reinforcement learning and its hybrid methods (combined with machine learning, neural networks, and evolutionary computation). We received 28 submissions, out of which 12 submissions were peer-reviewed and 16 submissions that were outside the scope of the special issue were excluded. After a thorough review process including comments and suggestions by reviewers, seven submissions were accepted. These included four theory and three application papers, indicating nearly equal contributions in theory and application aspects. The content of the first three papers is related to inverse reinforcement learning, while the fourth one is not directly related to reinforcement learning but it would be related to reinforcement learning in the future. All papers address issues surrounding cutting-edge technology in reinforcement learning. We would like to end by expressing that we hope and believe that this special issue can largely contribute to the development in the field of reinforcement learning while holding a wide international appeal.

Secure learning-based coordinated UAV–UGV framework design for medical waste transportation

A cost-effective solution with less human involvement must be developed for medical waste (MW) transportation. A learning-based coordinated unmanned aerial vehicle–unmanned ground vehicle (UAV–UGV) (CUU) framework, currently unavoidable use, with a transfer learning algorithm is suggested. A transfer learning algorithm is implemented for collision-free optimal path planning. In the framework, mobile ground robots collect medical waste from waste disposal centers through the pick-and-place technique. Then, networked drones lift the collected medical waste and fly through a predefined optimal trajectory. The framework considers the dynamic behavior of the environment and explores the actions for picking, placing, and dropping medical waste. A deep reinforcement learning mechanism has been incorporated for each successful or unsuccessful action by the framework to provide the rewards. With optimal policies, the coordinated UAV and UGV change their actions in dynamic conditions. An optimal cost of transportation of medical waste by the proposed framework is created by considering the weight of MW packets as the payload capacity of a CUU framework, the cost of steering the UAV and UGV, and the time required to transport the MW. The effectiveness of the CUU framework for MW transportation has been tested using MATLAB. The MW transportation data have been encrypted using an encryption key for security and authenticity.

Adaptive fuzzy echo state network optimal synchronization control of hybrid–order chaotic systems via reinforcement learning

In this paper, a novel optimal synchronization control scheme for fractional–integer hybrid–order chaotic systems is formulated. To deal with the fractional–order (FO) constraint, a transformation programme is developed and then the master system considered as an FO chaotic system is transformed into an integer–order one. A fuzzy echo state network (FESN) with the advantages of both fuzzy logic system and echo state network is introduced to approximate system uncertainty. Simultaneously, to alleviate the resource pressure, an optimal synchronization control is proposed in the light of the reinforcement learning mechanism, in which critic–actor update laws are constructed through the negative gradient of a positive function with regard to Bellman residual. Compared to typical fuzzy synchronization control, the devised FESN synchronization scheme with the same computational complexity possess superior approximation ability and synchronization performance. Ultimately, three simulation cases are exhibited to check the validity of the proposed approach.

A Procedural Constructive Learning Mechanism with Deep Reinforcement Learning for Cognitive Agents

Recent advancements in AI and deep learning have created a growing demand for artificial agents capable of performing tasks within increasingly complex environments. To address the challenges associated with continuous learning constraints and knowledge capacity in this context, cognitive architectures inspired by human cognition have gained significance. This study contributes to existing research by introducing a cognitive-attentional system employing a constructive neural network-based learning approach for continuous acquisition of procedural knowledge. We replace an incremental tabular Reinforcement Learning algorithm with a constructive neural network deep reinforcement learning mechanism for continuous sensorimotor knowledge acquisition, thereby enhancing the overall learning capacity. The primary emphasis of this modification centers on optimizing memory utilization and reducing training time. Our study presents a learning strategy that amalgamates deep reinforcement learning with procedural learning, mirroring the incremental learning process observed in human sensorimotor development. This approach is embedded within the CONAIM cognitive-attentional architecture, leveraging the cognitive tools of CST. The proposed learning mechanism allows the model to dynamically create and modify elements in its procedural memory, facilitating the reuse of previously acquired functions and procedures. Additionally, it equips the model with the capability to combine learned elements to effectively adapt to complex scenarios. A constructive neural network was employed, initiating with an initial hidden layer comprising one neuron. However, it possesses the capacity to adapt its internal architecture in response to its performance in procedural and sensorimotor learning tasks, inserting new hidden layers or neurons. Experimentation conducted through simulations involving a humanoid robot demonstrates the successful resolution of tasks that were previously unsolved through incremental knowledge acquisition. Throughout the training phase, the constructive agent achieved a minimum of 40% greater rewards and executed 8% more actions when compared to other agents. In the subsequent testing phase, the constructive agent exhibited a 15% increase in the number of actions performed in contrast to its counterparts.

A multi-swarm optimizer with a reinforcement learning mechanism for large-scale optimization

Large-scale global optimization (LSGO) problems involve numerous decision variables, are similar to real-world problems, and have generated research interest. To solve LSGO, a particle swarm optimizer (PSO) has been used. However, the many local optima and huge search space severely limit the effectiveness of the classic PSO. Dealing with the complexity of LSGO while avoiding the local optima is the main challenge of large-scale optimization algorithms. A multiswarm strategy has also been introduced to improve swarm diversity; however, it reduces the convergence speed. Previous studies have shown that reinforcement learning (RL) can improve the convergence ability of EAs owing to its increasing learning ability. In this study, we develop a multiswarm optimizer with an RL mechanism (MSORL) for LSGO. The MSORL includes a tri-particle group structure for subswarms to save the computational cost and balance the diversity and convergence. An RL-guided updating strategy is designed to enhance the convergence speed, and an adaptive tolerance-based search mechanism is employed to improve the diversity and avoid the local optima. The experimental results prove that the MSORL outperforms other state-of-the-art algorithms in terms of the convergence accuracy and speed.

Dynamics Learning Rate Bias in Pigeons: Insights from Reinforcement Learning and Neural Correlates

Research in reinforcement learning indicates that animals respond differently to positive and negative reward prediction errors, which can be calculated by assuming learning rate bias. Many studies have shown that humans and other animals have learning rate bias during learning, but it is unclear whether and how the bias changes throughout the entire learning process. Here, we recorded the behavior data and the local field potentials (LFPs) in the striatum of five pigeons performing a probabilistic learning task. Reinforcement learning models with and without learning rate biases were used to dynamically fit the pigeons’ choice behavior and estimate the option values. Furthemore, the correlation between the striatal LFPs power and the model-estimated option values was explored. We found that the pigeons’ learning rate bias shifted from negative to positive during the learning process, and the striatal Gamma (31 to 80 Hz) power correlated with the option values modulated by dynamic learning rate bias. In conclusion, our results support the hypothesis that pigeons employ a dynamic learning strategy in the learning process from both behavioral and neural aspects, providing valuable insights into reinforcement learning mechanisms of non-human animals.

Intelligent optimization method for real-time decision-making in laminated cooling configurations through reinforcement learning

Optimizing conjugate heat transfer in laminated configurations is pivotal for efficiently cooling turbine blades. The traditional approaches have limitations in achieving rapid optimization when design conditions change. The emergence of reinforcement learning offers an opportunity to address this issue. However, there is currently a notable scarcity of research on the application of reinforcement learning in the optimization of laminated cooling configurations. This paper presents a novel method using the Deep Q-Network (DQN) model for real-time optimization of laminated cooling configurations. A dataset comprising 246 sets of historical design data was generated through conjugate simulations and augmented with the incorporation of physical information. After undergoing off-policy reinforcement learning, the intelligent model was employed to address optimization problems associated with temperature levels and coolant usage. Results indicate that the intelligent model achieves accurate and real-time optimization for laminated cooling configuration, with an average time of 3 ms. The ability to identify the correct optimization direction is attributed to the formulation of reward and penalty mechanisms in reinforcement learning. Real-time optimization is enabled by establishing a forward computational path from design conditions to optimization results. The findings of this study are easily transferrable and contribute to cost savings in design.

Efficient inventory routing for Bike-Sharing Systems: A combinatorial reinforcement learning framework

Bike-sharing systems have become increasingly popular, providing a convenient, cost-effective, and environmentally friendly transportation option for urban commuters on short trips. However, an efficient and sustainable bike-sharing system faces a key challenge to dynamically balancing the supply and demand of bicycles through efficient inventory routing. This paper introduces a comprehensive combinatorial framework that tackles the critical challenges in the bike-sharing system's inventory routing problem. Firstly, we present a novel mathematical model that considers multiple delivery vehicle types and incorporates important factors like dispatch cost, service time, and user satisfaction, all while ensuring fair scheduling. The comprehensiveness of our model makes it highly applicable to real-world scenarios, addressing practical concerns faced by bike-sharing companies. Secondly, we leverage reinforcement learning mechanisms to gather quantitative information on the spatial and temporal patterns of demand and supply. With this data, we construct an effective regression model that accurately predicts station demand. Additionally, we propose an efficient heuristic approach to generate service sequences for delivery vehicle dispatching. Our approach employs a far-sighted strategy-based local iterative search algorithm to construct solutions, coupled with an adaptive exploration algorithm to continually improve solution quality. The proposed solution method is an innovative integration of reinforcement learning, demand prediction, and heuristic-based dispatching, significantly enhancing solution quality and computational efficiency. By bridging the gap between academic research and real-world practice, our framework offers practical and effective solutions for bike-sharing systems. Finally, we validate our proposed framework with extensive experimental results using real-world datasets. Our approach outperforms state-of-the-art algorithms within a short computational time, demonstrating its superiority in terms of solution quality compared to prior literature. Our research opens a new, viable direction for industrial practice, providing valuable insights for decision-makers to optimize bicycle inventory management in a smarter and more efficient way.

Opportunistic Block Validation for IoT Blockchain Networks

The blockchain network architecture is a promising technology for constructing highly secure Internet of Things (IoT) networks. IoT networks typically consist of many sensors and actuators. Blockchain network technology can be applied to secure control robots in smart factories or for reliable drone delivery in smart cities. The distributed ledger and data block validation across blockchain networks guarantee the ultimate data security. However, the current blockchain technology is restricted in terms of its overall deployment across IoT networks. A general permissionless blockchain technology typically targets high-performance network nodes with sufficient computing power and memory space. A blockchain node with less computing power and memory, such as an IoT sensor or actuator, cannot employ blockchain technology as a fully functional node. A lightweight blockchain provides practical blockchain availability to IoT networks. We propose an operational advance to develop a lightweight blockchain for IoT networks. The opportunistic block validation optimizes the block validation process. It measures the network vulnerability based on the node reputation, block validation degree, number of fraudulent messages, and network stability. The reinforcement learning mechanism employed in the opportunistic block validation determines whether block validation is to be performed. Two separately developed reinforcement learning methods can increase the processing performance while maintain the transaction data integrity. In addition, the proposed blockchain technology is easily implementable because it adopts a Hyperledger development environment. Directly embedding the proposed blockchain middleware platform in small computing devices proves the practicability of the proposed block validation mechanism.

Intelligent Control Scheme for Traffic Signals at Multi-intersections based on Pedestrian Safety

Traffic congestion has become a more and more serious problem. Meanwhile, pedestrian safety problem is another issue under the circumstance of multiple modes of vehicles that needed to be dealt with. And these two problems can be effectively solved if the traffic signal is set reasonably. This paper proposes an intelligent control scheme for traffic signals at multi-intersections based on pedestrian safety, which is designed by using multi-agent deep reinforcement learning and self-attention mechanism. And Centralized training decentralized execution is also applied in this paper which can effectively downsize the action dimension to make the model learn much faster. Though pedestrian safety and vehicle efficiency are two contradictory factors, this paper dexterously designs the reward function to let the model learn the optimal timing scheme. And the model outperforms the fixed-timed scheme and deep Q-learning scheme in the simulation traffic environment, with fewer conflicts between pedestrians and vehicles, shorter average delay time and higher average speed of vehicles at the same time.

Multi-Sensor Fusion Simultaneous Localization Mapping Based on Deep Reinforcement Learning and Multi-Model Adaptive Estimation.

In this study, we designed a multi-sensor fusion technique based on deep reinforcement learning (DRL) mechanisms and multi-model adaptive estimation (MMAE) for simultaneous localization and mapping (SLAM). The LiDAR-based point-to-line iterative closest point (PLICP) and RGB-D camera-based ORBSLAM2 methods were utilized to estimate the localization of mobile robots. The residual value anomaly detection was combined with the Proximal Policy Optimization (PPO)-based DRL model to accomplish the optimal adjustment of weights among different localization algorithms. Two kinds of indoor simulation environments were established by using the Gazebo simulator to validate the multi-model adaptive estimation localization performance, which is used in this paper. The experimental results of the proposed method in this study confirmed that it can effectively fuse the localization information from multiple sensors and enable mobile robots to obtain higher localization accuracy than the traditional PLICP and ORBSLAM2. It was also found that the proposed method increases the localization stability of mobile robots in complex environments.

Research on the Reform and Practice of Informatization Mode of Adult Higher and Continuing Education Academic Records Management in the Context of Three-Whole Parenting

Abstract Information technology reform is an important way for schools to realize efficient management and information technology should be introduced into the management of student registration files to accelerate the process of informationization of registration file management. This paper builds an informatization database of adult higher continuing education student registration file management based on informatization technology and carries out the design of the main business process of student registration files. The multi-layer stacked LSTM network in the ELMo model is utilized to learn the preprocessing of the text of the school records data, and the CART decision tree algorithm is improved based on the reinforcement learning mechanism in order to improve the classification prediction accuracy of the decision tree in the dataset. In order to verify the feasibility of the school registration file informationization database constructed in this paper, a test analysis is conducted for the application of data mining technology and informationization database. The results show that the check accuracy of the CART decision tree algorithm improved by the reinforcement learning mechanism is around 80%, and the throughput of this paper’s database reaches 4827ops/s when the data writing volume is 1*105GB. When the number of threads reaches 128, the throughput of this paper’s informative database is around 5486ops/s, and when the data volume reaches 12*105 entries, the database in this paper the query response time is around 602ms. Under the background of three-round education, it is necessary to fully integrate information technology with the management of school registration files, which will lead to the innovation of the informationization mode of the management of school registration files in adult higher education.

Intelligent navigation for the cruise phase of solar system boundary exploration based on Q-learning EKF

With the continuous advancement of deep space exploration missions, the solar system boundary exploration mission is established as one of the China's most important deep space scientific exploration missions. The mission of the solar system boundary exploration has many challenges such as ultra-remote detection distance, ultra-long operation time, and ultra-long communication delay. Therefore, the problem of high-precision autonomous navigation needs to be solved urgently. This paper designs an autonomous intelligent navigation method based on X-ray pulsars in the cruise phase, which estimate the motion state of the probe in real time. The proposed navigation method employs the Q-learning Extended Kalman filter (QLEKF) to improve navigation accuracy during long periods of self-determining running. The QLEKF selects automatically the error covariance matrix parameter of the process noise and the measurement noise by the reward mechanism of reinforcement learning. Compared to the traditional EKF and AEKF, the QLEKF improves the estimation accuracy of position and velocity. Finally, the simulation result demonstrates the effectiveness and the superiority of the intelligent navigation algorithm based on QLEKF, which can satisfy the high-precision navigation requirements in the cruise phase of the solar system boundary exploration.