Invalid Actions Research Articles

Optimizing intelligent penetration path planning using reinforcement learning: A focus on valid action masking and sample enhancement

Abstract. As penetration testing is a crucial technique for the rapid evolution of cybersecurity to identify system vulnerabilities, traditional penetration testing methods face the difficulty of large amounts of manual labor and professional knowledge, which makes them difficult to scale. This paper proposes a highly automated and efficient penetration testing approach based on the MASK-SALT-DQN algorithm. MASK-SALT-DQN is a reinforcement learning model that effectively optimizes penetration path planning by valid action masking and sample enhancement. Through valid action masking, the MASK-SALT-DQN model filters out redundant and invalid actions in the solution space, reducing the complexity of the action space and improving the convergence speed and efficiency. In addition, the sample enhancement method increases the frequency of critical exploitation actions and positive rewards in the sparse-reward environment, speeding up the learning process of the agent. Experiments have been conducted on the NASim attack simulation network to demonstrate the advantages of the proposed model over the baseline method. The experimental results show that our model has a significant performance advantage over baseline methods in large networks, and it can be applied to the field of scalable and efficient automated penetration testing.

The unmanned vehicle on-ramp merging model based on AM-MAPPO algorithm

In response to the issues of low merging success rates and poor safety in the on-ramp merging scenario within autonomous driving, we propose an on-ramp merging model for unmanned vehicles based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. Firstly, we introduce an Action-Mask (AM) to prevent the sampling of invalid actions during merging, thus enhancing safety by ensuring only valid actions are considered. Secondly, we incorporate noise advantage values to encourage unmanned vehicles to thoroughly explore the environment and avoid being trapped in local optimal solutions. Experimental results demonstrate that the AM-MAPPO algorithm model improves both safety and traffic efficiency.

A multi-agent reinforcement learning method for distribution system restoration considering dynamic network reconfiguration

Extreme weather, chain failures, and other events have increased the probability of wide-area blackouts, which highlights the importance of rapidly and efficiently restoring the affected loads. This paper proposes a multi-agent reinforcement learning method for distribution system restoration. Firstly, considering that the topology of the distribution system may change during network reconfiguration, a dynamic agent network (DAN) architecture is designed to address the challenge of input dimensions changing in neural network. Two encoders are created to capture observations of the environment and other agents respectively, and an attention mechanism is used to aggregate an arbitrary-sized neighboring agent feature set. Then, considering the operation constraints of the DSR, an action mask mechanism is implemented to filter out invalid actions, ensuring the security of the strategy. Finally, an IEEE 123-node test system is used for validation, and the experimental results showed that the proposed algorithm can effectively assist agents in accomplishing collaborative DSR tasks.

A Self-Attention-Based Deep Reinforcement Learning Approach for AGV Dispatching Systems.

The automated guided vehicle (AGV) dispatching problem is to develop a rule to assign transportation tasks to certain vehicles. This article proposes a new deep reinforcement learning approach with a self-attention mechanism to dynamically dispatch the tasks to AGV. The AGV dispatching system is modeled as a less complicated Markov decision process (MDP) using vehicle-initiated rules to dispatch a workcenter to an idle AGV. In order to deal with the highly dynamical environment, the self-attention mechanism is introduced to calculate the importance of different information. The invalid action masking technique is performed to alleviate false actions. A multimodal structure is employed to mix the features of various sources. Comparative experiments are performed to show the effectiveness of the proposed method. The properties of the learned policies are also investigated under different environment settings. It is discovered that the policies explore and learn the properties of different systems, and also smooth the traffic congestion. Under certain environment settings, the policy converges to a heuristic rule that assigns the idle AGV to the workcenter with the shortest queue length, which shows the adaptiveness of the proposed method.

Deep-learning based autonomous-exploration for UAV navigation

Unmanned Aerial Vehicles (UAVs), functioning as aerial robots, undertake multifarious missions within intricate and uncharted environments, necessitating autonomous exploration for efficient UAV navigation. This paper introduces a Deep Learning-based Autonomous UAV Exploration method (DLAE), which distinguishes itself from traditional autonomous ground robot exploration by employing cameras instead of radar sensors. This difference propels us toward proposing a hybrid action space that integrates both positional and yaw actions, aimed at circumventing the limitations imposed by the Field of View (FOV). Furthermore, to enhance learning efficiency in autonomous UAV exploration, we design an invalid action masking scheme. Significantly, our contribution includes the development of an autoregressive network model alongside a distinct training procedure specifically tailored for autonomous UAV exploration. Comprehensive experiments conducted across a variety of test maps reveal that our proposed method surpasses existing learning-based approaches, delivering superior exploration efficiency and reduced decision-making time in comparison to conventional strategies. Moreover, physical simulations have corroborated the effectiveness of our method in practical scenarios, showcasing its potential for real-world applications.

System-of-systems approach to spatio-temporal crowdsourcing design using improved PPO algorithm based on an invalid action masking

Spatio-temporal crowdsourcing (STC) is a typical case of complex system-of-systems (SoSs) design, wherein the primary objective is to allocate real-time tasks to suitable groups of workers. Over time, the STC allocation has gradually evolved into a dynamic matching involving three distinct entities: tasks, workers, and workplaces. Aiming at addressing the problems of poor convergence, slow response and sparse actions caused by the spatial complexity and time dynamics of the STC, this paper proposes an improved proximal policy optimization algorithm based on an invalid action masking (IAM-IPPO) for the SoSs design of the STC. Initially, the ternary dynamic matching (TDM) of tasks, workers and workplaces in the STC is described. Furthermore, the STC allocation is formulated as a Markov decision process, with the corresponding definition of state space, action space, and reward mechanism. On this basis, an invalid action masking (IAM) method is mainly introduced to update the policy-based network of proximal policy optimization (PPO), realizing sampling only from valid actions to masking invalid action selection. Subsequently, the algorithmic framework of IAM-IPPO is elaborated upon, and the model is trained to generate an effective allocation scheme. Comparative experiments are conducted on authentic datasets, aiming to assess performance indicators of the presented approach. The findings demonstrate a substantial enhancement in performance for the IAM-IPPO algorithm compared to other baselines, which is helpful in exploring excellent design schemes of the crowdsourcing SoSs, especially in dynamic large-scale cases.

Data-driven dynamic pricing and inventory management of an omni-channel retailer in an uncertain demand environment

In recent years, omni-channel retailing has become immensely popular among both retailers and consumers. In this approach, retailers often leverage their brick-and-mortar stores to fulfill online orders, leading to the need for simultaneous decision-making on replenishment and inventory rationing. This inventory strategy presents significant complexities in traditional dynamic pricing and inventory management problems, particularly in unpredictable market environments. Therefore, we have developed a dynamic pricing, replenishment, and rationing model for omni-channel retailers using a two-level partially observed Markov decision process to visualize the dynamic process. We propose to use a deep reinforcement learning algorithm, called Maskable LSTM-Proximal Policy Optimization (ML-PPO), which integrates the current observations and future predictions as input to the agent and uses the invalid action mask to guarantee the allowable actions. Our simulation experiments have demonstrated the ML-PPO's efficiency in maximizing retailer profit and service level, along with its generalized ability to tackle dynamic pricing and inventory management problems.

Model-Based Predictive Control and Reinforcement Learning for Planning Vehicle-Parking Trajectories for Vertical Parking Spaces.

This paper proposes a vehicle-parking trajectory planning method that addresses the issues of a long trajectory planning time and difficult training convergence during automatic parking. The process involves two stages: finding a parking space and parking planning. The first stage uses model predictive control (MPC) for trajectory tracking from the initial position of the vehicle to the starting point of the parking operation. The second stage employs the proximal policy optimization (PPO) algorithm to transform the parking behavior into a reinforcement learning process. A four-dimensional reward function is set to evaluate the strategy based on a formal reward, guiding the adjustment of neural network parameters and reducing the exploration of invalid actions. Finally, a simulation environment is built for the parking scene, and a network framework is designed. The proposed method is compared with the deep deterministic policy gradient and double-delay deep deterministic policy gradient algorithms in the same scene. Results confirm that the MPC controller accurately performs trajectory-tracking control with minimal steering wheel angle changes and smooth, continuous movement. The PPO-based reinforcement learning method achieves shorter learning times, totaling only 30% and 37.5% of the deep deterministic policy gradient (DDPG) and twin-delayed deep deterministic policy gradient (TD3), and the number of iterations to reach convergence for the PPO algorithm with the introduction of the four-dimensional evaluation metrics is 75% and 68% shorter compared to the DDPG and TD3 algorithms, respectively. This study demonstrates the effectiveness of the proposed method in addressing a slow convergence and long training times in parking trajectory planning, improving parking timeliness.

Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games

Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym-μRTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks.

EMExplorer: an episodic memory enhanced autonomous exploration strategy with Voronoi domain conversion and invalid action masking

Autonomous exploration is a critical technology to realize robotic intelligence as it allows unsupervised preparation for future tasks and facilitates flexible deployment. In this paper, a novel Deep Reinforcement Learning (DRL) based autonomous exploration strategy is proposed to efficiently reduce the unknown area of the workspace and provide accurate 2D map construction for mobile robots. Different from existing human-designed exploration techniques that usually make strong assumptions about the scenarios and the tasks, we utilize a model-free method to directly learn an exploration strategy through trial-and-error interactions with complex environments. To be specific, the Generalized Voronoi Diagram (GVD) is first utilized for domain conversion to obtain a high-dimensional Topological Environmental Representation (TER). Then, the Generalized Voronoi Networks (GVN) with spatial awareness and episodic memory is designed to learn autonomous exploration policies interactively online. For complete and efficient exploration, Invalid Action Masking (IAM) is employed to reshape the configuration space of exploration tasks to cope with the explosion of action space and observation space caused by the expansion of the exploration range. Furthermore, a well-designed reward function is leveraged to guide the learning of policies. Extensive baseline tests and comparative simulations show that our strategy outperforms the state-of-the-art strategies in terms of map quality and exploration speed. Sufficient ablation studies and mobile robot experiments demonstrate the effectiveness and superiority of our strategy.

Solving job shop scheduling problems via deep reinforcement learning

Deep reinforcement learning (DRL), as a promising technique, is a new approach to solve the job shop scheduling problem (JSSP). Although DRL method is effective for solving JSSP, there are still deficiencies in state representation, action space definition, and reward function design, which make it difficult for the agent to learn effective policy. In this paper, we model JSSP as a Markov decision process (MDP) and design a new state representation using the state features of bidirectional scheduling, which can not only enable the agent to capture more effective state information, improve its decision-making ability, but also effectively avoid the phenomenon of multiple optimal action selections in candidate action set. Invalid action masking (IAM) technique is employed to narrow the search space, which helps the agent avoid exploring suboptimal solutions. We evaluate the performance of the policy model on eight public test datasets: ABZ, FT, ORB, YN, SWV, LA, TA, and DMU. Extensive experimental results show that the proposed method on the whole has better optimization ability than the existing state-of-the-art models and priority dispatching rules.

A Multi-Agent Deep Reinforcement Learning-Based Popular Content Distribution Scheme in Vehicular Networks.

The Internet of Vehicles (IoV) enables vehicular data services and applications through vehicle-to-everything (V2X) communications. One of the key services provided by IoV is popular content distribution (PCD), which aims to quickly deliver popular content that most vehicles request. However, it is challenging for vehicles to receive the complete popular content from roadside units (RSUs) due to their mobility and the RSUs' constrained coverage. The collaboration of vehicles via vehicle-to-vehicle (V2V) communications is an effective solution to assist more vehicles to obtain the entire popular content at a lower time cost. To this end, we propose a multi-agent deep reinforcement learning (MADRL)-based popular content distribution scheme in vehicular networks, where each vehicle deploys an MADRL agent that learns to choose the appropriate data transmission policy. To reduce the complexity of the MADRL-based algorithm, a vehicle clustering algorithm based on spectral clustering is provided to divide all vehicles in the V2V phase into groups, so that only vehicles within the same group exchange data. Then the multi-agent proximal policy optimization (MAPPO) algorithm is used to train the agent. We introduce the self-attention mechanism when constructing the neural network for the MADRL to help the agent accurately represent the environment and make decisions. Furthermore, the invalid action masking technique is utilized to prevent the agent from taking invalid actions, accelerating the training process of the agent. Finally, experimental results are shown and a comprehensive comparison is provided, which demonstrates that our MADRL-PCD scheme outperforms both the coalition game-based scheme and the greedy strategy-based scheme, achieving a higher PCD efficiency and a lower transmission delay.

Cooperative Data Collection With Multiple UAVs for Information Freshness in the Internet of Things

Maintaining the freshness of information in the Internet of Things (IoT) is a critical yet challenging problem. In this paper, we study cooperative data collection using multiple Unmanned Aerial Vehicles (UAVs) with the objective of minimizing the total average Age of Information (AoI). We consider various constraints of the UAVs, including kinematic, energy, trajectory, and collision avoidance, in order to optimize the data collection process. Specifically, each UAV, which has limited on-board energy, takes off from its initial location and flies over sensor nodes to collect update packets in cooperation with the other UAVs. The UAVs must land at their final destinations with non-negative residual energy after the specified time duration to ensure they have enough energy to complete their missions. It is crucial to design the trajectories of the UAVs and the transmission scheduling of the sensor nodes to enhance information freshness. We model the multi-UAV data collection problem as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), as each UAV is unaware of the dynamics of the environment and can only observe a part of the sensors. To address the challenges of this problem, we propose a multi-agent Deep Reinforcement Learning (DRL)-based algorithm with centralized learning and decentralized execution. In addition to the reward shaping, we use action masks to filter out invalid actions and ensure that the constraints are met. Simulation results demonstrate that the proposed algorithms can significantly reduce the total average AoI compared to the baseline algorithms, and the use of the action mask method can improve the convergence speed of the proposed algorithm.

A reinforcement learning-based approach for online bus scheduling

Bus Scheduling Problem (BSP) is vital to save operational cost and ensure service quality. Existing approaches typically generate a bus scheduling scheme in an offline manner and then schedule vehicles according to the scheme. In practice, uncertain events such as traffic congestion occur frequently, which may make the originally planned bus scheduling scheme infeasible. This study proposes a Reinforcement Learning-based Bus Scheduling Approach (RL-BSA) for online bus scheduling. In RL-BSA, each departure time in a bus timetable is regarded as a decision point, and an agent makes a decision at the departure time to select a vehicle to depart at the time. The BSP is modeled as a Markov Decision Process (MDP) for the first time in literature. The state features are devised, which consist of real-time information of vehicles, including remaining working time, remaining driving time, rest time, number of executed trips and vehicle type. A reward function combining a final reward and a step-wise reward is devised. An invalid action masking approach is used to avoid the agent from selecting vehicles not meeting constraints. The agent is trained by interacting with a simulation environment and then the trained agent can schedule vehicles in an online manner. Experiments on real-world BSP instances show that RL-BSA can significantly reduce the number of vehicles used compared with the manual scheduling approach and Adaptive Large Neighborhood Search (ALNS). Under uncertain environment, RL-BSA can cover all departure times in the timetable without increasing the number of vehicles used.

Vehicular intelligent collaborative intersection driving decision algorithm in Internet of Vehicles

Connected and Automated Vehicles (CAV) are regarded as the developing trend of future transportation due to the advantages in terms of increasing traffic throughput, safety, and reducing energy consumption. One of the challenging CAV research problems is intelligent collaborative driving decision in dynamically changing traffic scenarios. The sixth-generation mobile communication technology (6G) plays an important role in attaining CAV intelligent driving in Internet of Vehicles (IoV) by provide significantly higher system performance, higher spectral efficiency, higher reliability, and higher security. This paper formulates the collaborative decision problem of CAVs at unsignalized intersections as a multi-agent reinforcement learning (MARL) problem, where CAVs entering an intersection safely cross the intersection and minimize their traffic time through collaborating to learn a strategy. An efficient and scalable MARL algorithm based on Proximal Policy Optimization (PPO) is applied to the dynamic intersection scenarios, where the parameter-sharing mechanism is used to improve the PPO algorithm to be a multi-agent version. Instead of the global reward, the local reward is used to promote cooperation between agents and achieve scalability. In addition, the action masking mechanism is adopted to improve the learning efficiency by filtering out invalid actions at each step. A joint simulation platform is built to verify the performance of the Parameter-Sharing PPO algorithm (PS-PPO). The simulation results show that the PS-PPO algorithm not only guarantees a low collision rate but also greatly reduces the vehicular travel time. This algorithm promotes cooperation among CAVs and performs well in the task of collaborative traffic flow at unsignalized intersections, especially in high-traffic volume scenarios. It is helpful to improve the safety and efficiency of traffic flow at unsignalized intersections.

TD3-Based EMS Using Action Mask and Considering Battery Aging for Hybrid Electric Dump Trucks

The hybrid electric dump truck is equipped with multiple power sources, and each powertrain component is controlled by an energy management strategy (EMS) to split the demanded power. This study proposes an EMS based on deep reinforcement learning (DRL) algorithm to extend the battery life and reduced total usage cost for the vehicle, namely the twin delayed deep deterministic policy gradient (TD3) based EMS. Firstly, the vehicle model is constructed and the optimization objective function, including battery aging cost and fuel consumption cost, is designed. Secondly, the TD3-based EMS is used for continuous action control of ICE power based on vehicle state, and the action mask is applied to filter out invalid actions. Thirdly, the simulations of the EMSs are trained under the CHTC-D driving cycle and C-WTVC driving cycle. The results show that the action mask improves the convergence efficiency of the strategies, and the proposed TD3-based EMS outperforms the deep deterministic policy gradient (DDPG) based EMS. Meanwhile, the battery life is extended by 36.17% under CHTC-D and 35.49% under C-WTVC, and the total usage cost is reduced by 4.30% and 2.49% when the EMS considers battery aging. In summary, the proposed TD3-based EMS can extend the battery life and reduce usage cost, and provides a method to solve the optimization problem for the EMS of hybrid power systems.

A Deep Reinforcement Learning Approach for Optimal Scheduling of Heavy-haul Railway

This paper presents an approach based on deep reinforcement learning to solve the combinatorial optimization problem of heavy-haul railway scheduling. The problem is modeled as a Markov decision process. Considering that the problem is characterized by complex process and strong constraints, a heavy-haul railway simulation environment based on discrete event scheduling is established, in which the optimization objective calculation and complex constraints can be achieved naturally. A reinforcement learning algorithm based on two-stage action sampling is proposed to alleviate the curse of dimensionality, in which invalid actions are masked out through staged action masking. The Effectiveness of the approach is verified by the experimental results.

A Novel Hybrid-ARPPO Algorithm for Dynamic Computation Offloading in Edge Computing

Applications consisting of a group of modular tasks can be offloaded to the multiaccess edge computing (MEC) for lower delay and energy consumption. In a dynamic MEC system, the fine-grained cooperative and dynamic offloading solution is necessary for the scenario of reusing tasks among devices. Considering the transmission cooperation, shared wireless bandwidth, and changing task queues on devices and edge servers, we formulate a joint offloading optimization problem to minimize the long-term average task execution cost. Although deep reinforcement learning (DRL) is a popular method for the dynamic problem, existing DRL algorithms are not suitable for our problem because of the hybrid discrete-continuous action spaces and constraints among action dimensions. Therefore, we propose a hybrid average reward proximal policy optimization (hybrid-ARPPO) algorithm to jointly optimize the offloading decisions, cooperative transmission ratios, and edge server assignments. First, we decompose our offloading problem into two subproblems. One is a tractable linear programming problem for continuous transmission ratios, and the other is a Markov decision process (MDP) only with discrete actions for offloading decisions and server assignments. Second, we take the expected average reward as the performance measure and deprecate the discount factor, which can reduce the work of tuning algorithms. Third, we design an action mask layer in the policy network of hybrid-ARPPO to filter invalid actions. Extensive experiments show the effectiveness of our hybrid-ARPPO in different system scales and task arrival patterns.

Global path planning algorithm based on double DQN for multi-tasks amphibious unmanned surface vehicle

It is a key to making path planning for an amphibious unmanned surface vehicle (USV). A global path planning algorithm based on double deep Q networks (DDQN) is proposed. Firstly, an environment model is constructed by an electronic nautical chart and elevation map to train and verify the algorithm. Secondly, based on the kinematics of amphibious USV, a Markov decision process (MDP) framework is built, and various reward functions are designed for diverse tasks. During the training, obstacles and water depth information of the environment are used, the amphibious USV agent is guided to the target area. Meanwhile, based on the prior knowledge, an action mask approach is integrated to deal with the invalid actions generated by the amphibious USV. Path smoothing is also integrated to smooth the path. According to different criteria, reasonable paths can be generated and adjusted by the weights of the reward function. To verify our algorithm, a small-scale simulation environment is established, and two scenarios are introduced. The results show that our DDQN algorithm can generate reasonable global paths for diverse tasks. Moreover, compared with DQN, A*, and RRT algorithms, the paths generated by our method have better performance.

Techniques for applying reinforcement learning to routing and wavelength assignment problems in optical fiber communication networks

We propose a novel application of reinforcement learning (RL) with invalid action masking and a novel training methodology for routing and wavelength assignment (RWA) in fixed-grid optical networks and demonstrate the generalizability of the learned policy to a realistic traffic matrix unseen during training. Through the introduction of invalid action masking and a new training method, the applicability of RL to RWA in fixed-grid networks is extended from considering connection requests between nodes to servicing demands of a given bit rate, such that lightpaths can be used to service multiple demands subject to capacity constraints. We outline the additional challenges involved for this RWA problem, for which we found that standard RL had low performance compared to that of baseline heuristics, in comparison with the connection requests RWA problem considered in the literature. Thus, we propose invalid action masking and a novel training method to improve the efficacy of the RL agent. With invalid action masking, domain knowledge is embedded in the RL model to constrain the action space of the RL agent to lightpaths that can support the current request, reducing the size of the action space and thus increasing the efficacy of the agent. In the proposed training method, the RL model is trained on a simplified version of the problem and evaluated on the target RWA problem, increasing the efficacy of the agent compared with training directly on the target problem. RL with invalid action masking and this training method outperforms standard RL and three state-of-the-art heuristics, namely, k shortest path first fit, first-fit k shortest path, and k shortest path most utilized, consistently across uniform and nonuniform traffic in terms of the number of accepted transmission requests for two real-world core topologies, NSFNET and COST–239. The RWA runtime of the proposed RL model is comparable to that of these heuristic approaches, demonstrating the potential for real-world applicability. Moreover, we show that the RL agent trained on uniform traffic is able to generalize well to a realistic nonuniform traffic distribution not seen during training, thus outperforming the heuristics for this traffic. Visualization of the learned RWA policy reveals an RWA strategy that differs significantly from those of the heuristic baselines in terms of the distribution of services across channels and the distribution across links.