Policy Networks Research Articles

The Use of continuous action representations to scale deep reinforcement learning for inventory control

Abstract Deep reinforcement learning (DRL) can solve complex inventory problems with a multi-dimensional state space. However, most approaches use a discrete action representation and do not scale well to problems with multi-dimensional action spaces. We use DRL with a continuous action representation for inventory problems with a large (multi-dimensional) discrete action space. To obtain feasible discrete actions from a continuous action representation, we add a tailored mapping function to the policy network that maps the continuous outputs of the policy network to a feasible integer solution. We demonstrate our approach to multi-product inventory control. We show how a continuous action representation solves larger problem instances and requires much less training time than a discrete action representation. Moreover, we show its performance matches state-of-the-art heuristic replenishment policies. This promising research avenue might pave the way for applying DRL in inventory control at scale and in practice.

Strategies for Strengthening Network Capacity in the Implementation of Food Security Policies at the Local Level in Indonesia

This study aims to analyze network management strategies in the interactive governance perspective in the implementation of food security policies in North Luwu Regency, Indonesia. The complexity of food security issues requires effective collaboration and interaction among various actors in the policy network. Using a qualitative approach with an explanatory case study design, this research examines seven indicators of network management strategies: network (de-)activating, constitutional reform, (de)coupling games, changing incentives, changing internal structure and position of actors, changing relations, and management by chaos. Data were collected through in-depth interviews, participatory observation, and document analysis, involving relevant stakeholders from government agencies, farmer organizations, and private sector partners. The findings reveal that the application of network management strategies facilitates interaction among network actors and contributes to the success of interactive governance in implementing food security policies. Network (de-)activating involves the engagement of new coalitions and resource mobilization to strengthen cooperation and interaction. Constitutional reform through changes in rules and resources helps develop coordination mechanisms and strengthen policy integration. (De) coupling games demonstrate the separation of strategies carried out by various actors to support a holistic approach in achieving food security. Changing incentives through incentive interventions can increase motivation and active participation of actors in realizing food security. Changing relations indicate changes in cooperative relationships and actor roles to enhance effective collaboration. Management by chaos through the application of principles of flexibility, adaptability, and the ability to manage uncertainty can strengthen interactive governance in facing complex and dynamic challenges. This study recommends strengthening cross-sectoral collaboration, developing a structured incentive system, enhancing the capacity of farmer organizations, and optimizing the use of information technology to improve the success of interactive governance in food security policy implementation.

Tensions, Challenges, and Opportunities: Internal Networks Supporting Gender Equality Policy Processes and Structural Change in Universities

Abstract This article examines the role of internal networks in supporting gender equality (GE) units and GE policy processes within universities, focusing on the tensions, challenges, and opportunities that can be encountered. Using two illustrative cases from Spain and employing a qualitative approach, it explores how these networks, bridging top-down and bottom-up approaches, navigate the balance between centralized and decentralized approaches in managing and implementing GE plans and the impact of institutionalization. Research has examined GE policy networks and women’s academic participation, but within-university GE networks are less studied. Tensions arise between institutionalization and feminist activism, coordinated action and idea generation, and local ownership and change agents’ limited authority. Challenges include obtaining institutional recognition while maintaining feminist sensibility, emphasizing the importance of nurturing networks. We highlight the productive power of resistance and the need to capitalize on context-specific opportunities for driving structural change in universities.

Adversarial imitation learning with deep attention network for swarm systems

AbstractSwarm systems consist of a large number of interacting individuals, which exhibit complex behavior despite having simple interaction rules. However, crafting individual motion policies that can manifest desired collective behaviors poses a significant challenge due to the intricate relationship between individual policies and swarm dynamics. This paper addresses this issue by proposing an imitation learning method, which derives individual policies from collective behavior data. The approach leverages an adversarial imitation learning framework, with a deep attention network serving as the individual policy network. Our method successfully imitates three distinct collective behaviors. Utilizing the ease of analysis provided by the deep attention network, we have verified that the individual policies underlying a certain collective behavior are not unique. Additionally, we have analyzed the different individual policies discovered. Lastly, we validate the applicability of the proposed method in designing policies for swarm robots through practical implementation on swarm robots.

USLC: Universal self‐learning control via physical performance policy‐optimization neural network

AbstractThis article proposes an online universal self‐learning control (USLC) algorithm based on a physical performance policy‐optimization neural network, which aims to solve the problem of universal self‐learning optimal control laws for nonlinear systems with various uncertain dynamics. As a key system characterization, this algorithm predicts the discrepancy between the optimal and current control laws by evaluating overall performance in each iterative learning cycle, leveraging an offline‐trained universal policy network. This approach is universal, as it does not rely on an exact system model and can adaptively control performance preferences across various tasks by customizing the physical performance cost weights. Using the established control law‐performance surface and contraction Lyapunov function, the necessary assumptions and proofs for the stable convergence of the system within a three‐dimensional manifold space are provided. To demonstrate the universality of USLC, simulation experiments are conducted on two different systems: a low‐order circuit system and a high‐order variable‐span aircraft attitude control system. The stable control achieved under varying initial values and boundary conditions in each system illustrates the effectiveness of the proposed method. Finally, the limitations of this study are discussed.

Failing to learn from teachers: the Social Work Early Career Framework in England

This article presents a critical analysis of the Social Worker Early Career Framework in England, planned for national roll-out in September 2026 to replace the Assessed and Supported Year in Employment for children and family social workers. The Social Worker Early Career Framework is based on the existing Teacher Early Career Framework in England, launched nationally in 2021. The article argues that both the problematic experience of its counterpart in teaching and the existing evidence around similar initiatives already implemented in social work suggest that the Social Worker Early Career Framework is highly unlikely to achieve its stated aims. These discussions are followed by a consideration of why the Social Worker Early Career Framework is being introduced despite this existing evidence. It is suggested that a relatively small, opaque and overlapping policy network dominating children’s services reform in England provides a possible explanation.

Model of Policy Network in Advocacy for the Formulation of Qanun Draft on Protection and Fulfillment of the Rights of People with Disabilities

This study aims to analyze the interaction and interdependence of policy networks, identify inhibiting and supporting factors, and formulate an effective policy network model in the formulation of the Qanun for the protection and fulfillment of the rights of persons with disabilities in North Aceh Regency. A qualitative approach was used in this study, with data collection methods including in-depth interviews, focus group discussions (FGD), observations, and document studies.The results of the study indicate that the interaction and interdependence among actors in the policy network in North Aceh Regency are still not optimal. The involved actors often work separately without adequate coordination, and differences in interests and priorities hinder the achievement of common goals. The main inhibiting factors include a lack of resources, differences in interests among actors, and low public awareness of the rights of persons with disabilities. On the other hand, supporting factors include the commitment of the local government, support from NGOs, and initiatives from the community of persons with disabilities.An effective policy network model includes enhancing coordination among actors, establishing regular communication forums, strengthening the capacity of involved actors, and active participation from the community of persons with disabilities. By implementing this model, it is expected that the formulation of the Qanun for the protection and fulfillment of the rights of persons with disabilities in North Aceh Regency can proceed more effectively and inclusively.

Message Action Adapter Framework in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has demonstrated significant potential in enabling cooperative agents. The communication protocol, which is responsible for message exchange between agents, is crucial in cooperation. However, communicative MARL systems still face challenges due to the noisy messages in complex multi-agent decision processes. This issue often stems from the entangled representation of observations and messages in policy networks. To address this, we propose the Message Action Adapter Framework (MAAF), which first trains individual agents without message inputs and then adapts a residual action based on message components. This separation isolates the effect of messages on action inference. We explore how training the MAAF framework with model-agnostic message types and varying optimization strategies influences adaptation performance. The experimental results indicate that MAAF achieves competitive performance across multiple baselines despite utilizing only half of the available communication, and shows an average improvement of 7.6% over the full attention-based communication approach. Additional findings suggest that different message types result in significant performance variations, emphasizing the importance of environment-specific message types. We demonstrate how the proposed architecture separates communication channels, effectively isolating message contributions.

The pupil outdoes the master: Imperfect demonstration-assisted trust region jamming policy optimization against frequency-hopping spread spectrum

Jamming decision-making is a pivotal component of modern electromagnetic warfare, wherein recent years have witnessed the extensive application of deep reinforcement learning techniques to enhance the autonomy and intelligence of wireless communication jamming decisions. However, existing researches heavily rely on manually designed customized jamming reward functions, leading to significant consumption of human and computational resources. To this end, under the premise of obviating designing task-customized reward functions, we propose a jamming policy optimization method that learns from imperfect demonstrations to effectively address the complex and high-dimensional jamming resource allocation problem against frequency hopping spread spectrum (FHSS) communication systems. To achieve this, a policy network is meticulously architected to consecutively ascertain jamming schemes for each jamming node, facilitating the construction of the dynamic transition within the Markov decision process. Subsequently, anchored in the dual-trust region concept, we design policy improvement and policy adversarial imitation phases. During the policy improvement phase, the trust region policy optimization method is utilized to refine the policy, while the policy adversarial imitation phase employs adversarial training to guide policy exploration using information embedded in demonstrations. Extensive simulation results indicate that our proposed method can approximate the optimal jamming performance trained under customized reward functions, even with rough binary reward settings, and also significantly surpass demonstration performance.

Real-Time Policy Optimization for UAV Swarms Based on Evolution Strategies

Multi-agent decision-making faces many challenges such as non-stationarity and sparse rewards, while the complexity and randomness of the real environment further complicate policy development. This paper addresses the high-dimensional policy optimization problems of unmanned aerial vehicle (UAV) swarms. By modeling the problem scenario as a Markov decision process, a real-time policy optimization algorithm based on evolution strategy (ES) pre-training is proposed. This approach combines decision-time planning with background planning to evaluate and integrate different sets of policy parameters in a temporal context. In the experimental phase, the policy network is trained using both ES and REINFORCE algorithms on a constructed simulation platform. Comparative experiments demonstrate the effectiveness of using ES for policy pre-training. Finally, the proposed real-time policy optimization algorithm further improves the performance of the swarm by approximately 10% in simulations, offering a feasible solution for adversarial games between swarms and extending the research scope of evolutionary algorithms.

Whole of prison approach to vaccination: co-development of information materials for people living in prison

Abstract Enhancing vaccination literacy is pivotal in addressing hesitancy, especially among incarcerated populations who may lack regular access to health information, thus ensuring health equity. Supported by the European Union, the RISE-Vac project endeavours to bolster vaccine literacy, accessibility, and uptake within European prisons. To discern vaccination knowledge gaps and preferences for information dissemination, consultations were conducted with incarcerated individuals in the United Kingdom (via the Prisoner Policy Network), France, and Moldova. A total of 344 responses were garnered: 224 from the UK, 70 from France, and 50 from Moldova. In addition a focus group was conducted in the UK. The key themes that emerged were common across the 3 participating countries: views of vaccination, prior knowledge about vaccines, areas of appetite for learning, availability of reliable information, and preferred mechanism for information sharing. Participants were particularly interested in learning about the effectiveness, side effects, and manufacturing of vaccines. Their invaluable input informed the creation of educational materials, notably a brochure slated for pilot testing in European prisons and a video clip. Notably, individuals with lived prison experience were actively engaged at every phase of this initiative, ensuring its relevance and efficacy.

Nonlinear Reinforcement Learning-Based Dynamic Test Case Prioritization with Anomaly Detection for Continuous Integration

This research article introduces DynamicR-TCP, a nonlinear approach to dynamic test case prioritization (TCP) leveraging reinforcement learning and anomaly detection. Designed to enhance the efficiency of software testing in continuous integration (CI) environments, the proposed model dynamically adapts to changing conditions by learning complex patterns from historical test data. A reinforcement learning agent, employing policy and value networks, guides nonlinear prioritization by optimizing the sequence of test case executions. The model integrates a sliding window strategy for adaptive focus on smaller test subsets and uses an experience replay buffer for iterative learning. An anomaly detection layer monitors test outcomes for deviations, triggering retraining when needed to maintain reliability. Experimental evaluation using the Defects4J dataset demonstrates the model’s superior fault detection rates, faster time to first fault, and higher computational efficiency compared to contemporary methods. An ablation study highlights the nonlinear synergy among the key components—reinforcement learning, sliding windows, and anomaly detection—showing significant performance gains when combined. This comprehensive framework advances software testing by providing an adaptive, efficient, and scalable solution, ensuring robust performance in dynamic CI environments.

Simulation of Coherent Excavator Operations in Earthmoving Tasks Based on Reinforcement Learning

Earthwork operations are critical to construction projects, with their safety and efficiency influenced by factors such as operator skill and working hours. Pre-construction simulation of these operations is essential for optimizing outcomes, providing key training for operators and improving safety awareness and operational efficiency. This study introduces a hierarchical cumulative reward mechanism that decomposes complex operational behaviors into simple, fundamental actions. The mechanism prioritizes reward function design elements, including order, size, and form, thus simplifying excavator operation simulation using reinforcement learning (RL) and enhancing policy network reusability. A 3D model of a hydraulic excavator was constructed with six degrees of freedom—comprising the boom, arm, bucket, base, and left/right tracks. The Proximal Policy Optimization (PPO) algorithm was applied to train four basic behaviors: scraping, digging, throwing, and turning back. Motion simulation was successfully achieved using diggable terrain resources. Results demonstrate that the simulated excavator, powered by RL neural networks, can perform coordinated actions and maintain smooth operational performance. This research offers practical implications by rapidly illustrating the full operational process before construction, delivering immersive movies, and enhancing worker safety and operational efficiency.

Reinforcement learning-based portfolio optimization with deterministic state transition

Portfolio optimization has attracted substantial interest within the artificial intelligence community due to its significant impact on financial decision-making, risk management, and market analysis. Reinforcement learning fits well with portfolio optimization because their goal is to maximize cumulative returns. In reinforcement learning, state transition probabilities are often unknown and must be estimated. However, in portfolio backtesting experiments, these probabilities are deterministic, making the conventional reinforcement learning approach to estimating state transitions suboptimal for portfolio optimization. Addressing this issue, this study decomposes the portfolio optimization into two core tasks: prediction and profit policy optimization and proposes a novel reinforcement learning framework that assumes deterministic state transition probabilities, comprised of three main modules: feature extraction, prediction, and profit strategy optimization. To model assets more effectively and comprehensively, we capture their temporal features, relational features, and market state. We introduce a patch-wise correlation method and attribute based gate to enhance feature extraction. In the profit policy module, we utilize a deterministic strategy, employing a recursive reinforcement learning method based on Monte Carlo sampling to train the policy network. This enables dynamic adjustments of asset investment weights, ensuring the maximization of cumulative returns. Extensive experiments conducted on cryptocurrency datasets demonstrate the superior performance of our approach, and achieving 36.6%-75.6% improvements in main measurements on cryptocurrency datasets.

CLlight: Enhancing representation of multi-agent reinforcement learning with contrastive learning for cooperative traffic signal control

Multi-agent reinforcement learning has shown great potential for coordinating multi-intersection traffic signals due to its powerful adaptive capabilities, treating each intersection as an agent. However, in the real world, different intersections possess differentiating characteristics such as unique vehicle distributions and traffic patterns. Most existing methods directly add neighboring intersection states to local intersections and optimize the cooperative policy network based on synthesized global features. This indirect optimization approach makes it difficult to thoroughly explore the mutual interactions among different intersection agents, preventing agents from truly learning features with cooperative awareness. To resolve these challenges, we introduce contrastive learning as representation task to the multi-intersection traffic signal control approach named CLlight for two-stage policy network updating. In the first stage, we utilize policy-based or actor-critic-based reinforcement learning methods such as A2C, SAC, and PPO to train policy networks with certain representational capabilities. In the second stage, by extracting pre- and post-masked features and reconstructing the post-masked features, the agents are encouraged to learn the similarities and differences between different intersection policies, which in turn enhances the cooperative and individual representation capabilities of the policy network. To the best of our knowledge, this is the first application of contrastive learning in the field of traffic signal control. Experimental results demonstrate, compared to other state-of-the-art traffic signal control methods, superior average travel time and average waiting time performance under various scenarios, tested on synthetic and real-world datasets.

Learning to construct a solution for UAV path planning problem with positioning error correction

Unmanned aerial vehicles (UAVs) are advanced flight systems. However, their positioning systems cause distance-dependent errors during flight. This study seeks to solve the UAV path planning problem with positioning error correction (UPEC) with an end-to-end method. Traditional methods struggle to balance solution quality and computational overload, and often have limited utilisation of scenario information. To overcome these issues, we propose a path planning model (PPM) based on deep reinforcement learning to solve the UPEC. The model has a complete structure that includes a mathematical model, feature engineering, solution process, neural policy network, scenario generation, training process, and test solution mechanism. Specifically, we first establish a Markov decision process (MDP) for UPEC and apply feature engineering with effective features to support decision-making. We then introduce a path planning neural network (PPNN) to represent the MDP policy. Based on the dataset generated from the multi-rule combination validation, we train the PPNN using the proposed RL algorithm with storage pool. Furthermore, we propose a backtracking mechanism to guarantee solution feasibility during the construction process. Extensive experiments demonstrate that the proposed PPM outperforms existing state-of-the-art algorithms in terms of solution quality and timeliness, and the backtracking mechanism effectively improves the scenario completion rate. The model study indicates the efficacy of our training algorithm and the generalisation of the PPNN. Additionally, our construction process is problem-tailored and more suitable for addressing UPEC than iterative search algorithms, because it effectively mitigates the impact of invalid nodes.

Efficient Multitask Reinforcement Learning Without Performance Loss.

We propose an iterative sparse Bayesian policy optimization (ISBPO) scheme as an efficient multitask reinforcement learning (RL) method for industrial control applications that require both high performance and cost-effective implementation. Under continual learning scenarios in which multiple control tasks are sequentially learned, the proposed ISBPO scheme preserves the previously learned knowledge without performance loss (PL), enables efficient resource use, and improves the sample efficiency of learning new tasks. Specifically, the proposed ISBPO scheme continually adds new tasks to a single policy neural network while completely preserving the control performance of previously learned tasks through an iterative pruning method. To create a free-weight space for adding new tasks, each task is learned through a pruning-aware policy optimization method called the sparse Bayesian policy optimization (SBPO), which ensures efficient allocation of limited policy network resources for multiple tasks. Furthermore, the weights allocated to the previous tasks are shared and reused in new task learning, thereby improving sample efficiency and the performance of new task learning. Simulations and practical experiments demonstrate that the proposed ISBPO scheme is highly suitable for sequentially learning multiple tasks in terms of performance conservation, efficient resource use, and sample efficiency.

‘Embers, and fragments’: social haunting in youth work, impact measurement and policy networks

ABSTRACT How do education policy processes and networks operate in sectors that have been chronically defunded? This article discusses elements of contestation and doubt in the context of a policy agenda around impact measurement in youth services in England. Drawing on interviews with policy makers, influencers and critics, it combines analysis of a policy network with theories and methodologies of social haunting, as developed by Avery Gordon, Justine Pors and others. Youth policy, in its present denuded form, is haunted by ghosts of policy past: a mourned tradition of publicly funded, practice-led youth work that refuses to disappear; and spectres of policy yet-to-come: an idealised yet somehow unconvincing future of evidence-based practice and social return on investment. We suggest that engaging with ghosts can enrich understandings of policy making by foregrounding the discomforts that haunt policy networks, questioning the assumed rationality of evidence-based policy and highlighting the contested and interconnected nature of policy and practice futures.

Dynamic Routing and Knowledge Re-Learning for Data-Free Black-Box Attack.

Deep learning models have emerged as strong and efficient tools that can be applied to a broad spectrum of complex learning problems and many real-world applications. However, more and more works show that deep models are vulnerable to adversarial examples. Compared to vanilla attack settings, this paper advocates a more practical setting of data-free black-box attack, for which the attackers can completely not access the structures and parameters of the target model, as well as the intermediate features and any training data associated with the model. To tackle this task, previous methods generate transferable adversarial examples from a transparent substitute model to the target model. However, we found that these works have the limitations of taking static substitute model structure for different targets, only using hard synthesized examples once, and still relying on data statistics of the target model. This may potentially harm the performance of attacking the target model. To this end, we propose a novel Dynamic Routing and Knowledge Re-Learning framework (DraKe) to effectively learn a dynamic substitute model from the target model. Specifically, given synthesized training samples, a dynamic substitute structure learning strategy is proposed to adaptively generate optimal substitute model structure via a policy network according to different target models and tasks. To facilitate the substitute training, we present a graph-based structure information learning to capture the structural knowledge learned from the target model. For the inherent limitation that online data generation can only be learned once, a dynamic knowledge re-learning strategy is proposed to adjust the weights of optimization objectives and re-learn hard samples. Extensive experiments on four public image classification datasets and one face recognition benchmark are conducted to evaluate the efficacy of our Drake. We can obtain significant improvement compared with state-of-the-art competitors. More importantly, our DraKe consistently achieves attack superiority for different target models (e.g., residual networks, and vision transformers), showing great potential for complex real-world applications.

Three-dimensional cooperative guidance with impact angle constraints via value-policy decomposed multi-agent reinforcement learning

For a three-dimensional impact angle-constrained time-coordination guidance problem, a value-policy decomposed multi-agent twin delayed deep deterministic (VPD-MATD3) policy gradient algorithm is proposed in this paper, and a temporal-spatial cooperative guidance law is designed consequently. The derived cooperative engagement kinematics is formulated as a value-policy decomposed partially observable Markov decision process. Value decomposition and policy decomposition are correspondingly proposed to address the two bottlenecks of reward coupling and policy coupling in 3D guidance, thus improving the training process. The training environment with multiple uncertainties, such as observation noise, maneuverability perturbation, autopilot time lag, switching communication topology, and random initialization, is subsequently constructed. Time-energy type reward functions are designed, incorporating energy consumption as a direct optimization indicator via a penalty term. A long short-term memory network layer is inserted inside the policy networks to suppress the negative effect of observation noise. The policy training curves verify the significant effect of the proposed algorithm in improving training convergence. The policy testing results illustrate the robustness and generalization of the VPD-MATD3 cooperative guidance law and its ability to cope with randomly switched communication topologies. Moreover, the comparative testing further reveals its time-energy suboptimal property.