Policy Learning Research Articles

A robotic skill transfer learning framework of dynamic manipulation for fabric placement

Placing fabric poses a challenge to robots since fabric with high dimensional configuration space can deform during manipulation. Existing methods for placing fabric mostly rely on static operations, which are inefficient and require a large workspace. Therefore, this study applies dynamic manipulation (manipulating uncontrollable parts of the fabric by swinging) to fabric placement, proposing a novel learning framework for robotic dynamic fabric placement skill learning and generalization. The proposed framework integrates reinforcement learning with imitation learning, leveraging expert demonstration data to guide and accelerate skill acquisition. Additionally, fabric characteristics are combined with imitation learning to enable the transfer and generalization of the learned policy to real-world environments The experiments suggest that the proposed framework is capable of achieving the placement tasks for a range of positions and fabrics. For success rate, the policy of the proposed framework ultimately achieves a flatness of exceeding 95% and a placement distance error of less than 2 mm. Moreover, the proposed approach is similar in operation time to the fastest method, while it can reduce the space required for manipulating the fabric by over 15%. Compared with other placement policies, it is promising because of its high accuracy, flexibility, efficiency, as well as adaptability.

Curious Explorer: a Provable Exploration Strategy in Policy Learning.

A coverage assumption is critical with policy gradient methods, because while the objective function is insensitive to updates in unlikely states, the agent may need improvements in those states to reach a nearly optimal payoff. However, this assumption can be unfeasible in certain environments, for instance in online learning, or when restarts are possible only from a fixed initial state. In these cases, classical policy gradient algorithms like REINFORCE can have poor convergence properties and sample efficiency. Curious Explorer is an iterative state space pure exploration strategy improving coverage of any restart distribution ρ. Using ρ and intrinsic rewards, Curious Explorer produces a sequence of policies, each one more exploratory than the previous one, and outputs a restart distribution with coverage based on the state visitation distribution of the exploratory policies. This paper main results are a theoretical upper bound on how often an optimal policy visits poorly visited states, and a bound on the error of the return obtained by REINFORCE without any coverage assumption. Finally, we conduct ablation studies with REINFORCE and TRPO in two hard-exploration tasks, to support the claim that Curious Explorer can improve the performance of very different policy gradient algorithms.

Advances in Deep Reinforcement Learning for Computer Vision Applications

Deep Reinforcement Learning (DRL) has become very popular for computer vision (CV), solving mostly visually complex environments with decision making and dynamic adaption to different situations. This article provides an introduction to the basic concepts of deep reinforcement learning with a special focus on their applications in computer vision tasks, including challenging problems and emerging solutions. It reviews various DRL algorithms such as Q-learning, policy gradient methods, and Actor-Critic models, explaining much of the modifications that are done to make it work on high-dimensional visual problems. Different applications of DRL in major CV applications like object detection, image segmentation, target tracking and image generation are reviewed to demonstrate the power as well as limitations of DRL in practice. More importantly, the novel paradigms such as hierarchical policy learning, adaptive reward design multi-task reinforcement learning and domain adaptation can be viewed as promising premises to improve model efficiency and generalizability in multiple scenarios. Finally, this paper mentions a few of the existing challenges like computational power cost and sample efficiency; as well as future paths for enhancing that can widen devout reinforcement learning in computer vision. Through this comprehensive overview, we aim to shed light on the promising synergies between DRL and CV, while identifying key areas for future research and application.

Acquiring New Knowledge Without Losing Old Ones for Effective Continual Dialogue Policy Learning

Acquiring New Knowledge Without Losing Old Ones for Effective Continual Dialogue Policy Learning

Long-Term and Short-Term Opponent Intention Inference for Football Multiplayer Policy Learning

Long-Term and Short-Term Opponent Intention Inference for Football Multiplayer Policy Learning

Comparative Analysis of Community Energy Projects: Policy Lessons for Kenya from the United Kingdom

Assuming a scope for policy learning across jurisdictions, this paper compared policies relevant to community energy projects (CEPs) in a developed country where they have been adopted (i.e. the UK), against policies in an emerging country where CEPs are scarce (i.e. Kenya). Through comparative analysis, successes and failures in the policies’ landscapes were examined, with the aim of distilling lessons that could enhance CEPs adoption in Kenya. Both countries have some similar core functional mechanisms in their policies, e.g. Feed-in-Tariffs, energy auctions, and grid access. These are demand-side in nature and create a market for CEPs while derisking investments to local communities. However, key differences exist, reflecting the levels of maturity in the energy mix and policy-making in either country. The UK has more demand-side and ‘implementation oriented’ policies, compared to Kenya. For Kenya, key lessons include incentives prioritising appropriate grant programs, Feed-in-Tariffs targeting CEPs, adopting Smart Export Guarantee, implementing grid connection agreements, and fostering centres for data and knowledge exchange. However, these require a laser-focus to make them bespoke to the Kenyan context. Our recommendations envisaged a carefully calibrated confluence of pull-push policy factors, involving: a CEPs-friendly policy landscape, CEPs targets under a long-term strategy, natural abundance of different RE sources, and, Kenya’s appetite for using different policy solutions for different needs. Two key areas for relevant research are worth mentioning: exploring where the equilibrium for cost-effective energy mix integrating CEPs lies, in Kenya’s future, and; empirically exploring systems-wide analyses to reveal the inter-dependencies and inter-relations of the various contextual and policy factors, aimed at CEPs as an outcome.

Comparing Nordic forest governance: Key informant perspectives

There is a growing call for comparative analyses of forest governance to facilitate knowledge exchange for the sustainable management of Nordic forest systems, addressing the needs of societal stakeholders in enhancing both the quality and quantity of forest resources. This study traces the development of nation-wide policy instruments implemented in four Nordic countries; Finland, Sweden, Norway and Denmark between 1970 and 2023, based on interviews among forest policy scholars and other research experts. Using an analytical framework that highlights the relations between policy intervention, change, impact, and learning, we compare central government practices and private actor initiatives in the Nordic countries that address the ecological, social, and economic values of forests. We find a common trend towards multifunctionality and soft governance as well as differences in the use of economic instruments and certification standardization. This study also considers insights into developments in Nordic forest governance, the enabling and impeding factors for past trends and outcomes and discusses the learning of this change towards equity in the multifunctional use of forests.

Accelerated Transfer Learning for Cooperative Transportation Formation Change via SDPA-MAPPO (Scaled Dot Product Attention-Multi-Agent Proximal Policy Optimization)

A method for cooperative transportation, which required formation change in a traveling environment, is gaining interest. Deep reinforcement learning is used in formation changes for multi-robot cases. The MADDPG (Multi-Agent Deep Deterministic Policy Gradient) method is popularly used for recognized environments. On the other hand, re-learning may be required in unrecognized circumstances by using the MADDPG method. Although the development of MADDPG using model-based learning and imitation learning has been applied to reduce learning time, it is unclear how the learning results are transferred when the number of robots changes. For example, in the GASIL-MADDPG (Generative adversarial self-imitation learning and Multi-agent Deep Deterministic Policy Gradient) method, how the results of three robot training can be transferred to the four robots’ neural networks is uncertain. Nowadays, Scaled Dot Product Attention (SDPA) has attracted attention and is highly impactful for its speed and accuracy in natural language processing. When transfer learning is combined with fast computation, the efficiency of edge-level re-learning is improved. This paper proposes a formation change algorithm that allows easy and fast multi-robot knowledge transfer using SDPA combined with MAPPO (Multi-Agent Proximal Policy Optimization), compared to other methods. This algorithm applies SDPA to multi-robot formation learning and performs fast learning by transferring the acquired knowledge of formation changes to a certain number of robots. The proposed algorithm is verified by simulating the robot formation change and was able to achieve dramatic high-speed learning capabilities. The proposed SDPA-MAPPO (Scaled Dot Product Attention-Multi-Agent Proximal Policy Optimization) learned 20.83 times faster than the Deep Dyna-Q method. Furthermore, using transfer learning from a three-robot to five-robot case, the method shows that the learning time can be reduced by about 56.57 percent. A scenario of three-robot to five-robot is chosen based on the number of robots often used in cooperative robots.

Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization

Abstract Training a task-oriented dialogue policy using deep reinforcement learning is promising but requires extensive environment exploration. The amount of wasted invalid exploration makes policy learning inefficient. In this paper, we define and argue that dead-end states are important reasons for invalid exploration. When a conversation enters a dead-end state, regardless of the actions taken afterward, it will continue in a dead-end trajectory until the agent reaches a termination state or maximum turn. We propose a Dead-end Detection and Resurrection (DDR) method that detects dead-end states in an efficient manner and provides a rescue action to guide and correct the exploration direction. To prevent dialogue policies from repeating errors, DDR also performs dialogue data augmentation by adding relevant experiences that include dead-end states and penalties into the experience pool. We first validate the dead-end detection reliability and then demonstrate the effectiveness and generality of the method across various domains through experiments on four public dialogue datasets.

Autonomous Robot Goal Seeking and Collision Avoidance in the Physical World: An Automated Learning and Evaluation Framework Based on the PPO Method

Mobile robot navigation is a critical aspect of robotics, with applications spanning from service robots to industrial automation. However, navigating in complex and dynamic environments poses many challenges, such as avoiding obstacles, making decisions in real-time, and adapting to new situations. Reinforcement Learning (RL) has emerged as a promising approach to enable robots to learn navigation policies from their interactions with the environment. However, application of RL methods to real-world tasks such as mobile robot navigation, and evaluating their performance under various training–testing settings has not been sufficiently researched. In this paper, we have designed an evaluation framework that investigates the RL algorithm’s generalization capability in regard to unseen scenarios in terms of learning convergence and success rates by transferring learned policies in simulation to physical environments. To achieve this, we designed a simulated environment in Gazebo for training the robot over a high number of episodes. The training environment closely mimics the typical indoor scenarios that a mobile robot can encounter, replicating real-world challenges. For evaluation, we designed physical environments with and without unforeseen indoor scenarios. This evaluation framework outputs statistical metrics, which we then use to conduct an extensive study on a deep RL method, namely the proximal policy optimization (PPO). The results provide valuable insights into the strengths and limitations of the method for mobile robot navigation. Our experiments demonstrate that the trained model from simulations can be deployed to the previously unseen physical world with a success rate of over 88%. The insights gained from our study can assist practitioners and researchers in selecting suitable RL approaches and training–testing settings for their specific robotic navigation tasks.

Learning in polycentric governance: Insights from the California Delta science enterprise

AbstractScience is critical for learning and adaptation of policy and governance systems. Increasingly, science is produced in the context of a science enterprise: a complex, polycentric institutional arrangement featuring multiple science forums and actors. The characteristics of these polycentric systems can influence whether and to what extent science supports policy‐relevant learning. Limited research, however, has examined how science enterprises function as polycentric systems and how they can be governed to support learning. Using a survey of actors involved in the science enterprise of the California Delta, we integrate the collective learning framework and ecology of games framework to analyze individual‐ and forum‐level drivers of perceived learning across the adaptive management cycle. The results suggest that social drivers such as leadership, trust, and engagement are most highly correlated with perceived learning. While science enterprise actors often perceive administrative and financial resource limitations, those constraints are less important for learning than social drivers.

Facilitating workplace learning in clinical practice, an evaluation among medical specialists

Purpose Training of medical students, interns and residents (junior professionals, JP) happens predominantly through workplace learning. This is supported by medical specialists (MS) and varies by MS, workplace, medical specialty, and social context. This study aimed to gain insights into the facilitation of workplace learning in clinical practice by MS. Method Thirteen semi-structured interviews were conducted with MS at the gynaecology and paediatrics department in an academic and three non-academic hospitals in the Netherlands. Participants were purposively sampled to be involved in the training of JP. Transcribed interviews were thematically analysed to describe facilitators and barriers for workplace learning. Results Workplace learning was interpreted as a broad concept, with some MS acknowledging its occurrence beyond created opportunities. Four themes revealed facilitators and barriers related to (1) Connection and alignment, (2) Attitude, (3) Organizational and learning dynamics, and (4) Policy and Logistics. Our research suggests MS’ beliefs and awareness regarding workplace learning guide their ability to recognize and utilize learning moments. Conclusions Our findings underscore the importance of a work environment for MS that is founded on interpersonal relationships, supportive logistics of healthcare institutions and a culture of active involvement within the team, all of which are essential for recognizing and utilizing learning opportunities.

AI and ML-based risk assessment of chemicals: predicting carcinogenic risk from chemical-induced genomic instability

Chemical risk assessment plays a pivotal role in safeguarding public health and environmental safety by evaluating the potential hazards and risks associated with chemical exposures. In recent years, the convergence of artificial intelligence (AI), machine learning (ML), and omics technologies has revolutionized the field of chemical risk assessment, offering new insights into toxicity mechanisms, predictive modeling, and risk management strategies. This perspective review explores the synergistic potential of AI/ML and omics in deciphering clastogen-induced genomic instability for carcinogenic risk prediction. We provide an overview of key findings, challenges, and opportunities in integrating AI/ML and omics technologies for chemical risk assessment, highlighting successful applications and case studies across diverse sectors. From predicting genotoxicity and mutagenicity to elucidating molecular pathways underlying carcinogenesis, integrative approaches offer a comprehensive framework for understanding chemical exposures and mitigating associated health risks. Future perspectives for advancing chemical risk assessment and cancer prevention through data integration, advanced machine learning techniques, translational research, and policy implementation are discussed. By implementing the predictive capabilities of AI/ML and omics technologies, researchers and policymakers can enhance public health protection, inform regulatory decisions, and promote sustainable development for a healthier future.

Propulsive landing of launchers’ first stages with Deep Reinforcement Learning

The planetary landing problem is gaining relevance in the space sector, spanning a wide range of applications from unmanned probes landing on other planetary bodies to reusable first and second stages of launcher vehicles. In the existing methodology there is a lack of flexibility in handling complex non-linear dynamics, in particular in the case of non-convexifiable constraints. It is therefore crucial to assess the performance of novel techniques and their advantages and disadvantages. The purpose of this work is the development of an integrated 6-DOF guidance and control approach based on reinforcement learning of deep neural network policies for fuel-optimal planetary landing control, specifically with application to a launcher first-stage terminal landing, and the assessment of its performance and robustness. 3-DOF and 6-DOF simulators are developed and encapsulated in MDP-like (Markov Decision Process) industry-standard compatible environments. Particular care is given in thoroughly shaping reward functions capable of achieving the landing both successfully and in a fuel-optimal manner. A cloud pipeline for effective training of an agent using a PPO reinforcement learning algorithm to successfully achieve the landing goal is developed.

The mediating role of societal context in translating policy learning into policy change: the case of Iranian police

ABSTRACT Policy learning is a common driver of reform in public organisations and relies in part on institutional legitimacy. For law enforcement, institutional legitimacy is particularly salient given the police’s monopoly on the legal use of force. This study uses the concept of policy learning to analyse power and reform in policing amidst civil unrest. Examining public protests in Iran, the study finds that flaws in management structures and protocols compromise the legitimacy of the police and weaken its organisational learning capacity. The analytical lens – learning loops – illustrates how and why organisational learning processes are disrupted. Findings show that first-loop or superficial learning dominates reform processes within the Iranian police, weakening understandings of societal context and compromising organisational effectiveness. The study contributes an empirical case of an understudied country context to literature about policy learning, institutional legitimacy, and public organisational reform.

How do policy belief and policy learning influence perceived policy effectiveness in a local CPM policy network?

This article studies policy implementation of arts-led community development catalyzed by the Creative Placemaking (CPM) policy as a complex policy network. This article uses the social influence model and qualitative interview analysis to investigate the relationship between policy beliefs and policy learning of structurally interdependent policy actors and their perceived effectiveness of the CPM policy. Hypotheses are tested using survey and interview data collected from policy actors involved in the East Franklinton Creative Revitalization project in Columbus, OH. The findings suggest that, with the network attributes of individual policy actors being controlled, broad policy core beliefs have a more significant impact on shaping perceptions of policy effectiveness than context-specific secondary beliefs. Strategic alignment in core policy beliefs and strong confidence in the local task environment both contribute to perceived policy effectiveness in economic growth. The study underscores the importance of social learning over technical learning in enhancing perceived policy effectiveness.

The Influence of The Independent Learning System on Student Achievement in The Medical Education Program

The Indonesian government launched the Independent Campus Learning (MBKM) policy in 2020. This policy aims to provide greater autonomy to universities and students in designing learning that is innovative and in line with the needs of the times. The aim of this research is to determine the influence of the Merdeka Belajar system on student achievement in the medical education study program. This research uses quantitative research methods. Data was collected through surveys using questionnaires and literature studies. The data that has been collected is then analyzed using a regression test using the SPSS program. The research results show that there is a significant influence between the MBKM system on the achievement of students in the Medical Education study program. The group of students who took part in the MBKM program showed higher learning achievement compared to the group of students who did not take part in the MBKM program. This program provides students with the opportunity to learn and develop more optimally, thereby producing graduates who are more qualified and ready to face the challenges of an increasingly complex world of work.

Interlocal learning mechanisms and policy diffusion: The case of new energy vehicles finance in Chinese cities

AbstractPolicy diffusion based on learning mechanisms has fascinated political science and public administration scholars for a long time. A robust and growing body of studies have identified the existence and importance of learning mechanisms in policy diffusion. However, there are still some gaps that need to be further improved. First, scholars identify the learning mechanism mainly based on indirect evidence, such as geographical proximity and successful innovation policies adopted by other jurisdictions, which lacking direct and systematic evidence. Second, little is known about how the hierarchical power structure affects the leap from learning behavior to policy adoption. This study provides direct evidence for the promoting effect of intergovernmental learning on policy diffusion by analyzing case of Chinese local financial subsidy policies for new energy vehicles. The empirical results reveal that policy learning in the form of site visits among local governments significantly promotes the policy diffusion, but superior government policy strategy attenuates the influence of interlocal learning on policy diffusion. Also, the initiators and themes of policy learning affect the learning–diffusion linkage, portraying the conditional effects and nuanced dynamics of interlocal policy learning in eliciting policy diffusion.

Urban Traffic Simulation with Shared Mobility Services: An Approach Using Spatiotemporal Network Kernel Density Estimation and MATSim

The popularity of smartphones and 4G/5G network have enabled various novel transportation modes based on shared mobility, such as app-based ride-hailing services (e.g., Uber and Lyft) and shared micromobility services (e.g., Veo and Gotcha). However, little is known about to what degree their operations impact urban traffic, which is important for transportation planning and policy making. These companies seldom share their ride data due to business and user privacy reasons, and they are young and still exploring and growing their market to new locations. Recently, transportation engineering researchers began to collect data in large cities trying to understand the transportation impacts of shared mobility services, but (1) there does not exist a general analytics framework applicable to any city, and (2) the studies were based on historical data and cannot project the future easily to catch up with the rapid development of shared mobility services. In this paper, we introduce a general framework for multi-modal urban traffic analytics. The goal is to build a digital twin of the transportation in a city, i.e., an accurate agent-based transportation simulation model, based on a medium-sized dataset of the interested transportation modes collected by the research group combined with other open data sources such as US Census Bureau. With this digital twin, transportation engineering researchers can flexibly analyze the impact of shared mobility services under different scenarios, such as “if the number of Uber drivers doubles” or “if the number of deployed e-scooters doubles”. The digital twin may also enable new opportunities, such as being an environment for learning policies with reinforcement learning. Our framework consists of three steps: (1) fitting the spatiotemporal distribution of the shared mobility rides on the underlying road network, (2) generating the travel day-plans of the entire urban population based on the learned spatiotemporal ride distribution and user-specified parameters, and (3) configuring an agent-based simulation software such as MATSim to execute the generated realistic day-plans, which provides detailed transportation data for analytics. At the core of our approach is a new spatiotemporal network kernel density estimation (KDE) method that fixes the flaw of prior methods where the contributions of different data samples are not equal. We also propose a crowdsourcing method to collect app-based ride data that is easy to carry out in any city. Using Birmingham, AL as an example, we demonstrate how our framework can be applied to help transportation engineering researchers analyze the impacts of Uber ride-hailing and Veo e-scooter services.

Deep Reinforcement Learning for the Management of the Wall Regeneration Cycle in Wall-Bounded Turbulent Flows

AbstractThe wall cycle in wall-bounded turbulent flows is a complex turbulence regeneration mechanism that remains not fully understood. This study explores the potential of deep reinforcement learning (DRL) for managing the wall regeneration cycle to achieve desired flow dynamics. To create a robust framework for DRL-based flow control, we have integrated the StableBaselines3 DRL libraries with the open-source direct numerical simulation (DNS) solver CaNS. The DRL agent interacts with the DNS environment, learning policies that modify wall boundary conditions to optimise objectives such as the reduction of the skin-friction coefficient or the enhancement of certain coherent structures’ features. The implementation makes use of the message-passing-interface (MPI) wrappers for efficient communication between the Python-based DRL agent and the DNS solver, ensuring scalability on high-performance computing architectures. Initial experiments demonstrate the capability of DRL to achieve drag reduction rates comparable with those achieved via traditional methods, although limited to short time intervals. We also propose a strategy to enhance the coherence of velocity streaks, assuming that maintaining straight streaks can inhibit instability and further reduce skin-friction. Our results highlight the promise of DRL in flow-control applications and underscore the need for more advanced control laws and objective functions. Future work will focus on optimising actuation intervals and exploring new computational architectures to extend the applicability and the efficiency of DRL in turbulent flow management.