Adversarial Policy Research Articles

Why hospitals hire tobacco lobbyists: conflicts of interest among lobbyists’ clients

Abstract Lobbyists sometimes represent clients with seemingly adversarial policy interests. We seek to explain the occurrence of such ostensible conflicts of interest. In hiring lobbyists, interests encounter a tradeoff between access and agency. Although some lobbyists promise access to lawmakers, they may not lobby as contracted. Interests hire seemingly conflicted lobbyists more often when access is costlier and reputational risks are smaller. We examine the hiring of tobacco lobbyists by health interests, given the possibilities for shirking and reputational damage. We find that institutions such as hospitals hire tobacco lobbyists regularly and more often than membership-based health groups. Intergroup competition for access and lobby laws, especially anti-conflict laws, affect the use of tobacco lobbyists independent of rates of multi-client lobbying. Conflicts are more common today than ever but interests can protect themselves somewhat from opportunistic agents. Our findings also suggest that reforms can improve the representation of client interests.

Know Your Enemy: Identifying Adversarial Behaviours in Deep Reinforcement Learning Agents (Student Abstract)

It has been shown that an agent can be trained with an adversarial policy which achieves high degrees of success against a state-of-the-art DRL victim despite taking unintuitive actions. This prompts the question: is this adversarial behaviour detectable through the observations of the victim alone? We find that widely used classification methods such as random forests are only able to achieve a maximum of ≈71% test set accuracy when classifying an agent for a single timestep. However, when the classifier inputs are treated as time-series data, test set classification accuracy is increased significantly to ≈98%. This is true for both classification of episodes as a whole, and for “live” classification at each timestep in an episode. These classifications can then be used to “react” to incoming attacks and increase the overall win rate against Adversarial opponents by approximately 17%. Classification of the victim’s own internal activations in response to the adversary is shown to achieve similarly impressive accuracy while also offering advantages like increased transferability to other domains.

Improve generalization of driving policy at signalized intersections with adversarial learning

Intersections are quite challenging among various driving scenes wherein the interaction of signal lights and distinct traffic actors poses great difficulty to learn a wise and robust driving policy. Current research rarely considers the diversity of intersections and stochastic behaviors of traffic participants. For practical applications, the randomness usually leads to some devastating events, which should be the focus of autonomous driving. This paper introduces an adversarial learning paradigm to boost the intelligence and robustness of driving policy for signalized intersections with dense traffic flow. Firstly, we design a static path planner which is capable of generating trackable candidate paths for multiple intersections with diversified topology. Next, a constrained optimal control problem (COCP) is built based on these candidate paths wherein the bounded uncertainty of dynamic models is considered to capture the randomness of driving environment. We propose adversarial policy gradient (APG) to solve the COCP wherein the adversarial policy is introduced to provide disturbances by seeking the most severe uncertainty while the driving policy learns to handle this situation by competition. Finally, a comprehensive system is established to conduct training and testing wherein the perception module is introduced and the human experience is incorporated to solve the yellow light dilemma. Simulation results indicate that the trained policy can handle the signal lights flexibly meanwhile realizing smooth and efficient passing with a humanoid paradigm. Besides, APG enables a large-margin improvement of the resistance to the abnormal behaviors and thus ensures a high safety level for the autonomous vehicle.

Attacking Deep Reinforcement Learning With Decoupled Adversarial Policy

While Deep Reinforcement Learning (DRL) has achieved outstanding performance in extensive applications, exploiting its vulnerability with adversarial attacks is essential towards building robust DRL systems. In this work, we aim to propose a novel Decoupled Adversarial Policy (DAP) for attacking the DRL mechanism, whereas the adversarial agent can decompose the adversarial policy into two separate sub-policies: 1) the switch policy which determines if an attacker should launch the attack, and 2) the lure policy which determines the action an attacker induces the victim to take. If the adversarial agent samples an injection action from the switch policy, the attacker can query the pre-constructed database for universal perturbation in the real-time manner, misleading the victim to take the induced action sampled from the lure policy. To train the adversarial agent to learn DAP, we utilize those samples wherein both of the sub-actions from DAP are not restricted by each other or by the external constraint, but can actually affect the attacker’s behaviors. Therefore, we propose trajectory clipping and padding in data pruning, and Decoupled Proximal Policy Optimization (DPPO) in optimizing. Extensive experiments on different Atari games demonstrate the effectiveness of our proposed method. In addition, it can simultaneously implement the real-time and few-steps attack, which outperforms the existing counterparts.

Identity-based subgroups and information exchange in adversarial policy networks

AbstractWhile information exchange is essential in the policy process, little is known about how divergent subgroups filter actors’ technical and political information exchange, blocking learning processes. Guided by social identity, group entitativity, and self-categorisation theories, we introduce the concept and measurement of identity-based subgroups referring to informal clusters shaped by the self-referent perception of similarities among actors. The identity-based subgroup is recognised as a precursor for coalition building in a policy subsystem but received inadequate attention in the research on Advocacy Coalition Framework. We examine how divergent identity-based subgroups moderate the links between relational embeddedness and technical/political information exchanges in an adversarial fracking policy network in New York. Our quadratic assignment procedure multiple regression indicated that, despite trust, policy actors from different identity-based subgroups are less likely to share technical and political information in the network. When two actors’ identity-based subgroups are different, competition is more likely associated with lower technical information exchange in the network. These findings extend research on information exchange in adversarial policy subsystems.

Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarios

Autonomous vehicles must be comprehensively evaluated before deployed in cities and highways. However, most existing evaluation approaches for autonomous vehicles are static and lack adaptability, so they are usually inefficient in generating challenging scenarios for tested vehicles. In this paper, we propose an adaptive evaluation framework to efficiently evaluate autonomous vehicles in adversarial environments generated by deep reinforcement learning. Considering the multimodal nature of dangerous scenarios, we use ensemble models to represent different local optimums for diversity. We then utilize a nonparametric Bayesian method to cluster the adversarial policies. The proposed method is validated in a typical lane-change scenario that involves frequent interactions between the ego vehicle and the surrounding vehicles. Results show that the adversarial scenarios generated by our method significantly degrade the performance of the tested vehicles. We also illustrate different patterns of generated adversarial environments, which can be used to infer the weaknesses of the tested vehicles.

Compatibility of the Incompatible: How Does Asymmetric Power Lead to Coexistence of Trust and Distrust in Adversarial Policy Networks?

ABSTRACT Asymmetric power has been found to hinder the development of trust that is essential for structuring policy subsystems or networks. Using a multinomial logistic analysis of a local hydraulic fracturing policy network in New York, our findings indicate that (1) power imbalance affects the coexistence of trust and distrust and (2) power imbalance interacting with certain types of social relationships (policy communication, regulation, and knowledge exchange) leads to the coexistence of trust and distrust. We suggest three implications for the governance of policy networks and policy process literature: increasing levels of trust do not guarantee decreasing levels of distrust, asymmetric power has an impact on structuring “ambivalent” relations, and adversarial policy subsystems can take four structural forms depending on the levels of trust and distrust.

A universal adversarial policy for text classifiers

Discovering the existence of universal adversarial perturbations had large theoretical and practical impacts on the field of adversarial learning. In the text domain, most universal studies focused on adversarial prefixes which are added to all texts. However, unlike the vision domain, adding the same perturbation to different inputs results in noticeably unnatural inputs. Therefore, we introduce a new universal adversarial setup – a universal adversarial policy, which has many advantages of other universal attacks but also results in valid texts – thus making it relevant in practice. We achieve this by learning a single search policy over a predefined set of semantics preserving text alterations, on many texts. This formulation is universal in that the policy is successful in finding adversarial examples on new texts efficiently. Our approach uses text perturbations which were extensively shown to produce natural attacks in the non-universal setup (specific synonym replacements). We suggest a strong baseline approach for this formulation which uses reinforcement learning. Its ability to generalise (from as few as 500 training texts) shows that universal adversarial patterns exist in the text domain as well.

Political and Technical Information Exchange in Adversarial Policy Networks (WITHDRAWN)

Information exchange is one of the essential elements of learning processes in policy subsystems or networks. The role of scientific and technical information in structuring policy subsystems or networks is a foundational tenet of the Advocacy Coalition Framework. Previous research focuses on trust and political influence as drivers of information exchange between actors in networks or subsystems. Few studies examine how low levels of trust and political power disparity, common in adversarial policy networks, are associated with information exchange. Using the quadratic assignment procedure multiple regression, we analyze political and technical information flows in an adversarial local hydraulic fracturing policy network in New York. We found that low levels of trust between policy actors increase political information exchange, and a larger power disparity constraints political information exchange, despite existing informal collaborative relationships. Also, low levels of trust strengthen technical information exchange between collaborative actors in the network. These findings extend research on policy networks and ACF scholarship by pinpointing novel contingencies of political and technical information exchange in an adversarial network.

Adversarial Policy Switching with Application to RTS Games

Complex games such as RTS games are naturally formalized as Markov games. Given a Markov game, it is often possible to hand-code or learn a set of policies that capture the diversity of possible strategies. It is also often possible to hand-code or learn an abstract simulator of the game that can estimate the outcome of playing two strategies against one another from any state. We consider how to use such policy sets and simulators to make decisions in large Markov games. Prior work has considered the problem using an approach we call minimax policy switching. At each decision epoch, all policy pairs are simulated against each other from the current state, and the minimax policy is chosen and used to select actions until the next decision epoch. While intuitively appealing, we show that this switching policy can have arbitrarily poor worst case performance. In response, we describe a modified algorithm, monotone policy switching, whose worst case performance, under certain conditions, is provably no worse than the minimax fixed policy in the set. We evaluate these switching policies in both a simulated RTS game and the real game Wargus. The results show the effectiveness of policy switching when the simulator is accurate, and also highlight challenges in the face of inaccurate simulations.

Explaining the legitimizing function of expert knowledge: how France got away with pain in farm animals

Scholars have long investigated connections between types of knowledge use and types of policy subsystem. Yet, most of them focus on the learning function of expert information. The legitimizing function of knowledge—when expertise serves as a substitute for decision (Boswell in J Eur Public Policy 15(4):471–488, 2008)—has attracted less attention. An empirically validated explanation of this function is still missing. This article tests existing hypotheses regarding which features of the subsystem are conducive to the legitimizing function. The demonstration rests upon a case study: France’s Ministry of Agriculture’s commissioning of INRA to carry out a systematic literature review on pain in farm animals. Two types of factors are involved in the legitimizing function of knowledge: environmental mechanisms (an adversarial policy subsystem, concentration of policy authority) and relational mechanisms (coalitions displaying epistemic uncertainty and exerting pressures on the source of policy authority, a policy broker mitigating the conflict between the two coalitions).

Optimal Adversarial Policies in the Multiplicative Learning System With a Malicious Expert

We consider a learning system based on the conventional multiplicative weight (MW) rule that combines experts' advice to predict a sequence of true outcomes. It is assumed that one of the experts is malicious and aims to impose the maximum loss on the system. The system's loss is naturally defined to be the aggregate absolute difference between the sequence of predicted outcomes and the true outcomes. We consider this problem under both offline and online settings. In the offline setting where the malicious expert must choose its entire sequence of decisions a priori, we show somewhat surprisingly that a simple greedy policy of always reporting false prediction is asymptotically optimal with an approximation ratio of 1+O√(ln N)/N, where N is the total number of prediction stages. In particular, we describe a policy that closely resembles the structure of the optimal offline policy. For the online setting where the malicious expert can adaptively make its decisions, we show that the optimal online policy can be efficiently computed by solving a dynamic program in O(N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ). We also discuss a generalization of our model to multi-expert settings. Our results provide a new direction for vulnerability assessment of commonly-used learning algorithms to internal adversarial attacks.

The Impact of Complex and Informed Adversarial Behavior in Graphical Coordination Games

How does system-level information impact the ability of an adversary to degrade performance in a networked control system? How does the complexity of an adversary's strategy affect its ability to degrade performance? This article focuses on these questions in the context of graphical coordination games where an adversary can influence a given fraction of the agents in the system, and the agents follow log-linear learning, a well-known distributed learning algorithm. Focusing on a class of homogeneous ring graphs of various connectivity, we begin by demonstrating that minimally connected ring graphs are the most susceptible to adversarial influence. We then proceed to characterize how both the sophistication of the attack strategies (static versus dynamic) and the informational awareness about the network structure can be leveraged by an adversary to degrade system performance. Focusing on the set of adversarial policies that induce stochastically stable states, our findings demonstrate that the relative importance between sophistication and information changes with the influencing power of the adversary. In particular, sophistication far outweighs informational awareness with regards to degrading system-level damage when the adversary's influence power is relatively weak. However, the opposite is true when an adversary's influence power is more substantial.

Increasing Trust in Development Processes Using Robust, Data-Driven Markov Games: An Application to PRESTIGE

In this article, we consider how to increase trust in development processes in which there is a risk for adversarial manipulation, and the adversary's objectives and resources are either ill-specified, imprecisely specified, or unknown. In such problems, we must hedge against the risk of misapprehension of attacker objectives and resources, which is further complicated in the absence of adversarial training data. We show how to model dynamic agent interaction, on the basis of partially observed or noise-corrupted data, using a partially observable Markov game (POMG) framework. We then propose a threefold heuristic solution procedure that: 1) uses the POMG to generate potential adversarial policies; 2) explicitly incorporates these adversarial policies in the construction of a robust defender policy by solving a robust dynamic program (DP); and 3) employs a probability matching heuristic in partially observable environments.

Learning adversarial policy in multiple scenes environment via multi-agent reinforcement learning

ABSTRACT Learning adversarial policy aims to learn behavioural strategies for agents with different goals, is one of the most significant tasks in multi-agent systems. Multi-agent reinforcement learning (MARL), as a state-of-the-art learning-based model, employs centralised or decentralised control methods to learn behavioural strategies by interacting with environments. It suffers from instability and slowness in the training process. Considering that parallel simulation or computation is an effective way to improve training performance, we propose a novel MARL method called Multiple scenes multi-agent proximal Policy Optimisation (MPO) in this paper. In MPO, we first simulate multiple parallel scenes in the training environment. Multiple policies control different agents in the same scene, and each policy also controls several identical agents from multiple scenes. Then, we expand proximal policy optimisation (PPO) with an improved actor-critic network, ensuring the stability of training in multi-agent tasks. The actor network only uses local information for decision making, and the critic network uses global information for training. Finally, effective training trajectories are computed with two criteria from multiple parallel scenes rather than single to accelerate the learning process. We evaluate our approach in two simulated 3D environments, one of which is Unity's official open-source soccer game, and the other is unmanned surface vehicles (USVs) built by Unity. Experiments demonstrate that MPO converges more stable and faster than benchmark methods in model training, and demonstrates excellent adversarial policy compared with benchmark models.

Behavioral Repertoire via Generative Adversarial Policy Networks

Learning algorithms are enabling robots to solve increasingly challenging real-world tasks. These approaches often rely on demonstrations and reproduce the behavior shown. Unexpected changes in the environment or in robot morphology may require using different behaviors to achieve the same effect, for instance, to reach and grasp an object in changing clutter. An emerging paradigm addressing this robustness issue is to learn a diverse set of successful behaviors for a given task, from which a robot can select the most suitable policy when faced with a new environment. In this article, we explore a novel realization of this vision by learning a generative model over policies. Rather than learning a single policy, or a small fixed repertoire, our generative model for policies compactly encodes an unbounded number of policies and allows novel controller variants to be sampled. Leveraging our generative policy network, a robot can sample novel behaviors until it finds one that works for a new scenario. We demonstrate this idea with an application of robust ball throwing in the presence of obstacles, as well as joint-damage-robust throwing. We show that this approach achieves a greater diversity of behaviors than an existing evolutionary approach, while maintaining good efficacy of sampled behaviors, allowing a Baxter robot to hit targets more often when ball throwing in the presence of varying obstacles or joint impediments.

Two Can Play That Game

Cyber-security is an important societal concern. Cyber-attacks have increased in numbers as well as in the extent of damage caused in every attack. Large organizations operate a Cyber Security Operation Center (CSOC), which forms the first line of cyber-defense. The inspection of cyber-alerts is a critical part of CSOC operations (defender or blue team). Recent work proposed a reinforcement learning (RL) based approach for the defender’s decision-making to prevent the cyber-alert queue length from growing large and overwhelming the defender. In this article, we perform a red team (adversarial) evaluation of this approach. With the recent attacks on learning-based decision-making systems, it is even more important to test the limits of the defender’s RL approach. Toward that end, we learn several adversarial alert generation policies and the best response against them for various defender’s inspection policy. Surprisingly, we find the defender’s policies to be quite robust to the best response of the attacker. In order to explain this observation, we extend the earlier defender’s RL model to a game model with adversarial RL, and show that there exist defender policies that can be robust against any adversarial policy. We also derive a competitive baseline from the game theory model and compare it to the defender’s RL approach. However, when we go further to exploit the assumptions made in the Markov Decision Process (MDP) in the defender’s RL model, we discover an attacker policy that overwhelms the defender. We use a double oracle like approach to retrain the defender with episodes from this discovered attacker policy. This made the defender robust to the discovered attacker policy and no further harmful attacker policies were discovered. Overall, the adversarial RL and double oracle approach in RL are general techniques that are applicable to other RL usage in adversarial environments.

Probing Narrative Cognition Theory: A Semantic Analysis of Multi-level Hydraulic Fracturing Policy Narratives in Arkansas

Contentious policy debates over the use of hydraulic fracturing technologies in the U.S. have produced inconsistent policies (John Kester and Moyer 2015; Zirogiannis et al. 2016). These debates hold serious implications across a variety of policy domains including energy, environment, economy, and national security. Competing policy narratives often oversimplify the issue by positioning hydraulic fracturing as either a threat to the environment or as an opportunity to realize an economic gain, leaving the issue vulnerable to disinformation campaigns (Old Tactics, New Tools: A Review of Russia’s Soft Cyber Influence Operations. 2017; U.S. House of Representatives 2018) which continue to be employed by outside interests in efforts to destabilize the U.S. energy market (U.S. House of Representatives 2018). According to the Narrative Policy Framework (NPF), meso-level narratives are utilized by competing advocacy coalitions within an adversarial policy subsystem (McBeth et al. 2007; Shanahan, Jones, and McBeth 2011). Meso-level policy narratives are strategically employed to persuade others to adopt one policy perspective over another. In Colorado, for instance, narratives on hydraulic fracturing emphasized either the associated economic benefits or the potential risks posed to human health and the environment (Heikkila et al. 2014). This divergence of narrative surely fails to support compromise and holds the potential to sustain related policy conflict (Weible and Heikkila 2017). The different views advanced by such narratives are likely to be evaluated, processed, and consumed by diverse individual policy actors, particularly those who are interested in the issue and willing to engage in the debate. While the NPF identifies narrative cognition as the process of extracting meaningful elements from the narrative to think about the policy issue at the micro-level, this concept has rarely been investigated in a systemic manner (but see Jones and Song 2014; Jorgensen, Song, and Jones 2018; Zanocco, Song, and Jones 2018). The controversial nature of the hydraulic fracturing policy debate provides a unique opportunity to further examine this concept of narrative cognition. As such, this chapter probes narrative cognition theory within the context of the hydraulic fracturing (HF) debate in Arkansas by comparing meso-level HF narratives to micro-level communications and examining what factors influence the process. While meso-level HF narratives are certainly accessible to both elite policy actors and the general public, this study narrows its focus to local policy elites residing in the state.

Mediation Centrality in Adversarial Policy Networks

Conflict resolution often involves mediators who understand the issues central to both sides of an argument. Mediators in complex networks represent key nodes that are connected to other key nodes in opposing subgraphs. Here we introduce a new metric, mediation centrality, for identifying good mediators in adversarial policy networks, such as the connections between individuals and their reasons for and against the support of controversial topics (e.g., state‐financed abortion). Using a process‐based account of reason mediation we construct bipartite adversarial policy networks and show how mediation defined over subgraph projections constrained to reasons representing opposing sides can be used to produce a measure of mediation centrality that is superior to centrality computed on the full network. We then empirically illustrate and test mediation centrality in a “policy fluency task,” where participants generated reasons for or against eight controversial policy issues (state‐subsidized abortion, bank bailouts, forced CO2 reduction, cannabis legalization, shortened naturalization, surrogate motherhood legalization, public smoking ban, and euthanasia legalization). We discuss how mediation centrality can be extended to adversarial policy networks with more than two positions and to other centrality measures.

A Rose by Any Other Name? A Subtle Linguistic Cue Impacts Anger and Corresponding Policy Support in Intractable Conflict

Given the central role of anger in shaping adversarial policy preferences in the context of intergroup conflict, its reduction may promote conflict resolution. In the current work, we drew on psycholinguistic research on the role of language in generating emotions to explore a novel, extremely subtle means of intervention. Specifically, we hypothesized that phrasing conflict-relevant policies in noun form (vs. verb form) would reduce anger and impact policy support correspondingly. Results across three experimental studies in the context of the Israeli-Palestinian conflict supported these expectations for both support for concessions (Studies 1–3) and retaliatory policies (Study 3), with reduction in anger mediating the salutary impact of noun form (vs. verb form) on policy support. These results expand our understanding of the influence of language on emotions and policies in the context of conflict and have applied relevance for conflict-resolution efforts.