Private Reward Research Articles

Robust Reward-Free Actor-Critic for Cooperative Multiagent Reinforcement Learning.

In this article, we consider centralized training and decentralized execution (CTDE) with diverse and private reward functions in cooperative multiagent reinforcement learning (MARL). The main challenge is that an unknown number of agents, whose identities are also unknown, can deliberately generate malicious messages and transmit them to the central controller. We term these malicious actions as Byzantine attacks. First, without Byzantine attacks, we propose a reward-free deep deterministic policy gradient (RF-DDPG) algorithm, in which gradients of agents' critics rather than rewards are sent to the central controller for preserving privacy. Second, to cope with Byzantine attacks, we develop a robust extension of RF-DDPG termed R2F-DDPG, which replaces the vulnerable average aggregation rule with robust ones. We propose a novel class of RL-specific Byzantine attacks that fail conventional robust aggregation rules, motivating the projection-boosted robust aggregation rules for R2F-DDPG. Numerical experiments show that RF-DDPG successfully trains agents to work cooperatively and that R2F-DDPG demonstrates robustness to Byzantine attacks.

A Game-Theoretic Federated Learning Framework for Data Quality Improvement

Federated learning is a promising distributed machine learning paradigm that has been playing a significant role in privacy-preserving machine learning tasks. However, alongside all its achievements, the framework has limitations. First, traditional frameworks assume that all clients want to improve model accuracy and so participation is voluntary. However, in reality, clients usually want to be appropriately compensated for the data and resources they will need to commit to the training process before contributing. Second, today's frameworks allow clients to perturb their parameter updates locally, which introduces a great deal of noise to the trained model and can seriously impact model accuracy. To address these concerns, we have developed a private reward game that incentivizes clients to contribute high-quality data to the training process. The game converges to a Nash equilibrium under the guarantee of joint differential privacy, and each client maximizes their reward following an equilibrium strategy. The noise injected into the model is reduced by introducing a centralized differential privacy model that aggregates the parameters and compensates clients via a data trading market. Experimental simulations show the rationales behind and effectiveness of the proposed game approach. Additionally, we present comparisons between different training models to demonstrate the performance of the proposed approach in real-world scenarios.

Fiscal Pressure and Public–Private Partnership Investment: Based on Evidence from Prefecture-Level Cities in China

Public–private partnership (PPP) policy is essential to alleviating the local government debt burden and improving resource allocation efficiency. This paper empirically examines the impact of fiscal pressure on PPP investment in Chinese prefecture-level cities from 2014 to 2019 using the ordinary least-squares (OLS) module. Moreover, we also investigate how fiscal pressure influences PPP investment and test the influenced mechanism from other perspectives. The results show the following. (1) Fiscal pressure on the government has a significant positive effect on PPP investment at the prefectural level. (2) The marketization process is the mediated effect of the relationship between fiscal pressure and PPP investment. Fiscal pressure will stimulate the regional marketization process, thus promoting PPP investment. (3) Fiscal pressure has a significant positive effect on PPP investment in the middle region, while the effect is not significant in the eastern and western regions. Meanwhile, the effect is not significant in central cities, but there is a significant positive effect in ordinary cities. (4) The effect of fiscal pressure on PPP investment is not significant in the private reward mode of government payment, but there is a significant positive effect in the mode of user payment and feasibly insufficient subsidies. Our studies could also provide practical suggestions for sustainable development of PPP policy and solving the fiscal pressure of the current economic recession under the COVID-19 pandemic.

How Private Is Your RL Policy? An Inverse RL Based Analysis Framework

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep-Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL analysis framework (PRIL) that involves performing reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using the Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.

Contracting with private rewards

AbstractThe canonical moral hazard model is extended to allow the agent to face endogenous and noncontractible uncertainty. The agent works for the principal and simultaneously pursues outside rewards. The contract offered by the principal thus manipulates the agent's work–life balance. The participation constraint is slack whenever it is optimal to distort the agent's work–life balance away from life compared to a symmetric‐information benchmark. Then, the agent's expected utility is high and he faces flatter incentives. Such contracts may be optimal when the two activities are strong substitutes in the agent's cost function or when reservation utility is low.

Collaboratively Learning the Best Option on Graphs, Using Bounded Local Memory

We consider multi-armed bandit problems in social groups wherein each individual has bounded memory and shares the common goal of learning the best arm/option. We say an individual learns the best option if eventually (as t - ∞) it pulls only the arm with the highest expected reward. While this goal is provably impossible for an isolated individual due to bounded memory, we show that, in social groups, this goal can be achieved easily with the aid of social persuasion (i.e., communication) as long as the communication networks/graphs satisfy some mild conditions. In this work, we model and analyze a type of learning dynamics which are wellobserved in social groups. Specifically, under the learning dynamics of interest, an individual sequentially decides on which arm to pull next based on not only its private reward feedback but also the suggestion provided by a randomly chosen neighbor. To deal with the interplay between the randomness in the rewards and in the social interaction, we employ the mean-field approximation method. Considering the possibility that the individuals in the networks may not be exchangeable when the communication networks are not cliques, we go beyond the classic mean-field techniques and apply a refined version of mean-field approximation. Notably, our results hold even if the communication graphs are highly sparse.

Collaboratively Learning the Best Option on Graphs, Using Bounded Local Memory

We consider multi-armed bandit problems in social groups wherein each individual has bounded memory and shares the common goal of learning the best arm/option. We say an individual learns the best option if eventually (as $t\diverge$) it pulls only the arm with the highest expected reward. While this goal is provably impossible for an isolated individual due to bounded memory, we show that, in social groups, this goal can be achieved easily with the aid of social persuasion (i.e., communication) as long as the communication networks/graphs satisfy some mild conditions. In this work, we model and analyze a type of learning dynamics which are well-observed in social groups. Specifically, under the learning dynamics of interest, an individual sequentially decides on which arm to pull next based on not only its private reward feedback but also the suggestion provided by a randomly chosen neighbor. To deal with the interplay between the randomness in the rewards and in the social interaction, we employ the \em mean-field approximation method. Considering the possibility that the individuals in the networks may not be exchangeable when the communication networks are not cliques, we go beyond the classic mean-field techniques and apply a refined version of mean-field approximation: Using coupling we show that, if the communication graph is connected and is either regular or has doubly-stochastic degree-weighted adjacency matrix, with probability → 1 as the social group size N → ∞, every individual in the social group learns the best option. If the minimum degree of the graph diverges as N → ∞, over an arbitrary but given finite time horizon, the sample paths describing the opinion evolutions of the individuals are asymptotically independent. In addition, the proportions of the population with different opinions converge to the unique solution of a system of ODEs. Interestingly, the obtained system of ODEs are invariant to the structures of the communication graphs. In the solution of the obtained ODEs, the proportion of the population holding the correct opinion converges to 1 exponentially fast in time. Notably, our results hold even if the communication graphs are highly sparse.

Pollination of Blumenbachia amana (Loasaceae): flower morphology and partitioned pollen presentation guarantee a private reward to a specialist pollinator

Pollination of Blumenbachia amana (Loasaceae): flower morphology and partitioned pollen presentation guarantee a private reward to a specialist pollinator

Liberating soil data for profitable agriculture and catchment health in the Corangamite region, Australia

ABSTRACTDetailed soil data has been collected in the Corangamite region of south-east Australia for over 80 years, as a testament to the productive value of the region’s agricultural soils. Soil science over that period has resulted in soil maps, soil investigation sites and archival materials that provide valuable baseline data for the analysis of trends over time. This legacy data has been brought together with contemporary data in the award-winning Soil Health Knowledge Base, an Internet portal based on spatial data infrastructure that interoperably federates data (open data, research data, industry data, sensor data, legacy data, crowdsourced data … any available data). The portal provides the best available data sources for research and consulting, as well as functions for both the private reward and the public good. The ultimate intent is to provide timely decision support for agricultural enterprises and catchment managers to protect, enhance and restore soil health.

Playing with Cards

A common response to claims of bias, harassment, or discrimination is to say that these claims are made in bad faith. Claimants are supposedly not motivated by a credible or even sincere belief that unfair or unequal treatment has occurred, but simply seek to illicitly gain public sympathy or private reward. Characterizing discrimination claims as systemically made in bad faith enables them to be screened and dismissed prior to engaging with them on their merits. This retort preserves the dominant groups' self-image as unprejudiced and innocent without having to risk critical analysis of the claim's substance.

Better Financial Innovation via Innovative Finance of Supervisors

Financial innovation may cause negative externalities and thus requires government interventions to correct incentives. However, with imperfect information, a simple Pigouvian tax cannot be implemented. We suggest to link the compensation of supervisors to the compensation of decision makers via a tax on the latter. Apart from inducing lower compensation in the financial sector, this mechanism provides an incentive for financial institutions to design contracts discouraging rent-seeking. As high compensation for rent-seeking would indirectly increase wages and thus talent of supervisors, it would also increase the expected punishment. Therefore, private rewards for rent-seeking will be reduced which improves economic efficiency.

Knights and Knaves in the English Medical Profession: the Case of Clinical Excellence Awards

AbstractWe elaborate Le Grand's thesis of ‘knights and knaves’ in terms of clinical excellence awards (CEAs), the ‘financial bonuses’ which are paid to over half of all English hospital specialists and which can be as much as £75,000 (€92,000) per year in addition to an NHS (National Health Service) salary. Knights are ‘individuals who are motivated to help others for no private reward’ while knaves are ‘self-interested individuals who are motivated to help others only if by doing so they will serve their private interests.’ Doctors (individually and collectively) exhibit both traits but the work of explanation of the inter-relationship between them has remained neglected. Through a textual analysis of written responses to a recent review of CEAs, we examine the ‘knightly’ and ‘knavish’ arguments used by medical professional stakeholders in defending these CEAs. While doctors promote their knightly claims, they are also knavish in shaping the preferences of, and options for, policy-makers. Policy-makers continue to support CEAs but have introduced revised criteria for CEAs, putting pressure on the medical profession to accept reforms. CEAs illustrate the enduring and flexible power of the medical profession in the UK in colonising reforms to their pay, and also the subtle inter-relationship between knights and knaves in health policy.

Team Contests with Multiple Pairwise Battles

We consider a multi-battle team contest in which players from two rival teams form pairwise matches to fight in distinct component battles, which are carried out sequentially or (partially) simultaneously. A team wins if and only if its players win a majority of battles. Each player benefits from his team's win, while he can also receive a private reward for winning his own battle. We find that the outcomes of past battles do not distort the outcomes of future battles. Neither the total expected effort nor the overall outcome of the contest depends on the contest's temporal structure or its feedback policy. (JEL C72, D72, D74, D82)

The impact of goals and public rewards on a prosocial behaviour: an exploratory economic experiment

Given that private rewards have mixed effectiveness in the realm of prosocial behaviours (Bénabou and Tirole, 2006), we explore whether a ‘public’ reward (denominated in additional units of public good provision) successfully encourages prosocial behaviour in the form of increased donations to a project aimed at improving atmospheric air quality. Our experiment consists of a dictator game designed to address (1) if the presence of a goal increases voluntary contributions to improving environmental quality and (2) if an additional public reward further increases voluntary contributions. We find a significant goal effect overall, yet no added public reward effect.

Governance, Risk Management, and Risk-Taking in Banks

This paper examines how governance and risk management affect risk-taking in banks. It distinguishes between good risks, which are risks that have an ex ante private reward for the bank on a stand-alone basis, and bad risks, which do not have such a reward. A well-governed bank takes the amount of risk that maximizes shareholder wealth subject to constraints imposed by laws and regulators. In general, this involves eliminating or mitigating all bad risks to the extent that it is cost effective to do so. The role of risk management in such a bank is not to reduce the bank's total risk per se. It is to identify and measure the risks the bank is taking, aggregate these risks in a measure of the bank's total risk, enable the bank to eliminate, mitigate and avoid bad risks, and ensure that its risk level is consistent with its risk appetite. Organizing the risk management function so that it plays that role is challenging because there are limitations in measuring risk and because, while more detailed rules can prevent destructive risk-taking, they also limit the flexibility of an institution in taking advantage of opportunities that increase firm value. Limitations of risk measurement and the decentralized nature of risk-taking imply that setting appropriate incentives for risk-takers and promoting an appropriate risk culture are essential to the success of risk management in performing its function.

Public praise vs. private pay: Effects of rewards on energy conservation in the workplace

Any solution to rising levels of CO2 depends on human behavior. One common approach to changing human behavior is rewarding desired behavior. Because financial incentives often have side effects that diminish efficacy, we predict that social rewards are more effective, because they invoke adherence to descriptive and injunctive social norms. We investigated this by measuring electricity use for 13weeks at a Dutch firm. Each week, employees were rewarded for conserving energy. They either received monetary rewards (€0–€5) or social rewards (grade points with a descriptive comment). Rewards were either private or public. In both the short and long term, public rewards outperformed private rewards, and social rewards outperformed monetary rewards. This suggests that private monetary rewards, although popular, may be ineffective. Instead, public social rewards may be a more promising approach to stimulating energy conservation. We argue that this approach should be considered more frequently by policy-makers.

Review: The Dictator's Handbook

THE DICTATOR'S HANDBOOK Why Bad Behavior is Almost Always Good Politics Bruce Bueno de Mesquita and Alistair Smith New York: PublicAffairs, 2012. 352pp $21.00 (paper) ISBN 978-1-6103-9184-9Is there a of in politics? That is, is there something that explains our major concerns - patterns of war and peace, political order and instability, the emergence and survival of democracy and dictatorship, and the politics of prosperity and poverty? Bruce Bueno de Mesquita and Alistair Smith claim to present a explaining many of these phenomena according to few core variables. This book provides stimulating answers to pressing political problems, drawn from a relatively simple and powerful model. At its best, the book can boast of providing analysis that appears counterintuitive at first but seems obvious once it is laid out. There are serious analytical problems at the core of the model, but the book is insightful nonetheless.This book is a codification for a popular audience of the theory of politics, elaborated by the authors, along with James D. Morrow and Randolph M. Siverson in numerous articles and, most comprehensively, in the dense, mathematical The Logic of Political Survival (MIT Press, 2003). The theory's basic assumption is that leaders' interest in political survival drives everything they do: war, repression, corruption, political reform, tax policy. What leaders have to do to survive depends, in turn, on how many people form a winning coalition - that is, how many they have to pay off to stay in power. Leaders must maintain a winning coalition drawn from the - those individuals lucky enough to play a role in deciding who gets to be in office. The three core variables in politics, for the authors, are the size of the winning coalition needed, the size of the selectorate that the winning coalition can be drawn from, and the amount of cash available to spread around. In small-coalition polities like dictatorships, leaders can stay in power through the judicious use of private rewards. Not every dictatorship is the same; in some dictatorships, there are plenty of selectorate members itching for the chance to get in on the action, bidding the price of loyalty down. In others, there are very few selectorate members, and the dictator is more vulnerable to them.In contrast, in a democracy, the selectorate is large, but so is the coalition needed to win. Voters need payoffs, just like strongmen in a dictatorship - but the difference in size is crucial. In a democracy, it would be far too costly to give enough voters a direct bribe. Leaders in large-coalition polities therefore distribute more public goods - that is, they must actually govern in the public interest, at least to a greater degree, to stay in power. Dictators need only keep the right people paid off; this is why democracies prosper faster than dictatorships. However, revolutions occur when there is not enough money to go around to keep people off the street or to pay the army. Since dictatorships are bad for prosperity, there is some hope - provided that there is not enough oil or diamonds to make the public interest irrelevant to the dictator's flow of money.That is the core of the book, developed over a few chapters that capture the skulduggery of coming to power and keeping it through bribes and payoffs. Much of the argument is older than the authors let on. It is hardly novel to say that democracy offers better public policy than dictatorship because leaders in the former are more required to act in the public interest. Indeed, it was almost twenty years ago that the late Mancur Olson put this argument in the contemporary political-economy terms of public goods provision. Second, financial crisis as a source of cracks in regimes and hence revolution is a staple of the democratization literature. Finally, the resource curse is by now as well-known an idea as there is in comparative politics. …

Public Praise vs. Private Pay: Effects of Rewards on Energy Conservation in the Workplace

Any solution to rising levels of CO2 depends on human behavior. One common approach to changing human behavior is rewarding desired behavior. Because financial incentives often have side effects that diminish efficacy, we predict that more psychologically oriented social rewards are more effective, because they invoke adherence to descriptive and injunctive social norms. We investigated this by measuring electricity use for 13 weeks at a Dutch firm. Each week, employees were rewarded for conserving energy. They either received monetary rewards (€0-€5) or social rewards (grade points with a descriptive comment). Rewards were either private or public. In both the short and long term, public rewards outperformed private rewards, and social rewards outperformed monetary rewards. This suggests that private monetary rewards, although popular, may be ineffective. Instead, public social rewards may be a more promising approach to stimulating energy conservation. Such social rewards do not crowd out intrinsic motivation, have less need for large-scale institutions or exogenous funding, and work regardless of who is paying the energy bill. Thus, we argue that the social norms approach should be considered more frequently as a valuable tool in the intervention tool-kit, especially when focusing on low-cost environmental behavior.

The Dark at the Top of the Stairs: Four Destructive Influences of Capital Punishment on American Criminal Justice

State execution is not a major part of America's system of criminal punishment, but it casts a long shadow over the principles and practices of the rest of criminal justice. This essay discusses four negative impacts of the death penalty on the larger system: (1) The hyper-extension of the substantive criminal law; (2) The symbolic transformation of harsh punishment to a private reward; (3) The creation and overuse of Life Without Possibility of Parole (LWOP) sentences; and (4) The diversion of legal and judicial resources from other pressing problems in the criminal system.

The NIH translational research center might trade public risk for private reward

The new National Center for Advancing Translational Sciences planned for the US National Institutes of Health intends to help transform biological findings into new therapeutic products. But if taxpayer funding of risky biomedical research translates into lucrative new medicines, the public should share in the economic benefits as well.