Risk-averse Policy Research Articles

Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning.

A key challenge in offline reinforcement learning (RL) is how to ensure the learned offline policy is safe, especially in safety-critical domains. In this article, we focus on learning a distributional value function in offline RL and optimizing a worst-case criterion of returns. However, optimizing a distributional value function in offline RL can be hard, since the crossing quantile issue is serious, and the distribution shift problem needs to be addressed. To this end, we propose monotonic quantile network (MQN) with conservative quantile regression (CQR) for risk-averse policy learning. First, we propose an MQN to learn the distribution over returns with non-crossing guarantees of the quantiles. Then, we perform CQR by penalizing the quantile estimation for out-of-distribution (OOD) actions to address the distribution shift in offline RL. Finally, we learn a worst-case policy by optimizing the conditional value-at-risk (CVaR) of the distributional value function. Furthermore, we provide theoretical analysis of the fixed-point convergence in our method. We conduct experiments in both risk-neutral and risk-sensitive offline settings, and the results show that our method obtains safe and conservative behaviors in robotic locomotion tasks.

Improving Risk-averse Distributional Reinforcement Learning with Adaptive Risk-seeking Exploration

In autonomous driving, reinforcement learning’s interactive learning capabilities have garnered interest. Among these, risk-averse distributional reinforcement learning has demonstrated promise in situations where safety is crucial. On the other hand, risk-averse policies degrade training performance by discouraging exploration. To provide an adaptive exploration policy, this work offers a novel framework for risk-averse distributional reinforcement learning. The proposed method is evaluated in a highly uncertain safety-critical scenario, motion planning for pedestrian avoidance in the face of uncertain interactions. The simulation results demonstrate improved safety, achieving training flexibility between exploration and exploitation.

Risk-averse stochastic dynamic power dispatch based on deep reinforcement learning with risk-oriented Graph-Gan sampling

The increasing penetration of renewable energy sources (RES) brings volatile stochasticity, which significantly challenge the optimal dispatch of power systems. This paper aims at developing a cost-effective and robust policy for stochastic dynamic optimization of power systems, which improves the economy as well as avoiding the risk of high costs in some critical scenarios with small probability. However, it is hard for existing risk-neutral methods to incorporate risk measure since most samples are normal. For this regard, a novel risk-averse policy learning approach based on deep reinforcement learning with risk-oriented sampling is proposed. Firstly, a generative adversarial network (GAN) with graph convolutional neural network (GCN) is proposed to learn from historical data and achieve risk-oriented sampling. Specifically, system state is modelled as graph data and GCN is employed to capture the underlying correlation of the uncertainty corresponding to the system topology. Risk knowledge is the embedded to encourage more critical scenarios are sampled while aligning with historical data distributions. Secondly, a modified deep reinforcement learning (DRL) with risk-measure under soft actor critic framework is proposed to learn the optimal dispatch policy from sampling data. Compared with the traditional deep reinforcement learning which is risk-neutral, the proposed method is more robust and adaptable to uncertainties. Comparative simulations verify the effectiveness of the proposed method.

A resilient inventory management of pharmaceutical supply chains under demand disruption

Inspired by the global supply chain disruptions caused by the COVID-19 pandemic, we study optimal procurement and inventory decisions for a pharmaceutical supply chain over a finite planning horizon. To model disruption, we assume that the demand for medical drugs is uncertain and shows spatiotemporal variability. To address demand uncertainty, we propose a two-stage optimization framework, where in the first stage, the total cost of pre-positioning drugs at distribution centers and its associated risk is minimized, while the second stage minimizes the cost of recourse decisions (e.g., reallocation, inventory management). To allow for different risk preferences, we propose to capture the risk of demand uncertainty through the expectation and worst-case measures, leading to two different models, namely (risk-neutral) stochastic programming and (risk-averse) robust optimization. We consider a finite number of scenarios to represent the demand uncertainty, and to solve the resulting models efficiently, we propose L-shaped decomposition-based algorithms. Through extensive numerical experiments, we illustrate the impact of various parameters, such as travel time, product’s shelf life, and waste due to transportation and storage, on the supply chain resiliency and cost, under optimal risk-neutral and risk-averse policies. These insights can assist decision makers in making informed choices.

Palliative Care, Intimacy, and Sexual Expression in the Older Adult Residential Care Context: "Living until You Don't".

Commonly, frail older adults move to residential care, a liminal space that is their home, sometimes a place of death, and a workplace. Residential facilities typically espouse person-centred values, which are variably interpreted. A critical approach to person-centred care that focuses on social citizenship begins to address issues endemic in diminishing opportunities for intimacy in the end-of-life residential context: risk-averse policies; limited education; ageism; and environments designed for staff convenience. A person-centred approach to residents’ expressions of intimacy and sexuality can be supported throughout end-of-life care. The present study utilised a constructionist methodology to investigate meanings associated with intimacy in the palliative and end-of-life care context. There were 77 participants, including residents, family members and staff, from 35 residential facilities. Analysis identified four key themes: care home ethos and intimacy; everyday touch as intimacy; ephemeral intimacy; and intimacy mediated by the built environment. Residents’ expressions of intimacy and sexuality are supported in facilities where clinical leaders provide a role-model for a commitment to social citizenship. Ageism, restrictive policies, care-rationing, functional care, and environmental hindrances contribute to limited intimacy and social death. Clinical leaders have a pivotal role in ensuring person-centred care through policies and practice that support residents’ intimate reciprocity.

Guest workers and development–security conflict: Managing labour migration at the Sino-Vietnamese border

This article investigates the increasing development–security conflict in China’s immigration management through the case of a policy trial regularizing Vietnamese labour migration in two Guangxi border cities. China’s border regions host low-income immigrant labourers from neighbouring nations. In the 2010s, China launched a series of policy initiatives to regulate temporary and irregular migrant flows. Based on fieldwork and policy research, this study analyses the development and early implementation of this trial, with a focus on state perspectives. It shows how state actors mobilize migrant temporariness and other policy tools within a negotiation process that aims to resolve tensions between developmental policy aims for transnational economic integration and a drive towards securitizing cross-border mobility. I conclude that state actors fail to reach a balance between the conflicting development and security concerns. I also argue that China’s current risk-averse policy environment makes the development–security policy conflict in its immigration management more difficult to resolve. My findings contribute to our understanding of contemporary Chinese policymaking, including immigration policymaking, as well as to the literature on the development–security nexus in temporary labour management schemes.

CEO gender and risk aversion: Further evidence using the composition of firm’s cash

Recent studies suggest that the conventional measure of cash holdings used to capture managerial risk-aversion may be inadequate. Using a new measure of physical cash, which adequately captures managerial risk-aversion, we deduce that female CEOs exhibit risk-aversion not only by maintaining higher physical cash levels but also by adopting risk-reducing policies. These risk-averse policies may be detrimental for some shareholders (despite evidence that female executives reduce agency conflicts) and they may impact the demand for female CEOs in the CEO labor markets. Our results are robust and lend support to the contention that female risk-aversion exists even beyond the “glass-ceiling”.

Risk averse policies foster bio-economic sustainability in mixed fisheries

This article examines the role of risk aversion on the sustainable management of mixed fisheries. We consider a bio-economic model of multiple species harvested by a single fleet with uncertain costs of effort. We assume that the regulatory agency aims at reaching MMEY (Multispecies Maximum Economic Yield) by maximizing the expected utility of total profits, where the utility function captures risk aversion. We show analytically that such a risk-averse MMEY mitigates the risk of biological and economic overexploitation of the different species and thus of biodiversity loss. However excessively high risk aversion also may also lessen food production at MMEY. Thus, risk aversion implies a trade-off between different bio-economic goals. These findings are illustrated with the case study of the Australian South East Fishery, where intermediate risk aversion levels allow for balanced bio-economic management objectives, therefore fostering sustainability.

Transition Based Discount Factor for Model Free Algorithms in Reinforcement Learning

Reinforcement Learning (RL) enables an agent to learn control policies for achieving its long-term goals. One key parameter of RL algorithms is a discount factor that scales down future cost in the state’s current value estimate. This study introduces and analyses a transition-based discount factor in two model-free reinforcement learning algorithms: Q-learning and SARSA, and shows their convergence using the theory of stochastic approximation for finite state and action spaces. This causes an asymmetric discounting, favouring some transitions over others, which allows (1) faster convergence than constant discount factor variant of these algorithms, which is demonstrated by experiments on the Taxi domain and MountainCar environments; (2) provides better control over the RL agents to learn risk-averse or risk-taking policy, as demonstrated in a Cliff Walking experiment.

Public support for the judicial branches under the COVID-19 pandemic

ABSTRACT In democratic countries, public support for the judiciary is crucial to prevent abuse of power by the government. While facing the coronavirus disease 2019 (COVID-19) pandemic, governments may overly restrict their citizens’ rights to prevent the spread of infectious diseases. This study examined whether voters would support a court's ruling if the court decided that the government's restriction of their rights was illegal and nullified the risk-averse policies. We suppose that voters faced with the risk of a pandemic will support a government with the expertise and resources to prevent a pandemic, and we set up the following hypotheses: If the courts nullify the measures against COVID-19, the public will not support the courts, and the government's failure to comply with the court's ruling will not diminish support for the government. We find support for these hypotheses in the contexts wherein the degree of restriction on voters’ rights is weak. Because there is a close relationship between “public support for the court” and “government abuse of power,” we need to consider which factors determine support for the court.

Robust risk-averse multi-armed bandits with application in social engagement behavior of children with autism spectrum disorder while imitating a humanoid robot

The stochastic multi-armed bandit problem is a standard model to solve the exploration–exploitation trade-off in sequential decision problems. In clinical trials, which are sensitive to outlier data, the goal is to learn a risk-averse policy to provide a trade-off between exploration, exploitation, and safety. In this paper, we present a risk-averse multi-armed bandit algorithm to solve a decision-making problem based on the social engagement behaviors of children with Autism Spectrum Disorder (ASD). The algorithm is carried out when children interact with a humanoid robot and imitate a sequence of the robot's movements. The proposed algorithm is based on the Best Empirical Sampled Average algorithm under Entropic Value-at-Risk as a risk measure to decide on the best sequence of movements that can improve the social engagement behaviors of the children with ASD while imitating the robot's movements. We provide a detailed experimental analysis to compare the performance of our proposed algorithm to some well-known risk-averse multi-armed bandit algorithms on some artificial scenarios and our real-world problem. The experimental results report that the proposed algorithm outperforms its competitors in terms of robustness, risk avoidance, and cumulative regret, promoting the social engagement behaviors of children with ASD when imitating a robot's movements.

Risk-Averse Caching Policies for YouTube Content in Femtocell Networks using Density Forecasting

The paper presents risk-neutral and risk-averse caching policies that can be deployed in a femtocell network with limited storage capacity to reduce the time delay of servicing content requests. The caching policies use a forecasting algorithm to estimate the cumulative distribution function of content requests based on the content features. Given the cumulative distribution function, a mixed-integer linear program is used to compute where to cache content in the femtocell network. The caching policies account for the uncertainty associated with estimating the content requests using the coherent Conditional Value-at-Risk (CVaR) measure. For a large number of content, a risk-neutral caching policy is constructed that accounts for both the content features and routing protocol that only requires the evaluation of a unimodular linear program. Using data from YouTube (comprising 25,000 videos) and the NS-3 simulator, the caching policies reduce the delay of retrieving content in femtocell networks compared with industry standard caching policies. Specifically, a 6 percent reduction in delay is achieved by accounting for the uncertainty, and a 60 percent reduction in delay is achieved if both the uncertainty and femtocell routing protocol are accounted for compared to the risk-neutral caching policy that neglects the routing protocol.

Alternative Work Arrangements, Payout Policy, and Employment: Evidence from Independent Contractor Misclassification Statute

Using the 2004 Massachusetts Independent Contractor Law that discourages the use of independent contractors, I find that firms adopt conservative financial policies by reducing payout since companies attempt to reduce financial risk in response to increased operating leverage. I find that the effect is more pronounced in industries with a higher proportion of independent contractors, and in firms with high financial constraints and operating leverage. I also find that the state government’s attempt to improve the mis-classified worker’s status reduces the number of employees at the firm level by 4.4%, highlighting that reclassification does not lead to an increase in the number of employees. The findings imply that heightened operating risk caused by the law may lead to risk-averse financial policies and subdued job creation at the firm level.

Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

Markov decision processes (MDPs) are the defacto framework for sequential decision making in the presence of stochastic uncertainty. A classical optimization criterion for MDPs is to maximize the expected discounted-sum payoff, which ignores low probability catastrophic events with highly negative impact on the system. On the other hand, risk-averse policies require the probability of undesirable events to be below a given threshold, but they do not account for optimization of the expected payoff. We consider MDPs with discounted-sum payoff with failure states which represent catastrophic outcomes. The objective of risk-constrained planning is to maximize the expected discounted-sum payoff among risk-averse policies that ensure the probability to encounter a failure state is below a desired threshold. Our main contribution is an efficient risk-constrained planning algorithm that combines UCT-like search with a predictor learned through interaction with the MDP (in the style of AlphaZero) and with a risk-constrained action selection via linear programming. We demonstrate the effectiveness of our approach with experiments on classical MDPs from the literature, including benchmarks with an order of 106 states.

Risk-averse flexible policy on ambulance allocation in humanitarian operations under uncertainty

Proactive ambulance management is constructive to improve the response efficiency for emergency medical service (EMS) systems under uncertainty. In this paper, we present a dynamic optimisation model concerning the ambulance dispatching and relocation. We develop a flexible operation policy driven by the interval rolling to match vehicles with calls in batch. We formulate the problem in Markov Decision Process and incorporate queues to minimise the average response and delay time. Considering the curse-of-dimensionality, we provide a simulation-based empirical dynamic programming with the state aggregation and post-decision state to solve the model. To further accelerate the computational efficiency, a greedy heuristic method is introduced to improve the quality of sampling operations. Then, a risk-averse model is developed based on the stochastic dominance strategy to improve operational reliability. We develop an equivalent linear programming to evaluate concave dominating functions. We test the performance by a numerical case and extract managerial insights for practitioners. Our results show that the proposed flexible and risk-averse solution outperforms the classic model on reducing the delay under uncertain calls. And the improvement is more active during peak hours, when real-time needs exceed available ambulances.

Optimizing conditional value-at-risk in dynamic pricing

Many industries use dynamic pricing on an operational level to maximize revenue from selling a fixed capacity over a finite horizon. Classical risk-neutral approaches do not accommodate the risk aversion often encountered in practice. We add to the scarce literature on risk aversion by considering the risk measure conditional value-at-risk (CVaR), which recently became popular in areas like finance, energy, or supply chain management. A key aspect of this paper is selling a single unit of capacity, which is highly relevant in, for example, the real estate market. We analytically derive the optimal policy and obtain structural results. The most important managerial implication is that the risk-averse optimal price is constant over large parts of the selling horizon, whereas the price continuously declines in the standard setting of risk-neutral dynamic pricing. This offers a completely new explanation for the price-setting behavior often observed in practice. For arbitrary capacity, we develop two algorithms to efficiently compute the value function and evaluate them in a numerical study. Our results show that applying a risk-averse policy, even a static one, often yields a higher CVaR than applying a dynamic, but risk-neutral, policy.

What does New Delhi’s engagement with the war in Syria (and Iraq) reveal about India as an International Actor?

ABSTRACTThis article examines whether New Delhi’s engagement with the ongoing conflict in Syria (and Iraq) demonstrates that India is leaving behind its traditionally cautious and risk-averse foreign policy stance and becoming a more proactive international actor. India has not engaged actively with and has not tried to wield substantial influence over the course of this conflict; instead, after a moderate burst of diplomatic activity in the second half of 2011 India has largely tried to avoid involvement. There are some indications that India is trying to become more proactive in its region, in trade diplomacy, and toward a rising China; yet, results sometimes fail to meet ambitions. Accordingly, the article concludes that, on balance, India’s foreign policy orientation has not yet changed significantly; it remains a cautious, reactive international actor unwilling to engage with and actively shape the outcome of crucial crises or conflicts.

Mean Variance Optimization in Markov Decision Processes with State-Dependent Risk Aversion

In many revenue management applications risk aversion is crucial. In dynamic decision problems it is challenging to find the right balance between maximizing expected profits and minimizing their variance. In this paper, we present an efficient dynamic programming approach to leverage mean variance optimization in discounted Markov decision processes with finite time horizon. We show how to derive optimal risk-sensitive policies which allow for state-dependent mean variance balancing. We compare our approach to risk-neutral policies as well as risk-averse policies of the exponential utility model. Based on state-probabilities, we analytically evaluate the expected evolution of accumulated profits as well as profit variances. Our general concepts are demonstrated for dynamic pricing applications.

Radical practice in a risk-averse environment: Learning from ATD Fourth World UK

INTRODUCTION: The escalation of coercive, risk-averse policy directives in Aotearoa New Zealand’s child and family social work sphere is undermining the profession’s potential to meet its social justice, human rights based aspirations. Social workers may need to look further afield for best practice models that facilitate emancipatory practice in neoliberal social policy environments. This article posits the radical practice of anti-poverty organisation ATD Fourth World in England (where child protection is characteristically risk-averse, individualised and coercive), as an alternative for work with families experiencing poverty and social exclusion.METHODS: We drew on the voices of ATD Fourth World activists cited in previous publications, alongside Activists(a-d) interviewed specifically for this article, and Activist(e) who contributed at a roundtable discussion on a previous project. Interviews focused on ATD Fourth World’s approach to working with families in poverty; three distinctive aspects emerged: the organisation’s philosophy on poverty, and its collaborative and relational family support model. We contrasted these three aspects with state child protection policies in Aotearoa New Zealand and England.FINDINGS: The often inflexible, top-down nature of state child protection policies, coupled with an atmosphere of policing, control and disregard for the impact of poverty, constrain social workers and families alike, eroding the crucial social worker/family relationship underpinning best practice. ATD Fourth World’s approach suggests that genuine strengths-based practice relies on nuanced understandings of poverty, a commitment to advance families’ wishes, and trusting relationships grounded in human dignity and commonality.CONCLUSION: The Aotearoa New Zealand reforms may amplify coercive, risk-averse tendencies in the state’s child protection system. Child and family social workers could consider adapting aspects of ATD Fourth World’s approach to resist or mitigate these coercive aspects and steer the reforms’ implementation in more emancipatory directions.

Risk-Sharing Benefits and the Capital Structure of Insurance Companies

Providing risk-sharing benefits to risk-averse policy holders is a primary function of insurance companies. We model that policy holders are paying a fee over the present value of indemnifications (i.e., technical provisions) to enjoy these risk-sharing benefits. This fee implies that a capital structure largely consisting of technical provisions is optimal for insurance firms, making the traditional Modigliani-Miller logic inappropriate for them. To support the issuance of technical provisions with socially desirable properties, insurance firms hold a surplus to absorb losses. We show that the Modigliani-Miller logic applies to the composition of this loss-absorption capacity. This explains why insurance companies may use, next to equity and technical provisions, financial debt in supporting their activities.