Sequential Decision-making Task Research Articles

Market Making With Signals Through Deep Reinforcement Learning

Deep reinforcement learning has recently been successfully applied to a plethora of diverse and difficult sequential decision-making tasks, ranging from the Atari games to robotic motion control. Among the foremost such tasks in quantitative finance is the problem of optimal market making. Market making is the process of simultaneously quoting limit orders on both sides of the limit order book of a security with the goal of repeatedly capturing the quoted spread while minimizing the inventory risk. Most of the existing analytical approaches to market making tend to be predicated on a set of strong, naive assumptions, whereas current machine learning-based approaches either resort to crudely discretized quotes or fail to incorporate additional predictive signals. In this paper, we present a novel framework for market making with signals based on model-free deep reinforcement learning, addressing these shortcomings. A new state space formulation incorporating outputs from standalone signal generating units, as well as a novel action space and reward function formulation, are introduced. The framework is underpinned by both ideas from adversarial reinforcement learning and neuroevolution. Experimental results on historical data demonstrate the superior reward-to-risk performance of the proposed framework over several standard market making benchmarks. More specifically, the resulting reinforcement learning agent achieves between 20-30% higher terminal wealth than the benchmarks while being exposed to only around 60% of their inventory risks. Finally, an insight into its policy is provided for the sake of interpretability.

Exploiting Relevance for Online Decision-Making in High-Dimensions

Many sequential decision-making tasks require choosing at each decision step the right action out of the vast set of possibilities by extracting actionable intelligence from high-dimensional data streams. Most of the times, the high dimensionality of actions and data makes learning of the optimal actions by traditional learning methods impracticable. In this work, we investigate how to discover and leverage sparsity in actions and data to enable fast learning. As our learning model, we consider a structured contextual multi-armed bandit (CMAB) with high dimensional arm (action) and context (data) sets, where the rewards depend only on a few relevant dimensions of the joint context-arm set, possibly in a non-linear way. We depart from the prior work by assuming a high-dimensional, continuum set of arms, and allow relevant context dimensions to vary for each arm. We propose anew online learning algorithm called CMAB with Relevance Learning (CMAB-RL). CMAB-RL enjoys a substantially improved regret bound compared to classical CMAB algorithms whose regrets depend on the number of dimensions dx and da of the context and arm sets. Importantly, we show that when the learner has prior knowledge on sparsity, given in terms of upper bounds dx and da on the number of relevant context and arm dimensions, then CMABRL achieves O(T <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1-1/(2+2dx+da)</sup> ) regret. Finally, we illustrate how CMAB algorithms can be used for optimal personalized blood glucose control in type 1 diabetes mellitus patients, and show that CMAB-RL outperforms other contextual MAB algorithms in this task.

An Optimized Parallel Implementation of Non-Iteratively Trained Recurrent Neural Networks

Abstract Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation through time (BPTT) which is prohibitively expensive, especially when the length of the time dependencies and the number of hidden neurons increase. To reduce the training time, extreme learning machines (ELMs) have been recently applied to RNN training, reaching a 99% speedup on some applications. Due to its non-iterative nature, ELM training, when parallelized, has the potential to reach higher speedups than BPTT. In this work, we present Opt-PR-ELM, an optimized parallel RNN training algorithm based on ELM that takes advantage of the GPU shared memory and of parallel QR factorization algorithms to efficiently reach optimal solutions. The theoretical analysis of the proposed algorithm is presented on six RNN architectures, including LSTM and GRU, and its performance is empirically tested on ten time-series prediction applications. Opt-PR-ELM is shown to reach up to 461 times speedup over its sequential counterpart and to require up to 20x less time to train than parallel BPTT. Such high speedups over new generation CPUs are extremely crucial in real-time applications and IoT environments.

Reliance on model-based and model-free control in obesity

Consuming more energy than is expended may reflect a failure of control over eating behaviour in obesity. Behavioural control arises from a balance between two dissociable strategies of reinforcement learning: model-free and model-based. We hypothesized that weight status relates to an imbalance in reliance on model-based and model-free control, and that it may do so in a linear or quadratic manner. To test this, 90 healthy participants in a wide BMI range [normal-weight (n = 31), overweight (n = 29), obese (n = 30)] performed a sequential decision-making task. The primary analysis indicated that obese participants relied less on model-based control than overweight and normal-weight participants, with no difference between overweight and normal-weight participants. In line, secondary continuous analyses revealed a negative linear, but not quadratic, relationship between BMI and model-based control. Computational modelling of choice behaviour suggested that a mixture of both strategies was shifted towards less model-based control in obese participants. Our findings suggest that obesity may indeed be related to an imbalance in behavioural control as expressed in a phenotype of less model-based control potentially resulting from enhanced reliance on model-free computations.

Learning adversarial attack policies through multi-objective reinforcement learning

Deep Reinforcement Learning has shown promising results in learning policies for complex sequential decision-making tasks. However, different adversarial attack strategies have revealed the weakness of these policies to perturbations to their observations. Most of these attacks have been built on existing adversarial example crafting techniques used to fool classifiers, where an adversarial attack is considered a success if it makes the classifier outputs any wrong class. The major drawback of these approaches when applied to decision-making tasks is that they are blind for long-term goals. In contrast, this paper suggests that it is more appropriate to view the attack process as a sequential optimization problem, with the aim of learning a sequence of attacks, where the attacker must consider the long-term effects of each attack. In this paper, we propose that such an attack policy must be learned with two objectives in view. On the one hand, the attack must pursue the maximum performance loss of the attacked policy. On the other hand, it also should minimize the cost of the attacks. Therefore, in this paper we propose a novel modelization of the process of learning an attack policy as a Multi-objective Markov Decision Process with two objectives: maximizing the performance loss of the attacked policy and minimizing the cost of the attacks. We also reveal the conflicting nature of these two objectives and use a Multi-objective Reinforcement Learning algorithm to draw the Pareto fronts for four well-known tasks: the GridWorld, the Cartpole, the Mountain car and the Breakout.

A linear threshold model for optimal stopping behavior

In many real-life decisions, options are distributed in space and time, making it necessary to search sequentially through them, often without a chance to return to a rejected option. The optimal strategy in these tasks is to choose the first option that is above a threshold that depends on the current position in the sequence. The implicit decision-making strategies by humans vary but largely diverge from this optimal strategy. The reasons for this divergence remain unknown. We present a model of human stopping decisions in sequential decision-making tasks based on a linear threshold heuristic. The first two studies demonstrate that the linear threshold model accounts better for sequential decision making than existing models. Moreover, we show that the model accurately predicts participants' search behavior in different environments. In the third study, we confirm that the model generalizes to a real-world problem, thus providing an important step toward understanding human sequential decision making.

S146. ASSOCIATION OF IMPAIRED MODEL-FREE DECISION-MAKING WITH ABERRANT STRIATAL DOPAMINE, BRAIN ACTIVATION, AND COGNITIVE DIFFICULTIES IN PATIENTS WITH SCHIZOPHRENIA DURING PSYCHOTIC REMISSION

BackgroundHuman decision-making ranges between the extremes of automatic and fast model-free behavior (i.e., relying only on previous outcomes) and more flexible, but computationally demanding model-based behavior (i.e., implementing cognitive models). Model-based/model-free decision-making can be investigated using sequential decision tasks and has been shown to be associated with presynaptic striatal dopamine synthesis. During phases of psychotic remission in schizophrenia, dopamine synthesis in the dorsal striatum is reduced. We hypothesized that particularly model-free decision-making is impaired in schizophrenia during psychotic remission and is associated with (i) abnormal dopamine synthesis in dorsal striatum, (ii) aberrant task-activation in dorsal striatum, and (iii) cognitive difficulties in patients (e.g., reduced speed).Methods26 patients with chronic schizophrenia, currently in psychotic remission, and 22 healthy controls (matched by age and gender) were enrolled in the study. Model-based/model-free decision-making was evaluated with a two-stage Markov decision task, followed by computational modeling of subjects’ learning behavior. Presynaptic dopamine synthesis was assessed by 18F-DOPA positron emission tomography and subsequent graphical Patlak analysis. Task-activation was measured by functional magnetic resonance imaging. Cognitive impairments were quantified by Trail-Making-Test A (among others). Associations between decision-making parameters, dopamine synthesis, task-activation, and cognitive impairments were tested by correlation analyses.ResultsPatients with schizophrenia showed selectively impaired model-free decision-making. 18F-DOPA uptake (i.e., presynaptic dopamine synthesis capacity) in the dorsal striatum was decreased in patients. Impaired model-free decision-making in patients correlated with (i) decreased dopamine synthesis in dorsal striatum, (ii) abnormal task-activation in dorsal striatum, and (iii) lower speed in Trail-Making-Test A.DiscussionResults demonstrate an association of reduced dorsal striatal dopamine synthesis and brain activity with impaired model-free decision-making in schizophrenia, which potentially contributes to cognitive difficulties.

Back to the Future – Temporal Adaptation of Text Representations

Language evolves over time in many ways relevant to natural language processing tasks. For example, recent occurrences of tokens 'BERT' and 'ELMO' in publications refer to neural network architectures rather than persons. This type of temporal signal is typically overlooked, but is important if one aims to deploy a machine learning model over an extended period of time. In particular, language evolution causes data drift between time-steps in sequential decision-making tasks. Examples of such tasks include prediction of paper acceptance for yearly conferences (regular intervals) or author stance prediction for rumours on Twitter (irregular intervals). Inspired by successes in computer vision, we tackle data drift by sequentially aligning learned representations. We evaluate on three challenging tasks varying in terms of time-scales, linguistic units, and domains. These tasks show our method outperforming several strong baselines, including using all available data. We argue that, due to its low computational expense, sequential alignment is a practical solution to dealing with language evolution.

Modeling Dynamic Allocation of Effort in a Sequential Task Using Discounting Models.

Most rewards in our lives require effort to obtain them. It is known that effort is seen by humans as carrying an intrinsic disutility which devalues the obtainable reward. Established models for effort discounting account for this by using participant-specific discounting parameters inferred from experiments. These parameters offer only a static glance into the bigger picture of effort exertion. The mechanism underlying the dynamic changes in a participant's willingness to exert effort is still unclear and an active topic of research. Here, we modeled dynamic effort exertion as a consequence of effort- and probability-discounting mechanisms during goal reaching, sequential behavior. To do this, we developed a novel sequential decision-making task in which participants made binary choices to reach a minimum number of points. Importantly, the time points and circumstances of effort allocation were decided by participants according to their own preferences and not imposed directly by the task. Using the computational model to analyze participants' choices, we show that the dynamics of effort exertion arise from a combination of changing task needs and forward planning. In other words, the interplay between a participant's inferred discounting parameters is sufficient to explain the dynamic allocation of effort during goal reaching. Using formal model comparison, we also inferred the forward-planning strategy used by participants. The model allowed us to characterize a participant's effort exertion in terms of only a few parameters. Moreover, the model can be adapted to a number of tasks used in establishing the neural underpinnings of forward-planning behavior and meta-control, allowing for the characterization of behavior in terms of model parameters.

Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making.

State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is currently unknown whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we studied how state and action representations adapt to reflect the structure of the world when such a priori knowledge is not available. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were unknown a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. Similarly, we found that the set of actions expanded with training, although the emergence of new action sequences was sensitive to the experimental parameters and specifics of the training procedure. We conclude that the profile of choices shows a gradual shift from simple representations to more complex structures compatible with the structure of the world.

Bridging Commonsense Reasoning and Probabilistic Planning via a Probabilistic Action Language

AbstractTo be responsive to dynamically changing real-world environments, an intelligent agent needs to perform complex sequential decision-making tasks that are often guided by commonsense knowledge. The previous work on this line of research led to the framework calledinterleaved commonsense reasoning and probabilistic planning(icorpp), which used P-log for representing commmonsense knowledge and Markov Decision Processes (MDPs) or Partially Observable MDPs (POMDPs) for planning under uncertainty. A main limitation of icorppis that its implementation requires non-trivial engineering efforts to bridge the commonsense reasoning and probabilistic planning formalisms. In this paper, we present a unified framework to integrate icorpp’s reasoning and planning components. In particular, we extend probabilistic action languagepBC+ to express utility, belief states, and observation as in POMDP models. Inheriting the advantages of action languages, the new action language provides an elaboration tolerant representation of POMDP that reflects commonsense knowledge. The idea led to the design of the systempbcplus2pomdp, which compiles apBC+ action description into a POMDP model that can be directly processed by off-the-shelf POMDP solvers to compute an optimal policy of thepBC+ action description. Our experiments show that it retains the advantages of icorppwhile avoiding the manual efforts in bridging the commonsense reasoner and the probabilistic planner.

Metacontrol of decision-making strategies in human aging.

Humans employ different strategies when making decisions. Previous research has reported reduced reliance on model-based strategies with aging, but it remains unclear whether this is due to cognitive or motivational factors. Moreover, it is not clear how aging affects the metacontrol of decision making, that is the dynamic adaptation of decision-making strategies to varying situational demands. In this cross-sectional study, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based strategies. In contrast to previous research, model-based strategies led to higher payoffs. Moreover, we manipulated the costs and benefits of model-based strategies by varying reward magnitude and the stability of the task structure. Compared to younger adults, older adults showed reduced model-based decision making and less adaptation of decision-making strategies. Our findings suggest that aging affects the metacontrol of decision-making strategies and that reduced model-based strategies in older adults are due to limited cognitive abilities.

Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications

Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand the minimum set of demonstrations needed to teach a specific sequential decisionmaking task. We formalize the problem of finding maximally informative demonstrations for IRL as a machine teaching problem where the goal is to find the minimum number of demonstrations needed to specify the reward equivalence class of the demonstrator. We extend previous work on algorithmic teaching for sequential decision-making tasks by showing a reduction to the set cover problem which enables an efficient approximation algorithm for determining the set of maximallyinformative demonstrations. We apply our proposed machine teaching algorithm to two novel applications: providing a lower bound on the number of queries needed to learn a policy using active IRL and developing a novel IRL algorithm that can learn more efficiently from informative demonstrations than a standard IRL approach.

Depression and Sequential Decision-Making Revisited.

Background: The effect of depression on decision-making is an important but still an unsettled issue. Although most studies have reported that clinically depressed participants show worse performance, there are also studies that have shown no or even positive effects. Specifically, von Helversen et al. (2011) were able to document a positive effect of depression on task performance in a sequential decision-making task called the secretary problem (SP). Here, we (1) aimed to replicate this study in an extended version using more trials and (2) modified it by including an additional condition in which negative feedback was given.Method: Eighty-two participants took part. They were split into two groups: 20/21 participants with major depression disorder (MDD) and 20/21 matched healthy participants. Participants completed the secretary problem either in the standard or in a modified version. Additionally, they answered questionnaires for assessing depression, personality, and intelligence.Results: We did not find any significant differences between clinically depressed and nondepressed individuals in any indicators of task performance, under both the original and modified conditions.Limitations: Our participants were ambulatory patients. The quality of depression may have been therefore less extreme. We did not assess or control for rumination.Conclusions: We were not able to detect any significant differences between the performances of healthy and clinically depressed participants in a sequential decision-making task.

Reward and avoidance learning in the context of aversive environments and possible implications for depressive symptoms

BackgroundAversive stimuli in the environment influence human actions. This includes valence-dependent influences on action selection, e.g., increased avoidance but decreased approach behavior. However, it is yet unclear how aversive stimuli interact with complex learning and decision-making in the reward and avoidance domain. Moreover, the underlying computational mechanisms of these decision-making biases are unknown.MethodsTo elucidate these mechanisms, 54 healthy young male subjects performed a two-step sequential decision-making task, which allows to computationally model different aspects of learning, e.g., model-free, habitual, and model-based, goal-directed learning. We used a within-subject design, crossing task valence (reward vs. punishment learning) with emotional context (aversive vs. neutral background stimuli). We analyzed choice data, applied a computational model, and performed simulations.ResultsWhereas model-based learning was not affected, aversive stimuli interacted with model-free learning in a way that depended on task valence. Thus, aversive stimuli increased model-free avoidance learning but decreased model-free reward learning. The computational model confirmed this effect: the parameter lambda that indicates the influence of reward prediction errors on decision values was increased in the punishment condition but decreased in the reward condition when aversive stimuli were present. Further, by using the inferred computational parameters to simulate choice data, our effects were captured. Exploratory analyses revealed that the observed biases were associated with subclinical depressive symptoms.ConclusionOur data show that aversive environmental stimuli affect complex learning and decision-making, which depends on task valence. Further, we provide a model of the underlying computations of this affective modulation. Finally, our finding of increased decision-making biases in subjects reporting subclinical depressive symptoms matches recent reports of amplified Pavlovian influences on action selection in depression and suggests a potential vulnerability factor for mood disorders. We discuss our findings in the light of the involvement of the neuromodulators serotonin and dopamine.

Sensorimotor subthalamic stimulation restores risk-reward trade-off in Parkinson's disease.

STN-DBS effectively treats motor symptoms of advanced PD. Nonmotor cognitive symptoms, such as impaired impulse control or decision making, may either improve or worsen with DBS. A potential mediating factor of DBS-induced modulation of cognition is the electrode position within the STN with regard to functional subareas of parallel motor, cognitive, and affective basal ganglia loops. However, to date, the volume of tissue activated and weighted stimulation of STN motor versus nonmotor territories are yet to be linked to differential DBS effects on cognition. We aim to investigate whether STN-DBS influences risk-reward trade-off decisions and analyze its dependency on electrode placement. Seventeen PD patients ON and OFF STN-DBS and 17 age-matched healthy controls conducted a sequential decision-making task with escalating risk and reward. We computed the effect of STN-DBS on risk-reward trade-off decisions, localized patients' bilateral electrodes, and analyzed the predictive value of volume of tissue activated in STN motor and nonmotor territories on behavioral change. We found that STN-DBS not only improves PD motor symptoms, but also normalizes overly risk-averse decision behavior in PD. Intersubject variance in electrode location could explain this behavioral change. Specifically, if STN-DBS activated preferentially STN motor territory, patients' risk-reward trade-off decisions more resembled those of healthy controls. Our findings support the notion of convergence of different functional circuits within the STN and imply a positive effect of well-placed STN-DBS on nonmotor cognitive functioning in PD. © 2018 International Parkinson and Movement Disorder Society.

Transcranial stimulation over right inferior frontal gyrus increases the weight given to private information during sequential decision-making.

Decision makers often follow other similarly situated people in making decisions, creating a sequential decision-making context. Although rational behavior is often to make the same choice as previous decision makers, which can result in an information cascade, people may assign inappropriately higher weight to their own private information and discount public information about predecessors’ choices. Recent findings suggest that overweighting private information may be associated with increased activities in the inferior frontal gyrus (IFG). In the present study, we employed transcranial direct current stimulation (tDCS) and developed a computational model to examine the causal relationship between right IFG (rIFG) and overweighting private information. Specifically, we applied three types of tDCS over rIFG while participants were completing a sequential decision-making task. Our results showed that anodal stimulation significantly increased the weight given to private information and decreased the response time in making a decision when private information conflicted with public information, but cathodal stimulation did not have such impacts. Importantly, the effect of anodal stimulation was significant in some conditions when information conflict or task difficulty reached a threshold that might trigger cognitive control-related processes. Our findings revealed the important role of rIFG in trade-off between considering private and public information during sequential decision-making.

Context-Dependent Risk Aversion: A Model-Based Approach.

Most research on risk aversion in behavioral science with human subjects has focused on a component of risk aversion that does not adapt itself to context. More recently, studies have explored risk aversion adaptation to changing circumstances in sequential decision-making tasks. It is an open question whether one can identify evidence, at the single subject level, for such risk aversion adaptation. We conducted a behavioral experiment on human subjects, using a sequential decision making task. We developed a model-based approach for estimating the adaptation of risk-taking behavior with single-trial resolution by modeling a subject's goals and internal representation of task contingencies. Using this model-based approach, we estimated the subject-specific adaptation of risk aversion depending on the current task context. We found striking inter-subject variations in the adaptation of risk-taking behavior. We show that these differences can be explained by differences in subjects' internal representations of task contingencies and goals. We discuss that the proposed approach can be adapted to a wide range of experimental paradigms and be used to analyze behavioral measures other than risk aversion.

Curriculum Design for Machine Learners in Sequential Decision Tasks

Existing work in machine learning has shown that algorithms can benefit from the use of curricula—learning first on simple examples before moving to more difficult problems. This work studies the curriculum-design problem in the context of sequential decision tasks, analyzing how different curricula affect learning in a Sokoban-like domain, and presenting the results of a user study that explores whether nonexperts generate effective curricula. Our results show that 1) the way in which evaluative feedback is given to the agent as it learns individual tasks does not affect the relative quality of different curricula, 2) nonexpert users can successfully design curricula that result in better overall performance than having the agent learn from scratch, and 3) nonexpert users can discover and follow salient principles when selecting tasks in a curriculum. We also demonstrate that our curriculum-learning algorithm can be improved by incorporating the principles people use when designing curricula. This work gives us insights into the development of new machine-learning algorithms and interfaces that can better accommodate machine- or human-created curricula.

The Reciprocal Relationships Between Escalation, Anger, and Confidence in Investment Decisions Over Time.

Research on escalation of commitment has predominantly been studied in the context of a single decision without consideration for the psychological consequences of escalating. This study sought to examine (a) the extent to which people escalate their commitment to a failing course of action in a sequential decision-making task, (b) confidence and anger as psychological consequences of escalation of commitment, and (c) the reciprocal relationship between escalation of commitment and confidence and anger. Participants were 110 undergraduate students who completed a series of investment decisions regarding a failing endeavor. Results revealed that although a high proportion of individuals escalate through all decisions, the extent to which they escalated decreased with each decision as they were less willing to invest money in the project. Furthermore, as participants escalated, confidence in one’s decision decreased and anger increased. Lastly, the analyses revealed that the relationship between escalation and confidence is reciprocal. Escalation was negatively associated with confidence, and confidence predicted escalation in the subsequent decision. These results highlight the importance of considering both the determinants and psychological consequences of escalation of commitment.