Final Reward Research Articles

Dream, Bliss, and the Shaping of Emotional Meaning in Beowulf

<i>Dream</i>, <i>Bliss</i>, and the Shaping of Emotional Meaning in <i>Beowulf</i>

DrugEx v2: de novo design of drug molecules by Pareto-based multi-objective reinforcement learning in polypharmacology

In polypharmacology drugs are required to bind to multiple specific targets, for example to enhance efficacy or to reduce resistance formation. Although deep learning has achieved a breakthrough in de novo design in drug discovery, most of its applications only focus on a single drug target to generate drug-like active molecules. However, in reality drug molecules often interact with more than one target which can have desired (polypharmacology) or undesired (toxicity) effects. In a previous study we proposed a new method named DrugEx that integrates an exploration strategy into RNN-based reinforcement learning to improve the diversity of the generated molecules. Here, we extended our DrugEx algorithm with multi-objective optimization to generate drug-like molecules towards multiple targets or one specific target while avoiding off-targets (the two adenosine receptors, A1AR and A2AAR, and the potassium ion channel hERG in this study). In our model, we applied an RNN as the agent and machine learning predictors as the environment. Both the agent and the environment were pre-trained in advance and then interplayed under a reinforcement learning framework. The concept of evolutionary algorithms was merged into our method such that crossover and mutation operations were implemented by the same deep learning model as the agent. During the training loop, the agent generates a batch of SMILES-based molecules. Subsequently scores for all objectives provided by the environment are used to construct Pareto ranks of the generated molecules. For this ranking a non-dominated sorting algorithm and a Tanimoto-based crowding distance algorithm using chemical fingerprints are applied. Here, we adopted GPU acceleration to speed up the process of Pareto optimization. The final reward of each molecule is calculated based on the Pareto ranking with the ranking selection algorithm. The agent is trained under the guidance of the reward to make sure it can generate desired molecules after convergence of the training process. All in all we demonstrate generation of compounds with a diverse predicted selectivity profile towards multiple targets, offering the potential of high efficacy and low toxicity.

Bending the Color Line: Hidden Identities and Sexual Transgressions in Charles W. Chesnutt's Short Stories

Bending the Color LineHidden Identities and Sexual Transgressions in Charles W. Chesnutt's Short Stories Anna Pochmara (bio) A major strand of African American fiction published at the turn of the twentieth century is structured by a revelation of an intimate secret. In most cases, the hidden information concerns the characters' racial identities and interracial family histories. This figuration is so prevalent that, according to M. Giulia Fabi, an analysis of "the tropes of miscegenation and passing" can best elucidate the central significance, literary artistry, and ideological complexity of these frequently underestimated and understudied narratives (2).1 To different extents, these works are shaped by the melodramatic mode, which "draws into a public arena desires, fears, values and identities which lie beneath the surface of the publicly acknowledged world" (Gledhill 33). This drive towards disclosure is most conspicuously visible in coincidental race-revelation scenes that propel many Black works of the age of realism. Typically for literary recognitions, these moments reveal a "perverse" and "improper" knowledge, a "monstrous transgression of a taboo" (Cave 3, 7, 209). In African American realist narratives, the revelation of the secret does not, however, result in a restoration of order or an unambiguous closure, which characterize conventional melodrama. On the contrary, the texts' endings are frequently open and depart from the "final reward of virtue" and punishment of villainy (Brooks 12). The works question both the melodramatic opposition between good and evil and the polarized division into whiteness and Blackness, which was central for Jim Crow racial classifications. The hidden identities and race revelations in African American fiction challenge the "rigid color line" and expose its invisibility and volatility (Chesnutt, "A Stream" 851). The melodramatic element that permeates a significant segment of turn-of-the-twentieth-century African American narratives is not exceptional when juxtaposed with the corpus of canonical U.S. fiction [End Page 56] of the period. Although traditionally perceived as a reaction against romantic, sentimentalist, and melodramatic aesthetics, American realist and naturalist narratives continued to employ romance, coincidence, and sensationalism rather than complete plausibility, exceptionalism rather than representativeness, and dramatic contrasts rather than a balanced perspective on social life (Chase 1–11; Pizer 264–69; Trachtenberg 190–91). As Keith Newlin points out, the melodramatic aesthetic is key for the works of the naturalist canon, which evoke an immediate reaction in the reader and express a Spencerian, determinist vision of the world (5–15). Similarly, elements of melodrama can also be found in realism. One of the most influential studies of this aesthetic, Peter Brooks's The Melodramatic Imagination, devotes an entire section to Henry James, a canonical realist author. His works, according to Brooks, express an "intensification of consciousness," and the characters' ethical conflicts and choices are imaged "in the polarized and heightened mode of melodrama" (197; see also 153–97). In Charles W. Chesnutt's "Her Virginia Mammy" and "Cicely's Dream," melodrama is key to his narrative aesthetic and it enables him to bend the "rigid color line" to reveal its absurdities. Just as in the case of James's writing, Chesnutt's use of melodrama stems from his specific approach to realism. As Ryan Simmons states, for Chesnutt "realism's purpose is not to document histories but to disassemble and re-create readers' methods for understanding these histories, all with the intent of changing not only what readers know, but also how they know it and how they are capable of responding to it" (3). His fiction is "designed to instill an understanding of racism" because "reality properly portrayed and rightly understood must compel action" (3, 5). Such an understanding of realism is superficially similar to the naturalist novel, which "moralizes or sets out to demonstrate a particular thesis" (Newlin 10). Yet, although Chesnutt diverts from the traditional program of objective, documentary realism (Becker 184–97), his fiction does not represent "a melodramatic vision of human beings at the mercy of forces over which they have little control but whose purpose is ultimately intelligible" (Newlin 15). Instead, he portrays self-aware, conflicted characters who interact with social forces that are too complex to be explained with Spencerian determinism. In Chesnutt's works, the melodramatic vision does not...

"He may exalt you at the expected time" (1 Pet. 5:6). The Conception of Time in the First Letter of Peter

In the First Letter of St. Peter we find a clear temporal direction toward a final reward, which will be a divine exaltation and appreciation. Its time is always specified as καιρός, as a concrete moment, a point in time. Otherwise χρόνος, as durative time, always specifies the time of earthly existence, especially from the beginning of Christian life. It is directed toward the final and definitive καιρός. This temporal direction is especially visible in many terms with the preposition εἰς, as hope, heritage in heaven, salvation, exaltation, appreciation, light, and the glory of God. These are especially frequent in the initial part of the Petrine Letter.

Data Efficient Reinforcement Learning for Integrated Lateral Planning and Control in Automated Parking System.

Reinforcement learning (RL) is a promising direction in automated parking systems (APSs), as integrating planning and tracking control using RL can potentially maximize the overall performance. However, commonly used model-free RL requires many interactions to achieve acceptable performance, and model-based RL in APS cannot continuously learn. In this paper, a data-efficient RL method is constructed to learn from data by use of a model-based method. The proposed method uses a truncated Monte Carlo tree search to evaluate parking states and select moves. Two artificial neural networks are trained to provide the search probability of each tree branch and the final reward for each state using self-trained data. The data efficiency is enhanced by weighting exploration with parking trajectory returns, an adaptive exploration scheme, and experience augmentation with imaginary rollouts. Without human demonstrations, a novel training pipeline is also used to train the initial action guidance network and the state value network. Compared with path planning and path-following methods, the proposed integrated method can flexibly co-ordinate the longitudinal and lateral motion to park a smaller parking space in one maneuver. Its adaptability to changes in the vehicle model is verified by joint Carsim and MATLAB simulation, demonstrating that the algorithm converges within a few iterations. Finally, experiments using a real vehicle platform are used to further verify the effectiveness of the proposed method. Compared with obtaining rewards using simulation, the proposed method achieves a better final parking attitude and success rate.

Scientific writing and publishing with IMRaD

The current trend of grading scientists depending on the amount and quality of scientific papers led to a pressure in publishing, which now tends to be reduced by opting for high-quality publications. Experimental research, the largest part of scientific research, is by far best presented in research articles – abbreviated IMRaD – a form of presentation developed in the last 100 years and now almost standardized. Even though writing seems simple to ever scientist, its rules can be learned by practice. It is of utmost importance that a well conducted science is presented in a properly written manuscript, as there are cases when one was delayed in publication due to a bad scientific writing. As with any other type of research papers, IMRaD also requires good knowledge when its main parts are concerned, in structuring its different sections, in using the most suitable language, in conducting the final improvements before submitting to a journal, or in interacting with other scientists during the peer-review process. Authors can follow the basic rules suggested in this paper and have to pay attention to the ethics of scientific writing throughout the development of their manuscript. The final reward, the publication, will be the accompanied by the reader satisfaction, the number of citations and, if it is the case, the mentoring of a young colleague in writing research.

Neural activity in human ventromedial prefrontal cortex reflecting the intention to save reward.

Saving behavior usually requires individuals to perform several consecutive choices before collecting the final reward. The overt behavior is preceded by an intention to perform an appropriate choice sequence. We studied saving sequences for which each participant rated the intention numerically as willingness to save. Each sequence resulted in a specific reward amount and thus had a particular value for the participant, which we assessed with a Becker-DeGroot-Marschak auction-like mechanism. Using functional MRI, we found that blood-oxygen-level-dependent signals in human ventromedial prefrontal cortex (vmPFC) correlated with the participant’s stated intention before each choice sequence. An adjacent vmPFC region showed graded activation that reflected the value of the sequence. These results demonstrate an involvement of vmPFC in intentional processes preceding sequential economic choices.

Hierarchical Decision and Control for Continuous Multitarget Problem: Policy Evaluation With Action Delay.

This paper proposes a hierarchical decision-making and control algorithm for the shepherd game, the seventh mission in the International Aerial Robotics Competition (IARC). In this game, the agent (a multirotor aerial robot) is required to contact targets (ground vehicles) sequentially and drive them to a certain boundary to earn score. During the game of 10 min, the agent should be fully autonomous without any human interference. Regarding the lower-level controller and dynamics of the agent, each action takes a duration of time to accomplish. Denoted as an action delay, in this paper, this action duration is nonconstant and is related to the final reward. Therefore, the challenging point is making the agent "aware of time" when applying a certain action. We solve this problem by two approaches: deep Q-networks and lookup table. The action delay predictor in the decision-level is fitted by a lower-level controller. Through simulations by the example of the shepherd game, the effectiveness and efficiency of this approach are validated. This paper helps our team winning the first prize in IARC 2017, and keeps the best record of this mission since it was released in 2013.

An optimal strategy to solve the Prisoner\u2019s Dilemma

Cooperation is a central mechanism for evolution. It consists of an individual paying a cost in order to benefit another individual. However, natural selection describes individuals as being selfish and in competition among themselves. Therefore explaining the origin of cooperation within the context of natural selection is a problem that has been puzzling researchers for a long time. In the paradigmatic case of the Prisoner’s Dilemma (PD), several schemes for the evolution of cooperation have been proposed. Here we introduce an extension of the Replicator Equation (RE), called the Optimal Replicator Equation (ORE), motivated by the fact that evolution acts not only at the level of individuals of a population, but also among competing populations, and we show that this new model for natural selection directly leads to a simple and natural rule for the emergence of cooperation in the most basic version of the PD. Contrary to common belief, our results reveal that cooperation can emerge among selfish individuals because of selfishness itself: if the final reward for being part of a society is sufficiently appealing, players spontaneously decide to cooperate.

Phase transition in a sequential assignment problem on graphs

We study the following sequential assignment problem on a finite graph $G=(V,E)$. Each edge $e\in E$ starts with an integer value $n_{e}\ge0$, and we write $n=\sum_{e\in E}n_{e}$. At time $t$, $1\le t\le n$, a uniformly random vertex $v\in V$ is generated, and one of the edges $f$ incident with $v$ must be selected. The value of $f$ is then decreased by $1$. There is a unit final reward if the configuration $(0,\ldots,0)$ is reached. Our main result is that there is a phase transition: as $n\to\infty$, the expected reward under the optimal policy approaches a constant $c_{G}>0$ when $(n_{e}/n:e\in E)$ converges to a point in the interior of a certain convex set $\mathcal{R}_{G}$, and goes to $0$ exponentially when $(n_{e}/n:e\in E)$ is bounded away from $\mathcal{R}_{G}$. We also obtain estimates in the near-critical region, that is when $(n_{e}/n:e\in E)$ lies close to $\partial\mathcal{R}_{G}$. We supply quantitative error bounds in our arguments.

Introducing upfront losses as well as gains decreases impatience in intertemporal choices with rewards

AbstractPeople tend to prefer smaller and sooner (SS) rewards over larger and later (LL) ones even when the latter are much larger. Previous research have identified several ways to enhance people’s patience. Adding to this literature, the current paper demonstrates that introduction of upfront losses as well as gains to both SS and LL rewards can decrease people’s impatience. This effect is incompatible with both the normative exponential and descriptive hyperbolic discounting models, which agree on the additive assumption and the independence assumption. We also exculde the integration explanation which assumes subjects integrate upfront money with final rewards and make a decision with bottom line at the end. We consider several possible explanations, including the salience hypothesis, which states that introducing upfront money makes the money dimension more salient than not and thus increases the attractiveness of LL options.

A novel variable delay Go/No-Go task to study attention, motivation and working memory in the head-fixed rodent

In order to parse the causal elements underlying complex behaviors and decision-making processes, appropriate behavioral methods must be developed and used in concurrence with molecular, pharmacological, and electrophysiological approaches. Presented is a protocol for a novel Go/No-Go behavioral paradigm to study the brain attention and motivation/reward circuitry in awake, head-restrained rodents. This experimental setup allows: (1) Pharmacological and viral manipulation of various brain regions via targeted guide cannula; (2) Optogenetic cell-type specific activation and silencing with simultaneous electrophysiological recording and; (3) Repeated electrophysiological single and multiple unit recordings during ongoing behavior. The task consists of three components. The subject first makes an observing response by initiating a trial by lever pressing in response to distinctive Go or No-Go tones. Then, after a variable delay period, the subject is presented with a challenge period cued by white noise during which they must respond with a lever press for the Go condition or withhold from lever pressing for the duration of the cue in the No-Go condition. After correctly responding during the challenge period (Challenge) and a brief delay, a final reward tone of the same frequency as the initiation tone is presented and sucrose reward delivery is available and contingent upon lever pressing. Here, we provide a novel procedure and validating data set that allows researchers to study and manipulate components of behavior such as attention, motivation, impulsivity, and reward-related working memory during an ongoing operant behavioral task while limiting interference from non task-related behaviors.

Neural Correlates of Temporal Credit Assignment in the Parietal Lobe

Empirical studies of decision making have typically assumed that value learning is governed by time, such that a reward prediction error arising at a specific time triggers temporally-discounted learning for all preceding actions. However, in natural behavior, goals must be acquired through multiple actions, and each action can have different significance for the final outcome. As is recognized in computational research, carrying out multi-step actions requires the use of credit assignment mechanisms that focus learning on specific steps, but little is known about the neural correlates of these mechanisms. To investigate this question we recorded neurons in the monkey lateral intraparietal area (LIP) during a serial decision task where two consecutive eye movement decisions led to a final reward. The underlying decision trees were structured such that the two decisions had different relationships with the final reward, and the optimal strategy was to learn based on the final reward at one of the steps (the “F” step) but ignore changes in this reward at the remaining step (the “I” step). In two distinct contexts, the F step was either the first or the second in the sequence, controlling for effects of temporal discounting. We show that LIP neurons had the strongest value learning and strongest post-decision responses during the transition after the F step regardless of the serial position of this step. Thus, the neurons encode correlates of temporal credit assignment mechanisms that allocate learning to specific steps independently of temporal discounting.

Social Comparison Affects Outcome Evaluation in The Cooperative Task: An ERP Study

Social comparison is a mental process through which people come to know themselves by evaluating their own abilities, attitudes, outcomes and believes in comparison with others. Previous studies found that social comparison influenced the evaluative process of the outcome in the brain. However, in all those studies, social comparison was manipulated as the comparison between the rewards of two participants, so it remains unclear whether the monetary value or the comparison affects the outcome evaluation. By dissociating the monetary value from comparison, the current study aimed to investigate how non-monetary social comparison affects the process of outcome evaluation in a cooperative task. Eighteen healthy undergraduates(10 males, 8 females) took part in the EPR experiment. Participants cooperated with a confederate to complete a gambling game on two connected computers. Each of them chucked one dice sequentially and randomly. If the numbers on the two dices totaled greater than 6 in the trail, they would win 1 yuan in the trial; if not, they would lose 1 yuan. The final reward would be portioned out equally between them. After they chucked the two dices, a "" or "" between their name cueing comparison feedback indicates which one get a larger number. Then a feedback screen informed about whether they win or loss. However, unknown to the participants, the feedback was independent of their performance. Trails of each conditions were equal. Both social comparison feedback and monetary feedback were included in the final statistical analyses. For FRN analysis, we measured the average amplitude in the 250-350 ms time window for the social comparison feedback and the monetary feedback. P300 amplitude was quantified as the positive peak in the time window of 300-600 ms after feedback onset. ERP results revealed that FRN and P300 were both sensitive to non-monetary social comparison. When compared with those who had a better performance, participants who got a smaller number showed a larger FRN and a smaller P300. FRN amplitude of gain was more negative than that of loss. P300 showed an opposite pattern relative to FRN. Social comparison did not affect the process of outcome evaluation of the cooperative task. These findings suggest that the encoding of social comparison occurs at the early stage of the outcome evaluation. And FRN could code not only the prediction errors for monetary reward but also the information in the social context. Our results also indicate that FRN responds to the most notable information in the current context.

Psychological interventions in pervasive developmental disorder: An overview.

Pervasive developmental disorders (PDDs) are characterized by several impairments in the domains of social communication, social interaction and expression of social attachment, and other aspects of development like symbolic play. As the role of drugs in treating these impairments is extremely limited, a variety of psychological interventions have been developed to deal with them. Some of these have strong empirical support, while others are relatively new and hence controversial. Though it may prove to be a daunting task to begin with, the final reward of being able to improve the life of a child with PDD is enormous and hugely satisfying. Therefore, knowledge of these psychological interventions is important for a mental health professional, in order to be effective in the profession. Present paper presents an overview of these techniques in the management of PDD.

Desire beyond the Château d'If

Desire beyond the Château d'If

A novel variable delay Go/No-Go task to study attention, motivation and working memory in the head-fixed rodent.

In order to parse the causal elements underlying complex behaviors and decision-making processes, appropriate behavioral methods must be developed and used in concurrence with molecular, pharmacological, and electrophysiological approaches. Presented is a protocol for a novel Go/No-Go behavioral paradigm to study the brain attention and motivation/reward circuitry in awake, head-restrained rodents. This experimental setup allows: (1) Pharmacological and viral manipulation of various brain regions via targeted guide cannula; (2) Optogenetic cell-type specific activation and silencing with simultaneous electrophysiological recording and; (3) Repeated electrophysiological single and multiple unit recordings during ongoing behavior. The task consists of three components. The subject first makes an observing response by initiating a trial by lever pressing in response to distinctive Go or No-Go tones. Then, after a variable delay period, the subject is presented with a challenge period cued by white noise during which they must respond with a lever press for the Go condition or withhold from lever pressing for the duration of the cue in the No-Go condition. After correctly responding during the challenge period (Challenge) and a brief delay, a final reward tone of the same frequency as the initiation tone is presented and sucrose reward delivery is available and contingent upon lever pressing. Here, we provide a novel procedure and validating data set that allows researchers to study and manipulate components of behavior such as attention, motivation, impulsivity, and reward-related working memory during an ongoing operant behavioral task while limiting interference from non task-related behaviors.

Other-Regarding or Self-Fulfilling: An Experiment on Working after Disappointment

It is a common sense that an individual would feel disappointed when his work results overpass other team members while his final reward is determined by the poorest performance of the group. One question is raised: How people decide the level of their working effort in the next period while having experienced disappointing feeling in the previous one? In this study, we try to test two hypotheses of the potential impacts of disappoint feeling: disappointment aversion and self-achievement. To do so, we conducted a laboratory experiment by applying a classical repeated minimum effort game under a real effort task at India. We use two different measurements of disappointment to estimate the effort supply of this individual in the next period. The study finds the existence of both the disappointment aversion and self-fulfillment seeking motivation for an individual under different scenarios. The effort supply does depend negatively on the disappoint feeling that has been shortly experienced, proving the motivation of disappoint aversion. Nevertheless, the subject’s motivation of self-achievement will dominate the behavior after having experienced continuously worsening disappointed feeling and therefore, one’s effort supply in the next period does not doomed to rely any more on the performance of one’s team members.

Mirrored prominent deck B phenomenon: frequent small losses override infrequent large gains in the inverted Iowa Gambling Task.

Since Bechara et al. pioneered its development, the Iowa Gambling Task (IGT) has been widely applied to elucidate decision behavior and medial prefrontal function. Although most decision makers can hunch the final benefits of IGT, ventromedial prefrontal lesions generate a myopic choice pattern. Additionally, the Iowa group developed a revised IGT (inverted IGT, iIGT) to confirm the IGT validity. Each iIGT trial was generated from the trial of IGT by multiplying by a “−” to create an inverted monetary value. Thus, bad decks A and B in the IGT become good decks iA and iB in the iIGT; additionally, good decks C and D in the IGT become bad decks iC and iD in the iIGT. Furthermore, IGT possessed mostly the gain trials, and iIGT possessed mainly the loss trials. Therefore, IGT is a frequent-gain–based task, and iIGT is a frequent-loss–based task. However, a growing number of IGT-related studies have identified confounding factors in IGT (i.e., gain-loss frequency), which are demonstrated by the prominent deck B phenomenon (PDB phenomenon). Nevertheless, the mirrored PDB phenomenon and guiding power of gain-loss frequency in iIGT have seldom been reexamined. This experimental finding supports the prediction based on gain-loss frequency. This study identifies the mirrored PDB phenomenon. Frequent small losses override occasional large gains in deck iB of the iIGT. Learning curve analysis generally supports the phenomenon based on gain-loss frequency rather than final outcome. In terms of iIGT and simple versions of iIGT, results of this study demonstrate that high-frequency loss, rather than a satisfactory final outcome, dominates the preference of normal decision makers under uncertainty. Furthermore, normal subjects prefer “no immediate punishment” rather than “final reward” under uncertainty.

A survey of biological collaboration models

Collaboration has been widely studied throughout multiple disciplines, and it has been applied to a broad range of fields. In collaboration games, players’ strategies are the main influence on the final rewards or payoffs. In the field of biology, the phenomena of collaboration in coalition formation and in foraging have encouraged researchers to make great efforts in studying them. Quantitative models have been developed to include the biological complexities and behaviors of natural creatures and also to identify and predict the optimal strategies for them. Also, the models are applied in other fields, such as those (e.g., wireless sensor and robot networks) in the engineering realm. The purpose of this paper is to introduce and discuss those models, to gain inspiration from the modeling methods and analysis methods, to introduce the applications of the models in engineering, and to propose research directions in collaboration with presenting problems as examples that may be potentially solved with bio-inspired solutions. This paper surveys the quantitative models, including the assumptions in the models, modeling methods, and analysis methods as well as other studies that focus on the mechanism of the evolution of cooperation in nature systems.