Social Reinforcement Learning Research Articles

Social aspiration reinforcement learning in Cournot games

Social aspiration reinforcement learning in Cournot games

Understanding the Role of Naive Learners in Cultural Change.

AbstractA change to a population's social network is a change to the substrate of cultural transmission, affecting behavioral diversity and adaptive cultural evolution. While features of network structure such as population size and density have been well studied, less is understood about the influence of social processes such as population turnover-or the repeated replacement of individuals by naive individuals. Experimental data have led to the hypothesis that naive learners can drive cultural evolution by better assessing the relative value of behaviors, although this hypothesis has been expressed only verbally. We conducted a formal exploration of this hypothesis using a generative model that concurrently simulated its two key ingredients: social transmission and reinforcement learning. We simulated competition between high- and low-reward behaviors while varying turnover magnitude and tempo. Variation in turnover influenced changes in the distributions of cultural behaviors, irrespective of initial knowledge-state conditions. We found optimal turnover regimes that amplified the production of higher reward behaviors through two key mechanisms: repertoire composition and enhanced valuation by agents that knew both behaviors. These effects depended on network and learning parameters. Our model provides formal theoretical support for, and predictions about, the hypothesis that naive learners can shape cultural change through their enhanced sampling ability. By moving from experimental data to theory, we illuminate an underdiscussed generative process that can lead to changes in cultural behavior, arising from an interaction between social dynamics and learning.

Associations between sleep health, negative reinforcement learning, and alcohol use among South Florida college students with elevated internalizing symptoms

Negative reinforcement is proposed to mediate associations between sleep and alcohol use, especially among people with depression and/or anxiety symptoms. Worse sleep (e.g., shorter duration, less efficiency, more irregular timing) exacerbates negative emotions, which alcohol may temporarily relieve. Not yet examined, we propose sleep indirectly impacts early stages of alcohol use via differences in negative reinforcement learning (NRL), since sleep impacts emotion, reward response, and learning. The current study aimed to replicate associations between sleep and alcohol use, test associations with NRL, and examine indirect associations between sleep health and alcohol use via NRL among 60 underage college students (ages 18-20 years, 77% female) varying in depression and anxiety symptoms. Participants wore Fitbit smartwatches and completed daily diaries measuring sleep and substance use for ∼14 days before completing two computer tasks assessing social (SNRL) and monetary (MNRL) negative reinforcement learning. Robust generalized linear models tested direct associations within the proposed model. SNRL performance was positively associated with alcohol use, but no other associations were observed. Statistical mediation models failed to indicate indirect effects of sleep on alcohol use via SNRL or MNRL performance. Post-hoc exploratory models examining depression and anxiety symptoms as moderators of direct associations indicated several interactions. Positive associations between sleep timing variability and alcohol use were weakened at higher anxiety symptom severity and stronger at higher depression symptom severity. The positive association between SNRL performance and alcohol use was also stronger at higher depression symptom severity. Among students with elevated depression symptoms, variable sleep timing and stronger SNRL performance were independently associated with more alcohol use, but indirect effects were not supported. Future research should replicate findings, confirm causality of interactions, and examine sleep timing and behavioral responses to negative social stimuli as targets for improving alcohol-related outcomes among underage college students with elevated depressive symptoms.

Differential deficits in social versus monetary reinforcement learning in schizophrenia: Associations with facial emotion recognition.

Despite evidence that individuals with schizophrenia (SZ) have an intact desire for social relationships, they have small social networks and report high levels of loneliness. Difficulty with reinforcement learning (RL), the ability to update behavior based on feedback, may inhibit the formation and maintenance of social relationships in SZ. However, impaired RL in SZ has largely been demonstrated via monetary tasks. Thus, it remains unclear whether SZ are similarly impaired in social and monetary RL, or whether social-specific factors may further inhibit their ability to learn from social feedback. Thirty-one individuals with SZ and 31 healthy controls (HCs) participated in a RL paradigm to test hypotheses about social versus monetary RL. SZ exhibited impaired RL compared to HCs in both social and monetary tasks. Further, a Group × Task interaction demonstrated that SZ was more impaired when learning from social than monetary reinforcement, F(1, 59) = 5.99, p = .017. This differential deficit to social RL was not accounted for by reported pleasure from social feedback, which did not differ between groups. Instead, SZ had poorer emotion recognition than HCs, t(1, 60) = 4.80, p < .001, particularly for negative emotions, and controlling for this eliminated the differential social RL impairment. These results suggest the possibility that difficulty recognizing social cues, especially those indicating negative feedback, may relate to a reduced ability to learn from others' feedback. Thus, future research could elucidate whether targeting these emotion recognition difficulties in treatment could serve as a potential mechanism for improving social functioning in SZ. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Modulation of Trust in Borderline Personality Disorder by Script-Based Imaginal Exposure to Betrayal.

Interpersonal and trust-related difficulties are central features of borderline personality disorder (BPD). In this study, we applied script-driven betrayal imagery to evoke mistrustful behavior in a social reinforcement learning task. In 21 BPD and 20 healthy control (HC) participants, we compared this approach to the standard confederate paradigm used in research studies. The script-driven imagery evoked a transient increase in negative affect and also decreased trusting behavior to a similar degree in both groups. Across conditions, we also replicated previously reported between-group differences in negative affect (increased in BPD) and task behavior (more sensitive to social cues in BPD). These results support the validity of script-driven imagery as an alternative social task stimulus. This script-driven imagery approach is appealing for clinical research studies on reinforcement learning because it eliminates deception, scales easily, and evokes disorder-specific states of social difficulty.

Changes in Learning From Social Feedback After Web-Based Interpretation Bias Modification: Secondary Analysis of a Digital Mental Health Intervention Among Individuals With High Social Anxiety Symptoms.

Biases in social reinforcement learning, or the process of learning to predict and optimize behavior based on rewards and punishments in the social environment, may underlie and maintain some negative cognitive biases that are characteristic of social anxiety. However, little is known about how cognitive and behavioral interventions may change social reinforcement learning in individuals who are anxious. This study assessed whether a scalable, web-based cognitive bias modification for interpretations (CBM-I) intervention changed social reinforcement learning biases in participants with high social anxiety symptoms. This study focused on 2 types of social reinforcement learning relevant to social anxiety: learning about other people and learning about one's own social performance. Participants (N=106) completed 2 laboratory sessions, separated by 5 weeks of ecological momentary assessment tracking emotion regulation strategy use and affect. Approximately half (n=51, 48.1%) of the participants completed up to 6 brief daily sessions of CBM-I in week 3. Participants completed a task that assessed social reinforcement learning about other people in both laboratory sessions and a task that assessed social reinforcement learning about one's own social performance in the second session. Behavioral data from these tasks were computationally modeled using Q-learning and analyzed using mixed effects models. After the CBM-I intervention, participants updated their beliefs about others more slowly (P=.04; Cohen d=-0.29) but used what they learned to make more accurate decisions (P=.005; Cohen d=0.20), choosing rewarding faces more frequently. These effects were not observed among participants who did not complete the CBM-I intervention. Participants who completed the CBM-I intervention also showed less-biased updating about their social performance than participants who did not complete the CBM-I intervention, learning similarly from positive and negative feedback and from feedback on items related to poor versus good social performance. Regardless of the intervention condition, participants at session 2 versus session 1 updated their expectancies about others more from rewarding (P=.003; Cohen d=0.43) and less from punishing outcomes (P=.001; Cohen d=-0.47), and they became more accurate at learning to avoid punishing faces (P=.001; Cohen d=0.20). Taken together, our results provide initial evidence that there may be some beneficial effects of both the CBM-I intervention and self-tracking of emotion regulation on social reinforcement learning in individuals who are socially anxious, although replication will be important.

Examining social reinforcement learning in social anxiety

Background and objectivesReinforcement learning biases have been empirically linked to anhedonia in depression and theoretically linked to social anhedonia in social anxiety disorder, but little work has directly assessed how socially anxious individuals learn from social reward and punishment. MethodsN = 157 individuals high and low in social anxiety symptoms completed a social probabilistic selection task that involved selecting between pairs of neutral faces with varying probabilities of changing to a happy or angry face. Computational modeling was performed to estimate learning rates. Accuracy in choosing the more rewarding face was also analyzed. ResultsNo significant group differences were found for learning rates. Contrary to hypotheses, participants high in social anxiety showed impaired punishment learning accuracy; they were more accurate at choosing the most rewarding face than they were at avoiding the most punishing face, and their punishment learning accuracy was lower than that of participants low in social anxiety. Secondary analyses found that high (vs. low) social anxiety participants were less accurate at selecting the more rewarding face on more (vs. less) punishing face pairs. LimitationsStimuli were static, White, facial images, which lack important social cues (e.g., movement, sound) and diversity, and participants were largely non-Hispanic, White undergraduates, whose social reinforcement learning may differ from individuals at different developmental stages and those holding more marginalized identities. ConclusionsSocially anxious individuals may be less accurate at learning to avoid social punishment, which may maintain negative beliefs through an interpersonal stress generation process.

Memory-based crowd-aware robot navigation using deep reinforcement learning

The evolution of learning techniques has led robotics to have a considerable influence in industrial and household applications. With the progress in technology revolution, the demand for service robots is rapidly growing and extends to many applications. However, efficient navigation of service robots in crowded environments, with unpredictable human behaviors, is still challenging. The robot is supposed to recognize surrounding information while navigating, and then act accordingly. To address this issue, the proposed method crowd Aware Memory-based Reinforcement Learning (CAM-RL) uses gated recurrent units to store the relative dependencies among the crowd, and utilizes the human–robot interactions in the reinforcement learning framework for collision-free navigation. The proposed method is compared with the state-of-the-art techniques of multi-agent navigation, such as Collision Avoidance with Deep Reinforcement Learning (CADRL), Long Short-Term Memory Reinforcement Learning (LSTM-RL) and Social Attention Reinforcement Learning (SARL). Experimental results show that the proposed method can identify and learn human–robot interactions more extensively and efficiently than above-mentioned methods while navigating in a crowded environment. The proposed method achieved a success rate of greater than or equal to 99% and a collision rate of less than or equal to 1% in all test case scenarios, which is better compared to the previously proposed methods.

Neurobiological correlates of the social and emotional impact of peer victimization: A review.

Peer victimization is very common during late childhood and adolescence. Despite the relatively reduced number of studies, the neurobiological underpinnings of the negative impact of peer victimization experiences have received increasing attention in recent years. The present selective review summarizes the most recent available evidence and provides a general overview of the impact of peer victimization experiences on social processing and decision-making at the neurobiological level, highlighting the most pressing areas requiring further research. Three key cognitive areas show a clear negative impact of peer victimization and bullying experiences: social valuation processing, reward and reinforcement learning and self-regulation processes. Victims show enhanced activation in key regions of the limbic system including the amygdala, rostral and dorsal anterior cingulate cortices, suggestive of enhanced sensitivity to social stimuli. They also show enhanced recruitment of lateral prefrontal regions crucially involved in cognitive and emotional regulation processes, and abnormal reward-related striatal function. The presence of psychopathology is a complex factor, increased as a consequence of peer victimization, but that also constitutes vulnerability to such experiences.

Modeling Variation in Empathic Sensitivity Using Go/No-Go Social Reinforcement Learning.

The online version contains supplementary material available at 10.1007/s42761-022-00119-4.

Ventral subicular lesion impairs pro-social empathy-like behavior in adult Wistar rats

The subiculum, an important structure of the hippocampal formation, regulates spatial information processing, social cognition, and affective behavior. Earlier we demonstrated deficits in sociability and social novelty as a measure of social cognition in ventral subicular lesioned (VSL) rats. The present study investigated empathy-like pro-social behavior and the associated affective states in VSL rats. The ability of free rats to release trapped cagemates was assessed using a modified door-opening empathy task.The rat pairs (free rat and the trapped cagemate) used were from the same group and tested for eight days to assess the pro-social behavior displayed by the free rats. The controlfree rats learned to open the door quickly to release the trapped cagemate and both the rats displayed social responses by emitting ‘hedonic’ calls (50-kHz ultrasonic vocalizations) while playing after the release. The VSLfree rats, however, were less exploratory, displayed apathy towards the trapped cagemate, demonstrated freezing behavior following door-opening and did not interact with the cagemate even after its release. These findings indicate deficits of social motivation and reinforcement learning associated with lesions in possibly both the rats. In addition, the VSL rat pairs elicited more 22-kHz ‘alarm’ calls and fewer 50-kHz ‘hedonic’ calls highlighting the lesion-induced alterations of contextual processing and threat perception abilities. In conclusion, VSL led to significant pro-social deficits implicating the role of ventral subiculum in social cognition and empathy. More studies are needed to substantiate whether the subiculum is implicated in social deficits associated with psychiatric conditions such as autism spectrum disorder.

Method for solving social dilemmas based on multi-agent learning and reputation

Many real-world problems contain social dilemmas that express contradictions between individual and public interests. Such tasks can be solved using deep multi-agent learning. This article focuses on reputation-based social dilemma solving and multi-agent reinforcement learning. Target of the work is the development of a method for solving social dilemmas based on deep multi-agent reinforcement learning. A method for solving social dilemmas based on deep multi-agent reinforcement learning is proposed. The difference between the method and the known analogs is based on the use of the agent's reputation as a part of the reward function. Reputation, in turn, depends on the degree of cooperation of an agent with other agents. The software implementation of the method is done in Python using the open library Pytorch. The effectiveness of the method was assessed on the basis of comparison with methods for solving social dilemmas without taking into account cooperation between agents. The results of the work can be used to create software for unmanned vehicles of the city's transport system. A balance between collective and individual interests by solving social dilemmas in a “smart city” can be achieved through the application of the method proposed in the article.

VTA dopamine neuron activity encodes social interaction and promotes reinforcement learning through social prediction error.

Social interactions are motivated behaviors that in many species facilitate learning. However, how the brain encodes the reinforcing properties of social interactions remains elusive. Here, using in vivo recording in freely moving mice, we show that dopamine (DA) neurons of the ventral tegmental area (VTA) increase their activity during interactions with an unfamiliar conspecific and display heterogeneous responses. Using a social instrumental task (SIT), we then show that VTA DA neuron activity encodes social prediction error and drives social reinforcement learning. Thus, our findings suggest that VTA DA neurons are a neural substrate for a social learning signal that drives motivated behavior.

You Were Always on My Mind: Introducing Chef's Hat and COPPER for Personalized Reinforcement Learning.

Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study proposes the Chef’s Hat simulation environment, which implements a multi-agent competitive card game that is a complete reproduction of the homonymous board game, designed to provoke competitive strategies in humans and emotional responses. The game was shown to be ideal for developing personalized reinforcement learning, in an online learning closed-loop scenario, as its state representation is extremely dynamic and directly related to each of the opponent’s actions. To adapt current reinforcement learning agents to this scenario, we also developed the COmPetitive Prioritized Experience Replay (COPPER) algorithm. With the help of COPPER and the Chef’s Hat simulation environment, we evaluated the following: (1) 12 experimental learning agents, trained via four different regimens (self-play, play against a naive baseline, PER, or COPPER) with three algorithms based on different state-of-the-art learning paradigms (PPO, DQN, and ACER), and two “dummy” baseline agents that take random actions, (2) the performance difference between COPPER and PER agents trained using the PPO algorithm and playing against different agents (PPO, DQN, and ACER) or all DQN agents, and (3) human performance when playing against two different collections of agents. Our experiments demonstrate that COPPER helps agents learn to adapt to different types of opponents, improving the performance when compared to off-line learning models. An additional contribution of the study is the formalization of the Chef’s Hat competitive game and the implementation of the Chef’s Hat Player Club, a collection of trained and assessed agents as an enabler for embedding human competitive strategies in social continual and competitive reinforcement learning.

The actions of others act as a pseudo-reward to drive imitation in the context of social reinforcement learning.

While there is no doubt that social signals affect human reinforcement learning, there is still no consensus about how this process is computationally implemented. To address this issue, we compared three psychologically plausible hypotheses about the algorithmic implementation of imitation in reinforcement learning. The first hypothesis, decision biasing (DB), postulates that imitation consists in transiently biasing the learner's action selection without affecting their value function. According to the second hypothesis, model-based imitation (MB), the learner infers the demonstrator's value function through inverse reinforcement learning and uses it to bias action selection. Finally, according to the third hypothesis, value shaping (VS), the demonstrator's actions directly affect the learner's value function. We tested these three hypotheses in 2 experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task. We show through model comparison and model simulation that VS provides the best explanation of learner's behavior. Results replicated in a third independent experiment featuring a larger cohort and a different design (N = 302). In our experiments, we also manipulated the quality of the demonstrators' choices and found that learners were able to adapt their imitation rate, so that only skilled demonstrators were imitated. We proposed and tested an efficient meta-learning process to account for this effect, where imitation is regulated by the agreement between the learner and the demonstrator. In sum, our findings provide new insights and perspectives on the computational mechanisms underlying adaptive imitation in human reinforcement learning.

Testing the reinforcement learning hypothesis of social conformity.

Our preferences are influenced by the opinions of others. The past human neuroimaging studies on social conformity have identified a network of brain regions related to social conformity that includes the posterior medial frontal cortex (pMFC), anterior insula, and striatum. Since these brain regions are also known to play important roles in reinforcement learning (i.e., processing prediction error), it was previously hypothesized that social conformity and reinforcement learning have a common neural mechanism. However, although this view is currently widely accepted, these two processes have never been directly compared; therefore, the extent to which they shared a common neural mechanism had remained unclear. This study aimed to formally test the hypothesis. The same group of participants (n = 25) performed social conformity and reinforcement learning tasks inside a functional magnetic resonance imaging (fMRI) scanner. Univariate fMRI data analyses revealed activation overlaps in the pMFC and bilateral insula between social conflict and unsigned prediction error and in the striatum between social conflict and signed prediction error. We further conducted multivoxel pattern analysis (MVPA) for more direct evidence of a shared neural mechanism. MVPA did not reveal any evidence to support the hypothesis in any of these regions but found that activation patterns between social conflict and prediction error in these regions were largely distinct. Taken together, the present study provides no clear evidence of a common neural mechanism between social conformity and reinforcement learning.

Social reinforcement learning as a predictor of real-life experiences in individuals with high and low depressive symptomatology.

Several studies have reported diminished learning from non-social outcomes in depressed individuals. However, it is not clear how depression impacts learning from social feedback. Notably, mood disorders are commonly associated with deficits in social functioning, which raises the possibility that potential impairments in social learning may negatively affect real-life social experiences in depressed subjects. Ninety-two participants with high (HD; N = 40) and low (LD; N = 52) depression scores were recruited. Subjects performed a learning task, during which they received monetary outcomes or social feedback which they were told came from other people. Additionally, participants answered questions about their everyday social experiences. Computational models were fit to the data and model parameters were related to social experience measures. HD subjects reported a reduced quality and quantity of social experiences compared to LD controls, including an increase in the amount of time spent in negative social situations. Moreover, HD participants showed lower learning rates than LD subjects in the social condition of the task. Interestingly, across all participants, reduced social learning rates predicted higher amounts of time spent in negative social situations, even when depression scores were controlled for. These findings indicate that deficits in social learning may affect the quality of everyday social experiences. Specifically, the impaired ability to use social feedback to appropriately update future actions, which was observed in HD subjects, may lead to suboptimal interpersonal behavior in real life. This, in turn, may evoke negative feedback from others, thus bringing about more unpleasant social encounters.

Social anxiety and dynamic social reinforcement learning in a volatile environment.

Adaptive social behavior requires learning probabilities of social reward and punishment, and updating these probabilities when they change. Given prior research on aberrant reinforcement learning in affective disorders, this study examines how social anxiety affects probabilistic social reinforcement learning and dynamic updating of learned probabilities in a volatile environment. N=222 online participants completed questionnaires and a computerized ball-catching game with changing probabilities of reward and punishment. Dynamic learning rates were estimated to assess the relative importance ascribed to new information in response to volatility. Mixed-effects regression was used to analyze throw patterns as a function of social anxiety symptoms. Higher social anxiety predicted fewer throws to the previously punishing avatar and different learning rates after certain role changes, suggesting that social anxiety may be characterized by difficulty updating learned social probabilities. Socially anxious individuals may miss the chance to learn that a once-punishing situation no longer poses a threat.

Social Value Unblocks Pavlovian Reinforcement Learning in Male Rats

Many species, including humans, are sensitive to social signals and the valuation of these social cues is important in maintaining social interactions. Social value, derived from social cues, thus could act as a reinforcer and influence reward learning. Here, we introduce a novel task in rats that investigates if social value can drive reinforcement learning about novel stimuli. Using the blocking/unblocking paradigm originally developed in the (non-social) reinforcement learning literature, we found that when actor rats have fully learned a stimulus-reward association producing reward for themselves, adding a cue that predicted an additional reward delivery to a partner rat unblocked associative learning about this cue in the actor rats. In contrast, additional cues that did not predict additional reward remained blocked from acquiring associative value. In a control experiment where putative social cue exchange between the partnered rats was prevented, the normally unblocked cues now remained blocked as expected. Taken together, these results suggest that social value can drive reinforcement learning in rats, and that the transmission of social cues is necessary for this learning, driven by social value, to occur.

Neural modulation of social reinforcement learning by intranasal oxytocin in male adults with high-functioning autism spectrum disorder: a randomized trial.

Reduced social motivation is a hallmark of individuals with autism spectrum disorders (ASDs). Although the exact neural mechanisms are unclear, oxytocin has been shown to enhance motivation and attention to social stimuli, suggesting a potential to augment social reinforcement learning as the central mechanism of behavioral interventions in ASD. We tested how reinforcement learning in social contexts and associated reward prediction error (RPE) signals in the nucleus accumbens (NAcc) were modulated by intranasal oxytocin. Male adults with a childhood diagnosis of ASD (n = 15) and healthy controls (n = 24; aged 18-26 years) performed a probabilistic reinforcement learning task during functional magnetic resonance imaging in a single-center (research center in Germany), randomized double-blind, placebo-controlled cross-over trial. The interventions were intranasal oxytocin (Syntocinon®, Novartis; 10 puffs = 20 international units (IUs) per treatment) and placebo spray. Using computational modeling of behavioral data, trial-by-trial RPE signals were assessed and related to brain activation in NAcc during reinforcing feedback in social and non-social contexts. The order of oxytocin/placebo was randomized for 60 participants. Twenty-one participants were excluded from analyses, leaving 39 for the final analysis. Behaviorally, individuals with ASD showed enhanced learning under oxytocin when the learning target as well as feedback was social as compared to non-social (social vs. non-social target: 87.09% vs. 71.29%, 95% confidence interval (CI): 7.28-24.33, p = .003; social vs. non-social feedback: 81.00% vs. 71.29%, 95% CI: 2.81-16.61, p = .027). Correspondingly, oxytocin enhanced the correlation of the RPE signal with NAcc activation during social (vs. non-social) feedback in ASD (3.48 vs. -1.12, respectively, 95% CI: 2.98-6.22, p = .000), whereas in controls, this effect was found in the placebo condition (2.90 vs. -1.14, respectively, 95% CI: 1.07-7.01, p = .010). In ASD, a similar pattern emerged when the learning target was social (3.00 vs. -0.64, respectively, 95% CI: -0.13 to 7.41, p = .057), whereas controls showed a reduced correlation for social learning targets under oxytocin (-0.70 vs. 2.72, respectively, 95% CI: -5.86 to 0.98, p = .008). The current data suggest that intranasal oxytocin has the potential to enhance social reinforcement learning in ASD. Future studies are warranted that investigate whether oxytocin can potentiate social learning when combined with behavioral therapies, resulting in greater treatment benefits than traditional behavior-only approaches.