Editor's evaluation: Decoding subjective emotional arousal from EEG during an immersive virtual reality experience

Abstract

Article Figures and data Abstract Editor's evaluation eLife digest Introduction Materials and methods Results Discussion Appendix 1 Data availability References Decision letter Author response Article and author information Metrics Abstract Immersive virtual reality (VR) enables naturalistic neuroscientific studies while maintaining experimental control, but dynamic and interactive stimuli pose methodological challenges. We here probed the link between emotional arousal, a fundamental property of affective experience, and parieto-occipital alpha power under naturalistic stimulation: 37 young healthy adults completed an immersive VR experience, which included rollercoaster rides, while their EEG was recorded. They then continuously rated their subjective emotional arousal while viewing a replay of their experience. The association between emotional arousal and parieto-occipital alpha power was tested and confirmed by (1) decomposing the continuous EEG signal while maximizing the comodulation between alpha power and arousal ratings and by (2) decoding periods of high and low arousal with discriminative common spatial patterns and a long short-term memory recurrent neural network. We successfully combine EEG and a naturalistic immersive VR experience to extend previous findings on the neurophysiology of emotional arousal towards real-world neuroscience. Editor's evaluation Hofmann et al., investigate the link between two phenomena, emotional arousal and cortical α activity. Although α activity is tightly linked to the first reports of electric activity in the brain nearly 100 years ago, a comprehensive characterization of this phenomenon is elusive. One of the reasons is that EEG, the major method to investigate electric activity in the human brain, is susceptible to motion artifacts and, thus, mostly used in laboratory settings. Here, the authors combine EEG with virtual reality (VR) to give experimental participants a roller coaster ride with high immersion. The ride, literally, leads to large ups and downs in emotional arousal, which is quantified by the subjects during a later rerun. Several different decoding methods were evaluated, and each showed above-chance levels of performance, substantiating a link between lower levels of parietal/occipital α and subjective arousal in a quasi-naturalistic setting. https://doi.org/10.7554/eLife.64812.sa0 Decision letter eLife's review process eLife digest Human emotions are complex and difficult to study. It is particularly difficult to study emotional arousal, this is, how engaging, motivating, or intense an emotional experience is. To learn how the human brain processes emotions, researchers usually show emotional images to participants in the laboratory while recording their brain activity. But viewing sequences of photos is not quite like experiencing the dynamic and interactive emotions people face in everyday life. New technologies, such as immersive virtual reality, allow individuals to experience dynamic and interactive situations, giving scientists the opportunity to study human emotions in more realistic settings. These tools could lead to new insights regarding emotions and emotional arousal. Hofmann, Klotzsche, Mariola et al. show that virtual reality can be a useful tool for studying emotions and emotional arousal. In the experiment, 37 healthy young adults put on virtual reality glasses and ‘experienced’ two virtual rollercoaster rides. During the virtual rides, Hofmann, Klotzsche, Mariola et al. measured the participants' brain activity using a technique called electroencephalography (EEG). Then, the participants rewatched their rides and rated how emotionally arousing each moment was. Three different computer modeling techniques were then used to predict the participant’s emotional arousal based on their brain activity. The experiments confirmed the results of traditional laboratory experiments and showed that the brain’s alpha waves can be used to predict emotional arousal. This suggests that immersive virtual reality is a useful tool for studying human emotions in circumstances that are more like everyday life. This may make future discoveries about human emotions more useful for real-life applications such as mental health care. Introduction While humans almost constantly interact with complex, dynamic environments, lab-based studies typically use simplified stimuli in passive experimental situations. Trading realism for experimental control happens at the expense of the representativity of the experimental design (Brunswik, 1955), that is, the degree to which effects found in the lab generalize to practical everyday-life conditions. This may be particularly true for affective phenomena like emotions. Emotional arousal as a fundamental property of affective experience Emotions are subjective, physiological, and behavioural responses to personally meaningful external stimuli (Mauss and Robinson, 2009) or self-generated mental states (e.g., memories; Damasio et al., 2000) and underlie our experience of the world (James, 1884; James, 1890; Seth, 2013). Emotions are crucial for physical and mental health (Gross and Muñoz, 1995) and their investigation has long been at the core of experimental psychology (Wundt and Judd, 1897). Dimensional accounts conceptualize emotions along the two axes of valence and arousal (Duffy, 1957; Kuppens et al., 2013; Russell, 1980; Russell and Barrett, 1999; Wundt and Judd, 1897): valence differentiates states of pleasure and displeasure, while emotional arousal describes the degree of activation or intensity that accompanies an emotional state. [Different types of arousal have been proposed and investigated, such as sexual, autonomic, emotional (Russell, 1980); also in the context of altered states of consciousness, for example, through anaesthesia or sleep. They may share psychological (e.g., increase in sensorimotor and emotional reactivity; Pfaff et al., 2012) and physiological aspects (e.g., sympathetic activation) but are not synonymous. We here explicitly refer to arousal in the context of the subjective experience of emotions]. Emotions have been linked to activity in the autonomic (ANS) and the central nervous system (Dalgleish, 2004). It has thereby been difficult to consistently associate individual, discrete emotion categories with specific response patterns in the ANS (Kragel and Labar, 2013; Kreibig, 2010; Siegel et al., 2018) or in distinct brain regions (Lindquist et al., 2012; but Saarimäki et al., 2016). Rather, emotions seem to be dynamically implemented by sets of brain regions and bodily activations that are involved in basic, also non-emotional psychological operations (i.e., ‘psychological primitives’; Lindquist et al., 2012). In this view, humans are typically in fluctuating states of pleasant or unpleasant arousal (‘core affect’; Russell and Barrett, 1999; Lindquist, 2013), which can be influenced by external stimuli. Emotional arousal could thereby be a ‘common currency’ to compare different stimuli or events (Lindquist, 2013) and represent fundamental neural processes that underlie a variety of emotions (Wilson-Mendenhall et al., 2013). It can fluctuate quickly – on the order of minutes (Kuppens et al., 2010) or seconds (Mikutta et al., 2012) – and has been connected to ANS activity, as measured by pupil diameter (Bradley et al., 2008) or skin conductance (Bach et al., 2010). At the brain level, emotional arousal was linked to lower alpha power, particularly over parietal electrodes (Luft and Bhattacharya, 2015; Koelstra et al., 2012). The parieto-occipital alpha rhythm, typically oscillating in the frequency range of 8–13 Hz, is the dominant EEG rhythm in awake adults with eyes closed (Berger, 1929), where it varies with vigilance (Olbrich et al., 2009). However, in tasks of visual processing (i.e., with eyes open), parieto-occipital alpha power was linked to active attentional processes (e.g., distractor suppression; Kelly et al., 2006; Klimesch, 2012) or, more generally, to functional inhibition for information gating (Jensen and Mazaheri, 2010). Physiologically, alpha oscillations were associated with large-scale synchronization of neuronal activity (Buzsáki, 2006) and metabolic deactivation (Moosmann et al., 2003). In sum, bodily responses interact in complex ways across situations, and activity in the brain is central for emotions and their subjective component (Barrett, 2017; Seth, 2013). As arousal is a fundamental property not only of emotions but of subjective experience in general (Adolphs et al., 2019), an investigation of its neurophysiology, reflected in neural oscillations, is essential to understanding the biology of the mind. Studying emotional arousal and its neurophysiology in the lab Studies that investigated emotions or emotional arousal in laboratory environments typically used static images. For example, more emotionally arousing relative to less emotionally arousing (e.g., neutral) pictures were associated with an event-related desynchronization, that is, a decrease in the power of alpha oscillations in posterior channels (De Cesarei and Codispoti, 2011; Schubring and Schupp, 2019; but Uusberg et al., 2013). In a study, in which emotional arousal was induced through pictures and music, blocks of higher emotional arousal were associated with decreased alpha power compared to blocks of lower emotional arousal (Luft and Bhattacharya, 2015). However, emotion-eliciting content that is repeatedly presented in trials creates an artificial experience for participants (Bridwell et al., 2018); it hardly resembles natural human behaviour and its (neuro-)physiology, which unfolds over multiple continuous timescales (Huk et al., 2018). Moreover, such presentations lack a sense of emotional continuity. External events often do not appear suddenly but are embedded in an enduring sequence, in which emotions build up and dissipate. Real-life scenarios also include anticipatory aspects where emotional components can be amplified or even suppressed, thus rendering the relationship between the corresponding neuronal events and subjective experience more complex than the one typically studied with randomized or partitioned presentations of visual or auditory stimuli. Virtual reality (VR) technology – particularly immersive VR, in which the user is completely surrounded by the virtual environment – affords the creation and presentation of computer-generated scenarios that are contextually rich and engaging (Diemer et al., 2015). As more naturalistic (i.e., dynamic, interactive, and less decontextualized) experiments allow to study the brain under conditions it was optimized for (Gibson, 1978; Hasson et al., 2020), their findings may more readily generalize to real-world circumstances and provide better models of the brain (Matusz et al., 2019; Shamay-Tsoory and Mendelsohn, 2019). In this study, we aimed to link subjective emotional arousal with alpha power in a naturalistic setting. Participants completed an immersive VR experience that included virtual rollercoaster rides while their EEG was recorded. They then continuously rated their emotional arousal while viewing a replay of their previous experience (McCall et al., 2015). Methodological challenges of naturalistic experiments To tackle the challenges of data acquired in naturalistic settings and with continuous stimuli, we made use of recent advances in signal processing and statistical modelling: spatial filtering methods (originally developed for brain-computer interfaces [BCIs]; Blankertz et al., 2008) have recently gained popularity in cognitive neuroscience (Cohen, 2018; Zuure and Cohen, 2020), where they have been used to analyze continuous data collected in naturalistic experiments, for example, to find inter-subject correlations in neuroimaging data of participants watching the same movie (Dmochowski et al., 2012; Gaebler et al., 2014). For the present experiment, two spatial filtering methods were applied to link alpha power and subjective emotional arousal: source power comodulation (SPoC; Dähne et al., 2014) and common spatial patterns (CSP; Blankertz et al., 2008; Ramoser et al., 2000). SPoC is a supervised regression approach, in which a target variable (here: subjective emotional arousal) guides the extraction of relevant M/EEG oscillatory components (here: alpha power). SPoC has been used to predict single-trial reaction times from alpha power in a hand motor task (Meinel et al., 2016), muscular contraction from beta power (Sabbagh et al., 2020), and difficulty levels of a video game from theta and alpha power (Naumann et al., 2016). CSP is used to decompose a multivariate signal into components that maximize the difference in variance between distinct classes (here: periods of high and low emotional arousal). CSP thereby allows optimizing the extraction of power-based features from oscillatory signals, which can then be applied for training classifiers to solve binary or categorical prediction problems. CSP is being used with EEG for BCI (Blankertz et al., 2008) or to decode workload (Schultze-Kraft et al., 2016). In addition to M/EEG-specific spatial filtering methods, non-linear machine learning methods are suited for the analysis of continuous, multidimensional recordings from naturalistic experiments. Deep neural networks transform high-dimensional data into target output variables (here: different states of emotional arousal) by finding statistical invariances and hidden representations in the input (Goodfellow et al., 2016; LeCun et al., 2015; Schmidhuber, 2015). For time-sequential data, long short-term memory (LSTM) recurrent neural networks (RNNs) are particularly suited (Greff et al., 2017; Hochreiter and Schmidhuber, 1995; Hochreiter and Schmidhuber, 1997). Via nonlinear gating units, the LSTM determines which information flows in and out of the memory cell in order to find long- and short-term dependencies over time. LSTMs have been successfully applied for speech recognition (Graves et al., 2013), language translation (Luong et al., 2015), or scene analysis in videos (Donahue et al., 2015), but also to detect emotions in speech and facial expressions (Wöllmer et al., 2010; Wöllmer et al., 2008) or workload in EEG (Bashivan et al., 2016; Hefron et al., 2017). In comparison to other deep learning methods, LSTMs are ‘quick learners’ due to their efficient gradient flow and thus suitable for the continuous and sparse data recorded under naturalistic stimulation with VR. The present study tested the hypothesis of a negative association between parieto-occipital alpha power and subjective emotional arousal under dynamic and interactive stimulation. Combining immersive VR and EEG, this study aimed to (1) induce variance in emotional arousal in a naturalistic setting and (2) capture the temporally evolving and subjective nature of emotional arousal via continuous ratings in order to (3) assess their link to oscillations of brain activity in the alpha frequency range. The link between subjective emotional arousal and alpha power was then tested by decoding the former from the latter using the three complementary analysis techniques SPoC, CSP, and LSTM. Materials and methods Participants Forty-five healthy young participants were recruited via the participant database at the Berlin School of Mind and Brain (an adaption of ORSEE; Greiner, 2015). Previous studies on the relationship between emotional arousal and neural oscillations reported samples of 19–32 subjects (e.g., Koelstra et al., 2012; Luft and Bhattacharya, 2015). We recruited more participants to compensate for anticipated dropouts due to the VR setup and to ensure a robust estimate of the model performances. Inclusion criteria were right-handedness, normal or corrected-to-normal vision, proficiency in German, no (self-reported) psychiatric, or neurological diagnoses in the past 10 years, and less than 3 hr of experience with VR. Participants were requested to not drink coffee or other stimulants 1 hr before coming to the lab. The experiment took ~2.5 hr, and participants were reimbursed with 9€ per hour. They signed informed consent before their participation, and the study was approved by the Ethics Committee of the Department of Psychology at the Humboldt-Universität zu Berlin. Setup, stimuli, and measures The experiment was conducted in a quiet room, in which the temperature was kept constant at 24°C. Neurophysiology/EEG Request a detailed protocol Thirty channels of EEG activity were recorded in accordance with the international 10/20 system (Sharbrough et al., 1991) using a mobile amplifier (LiveAmp32) and active electrodes (actiCap; both by BrainProducts, Gilching, Germany, RRID:SCR_009443). Two additional electrooculogram (EOG) electrodes were placed below and next to the right eye to track eye movements. Data were sampled at 500 Hz with a hardware-based low-pass filter at 131 Hz and referenced to electrode FCz. The amplifier was placed on a high table in the back of the participant to minimize the pull on electrode cables and provide maximal freedom for head movements. The VR headset was placed carefully on top of the EEG cap, and impedances were brought below 10 kΩ. With the same amplifier, electrocardiography and galvanic skin responses were additionally acquired. These peripheral physiological data and the inter-individual differences in interoceptive accuracy are beyond the scope of this paper, and their results will be reported elsewhere. VR HMD Request a detailed protocol An HTC Vive head-mounted display (HMD; HTC, New Taipei, Taiwan) and headphones (AIAIAI Tracks, ApS, Copenhagen, Denmark) were placed on top of the EEG cap using small, custom-made cushions to avoid pressure artefacts and increase comfort. The HTC Vive provides stereoscopy with two 1080 × 1200-pixel OLED displays, a 110° field-of-view, and a frame rate of 90 Hz. The user’s head position is tracked using infrared light, accelerometry, and gyroscopy. Head movements were recorded by adapting scripts from Thor, 2016. Immersive VR experience/stimulation Request a detailed protocol Stimulation comprised two commercially available rollercoaster rides (‘Russian VR Coasters’ by Funny Twins Games, Ekaterinburg, Russia, on Steam) that were separated by a 30 s break (during which participants kept their eyes open and looked straight): the ‘Space’ rollercoaster, a 153 s ride through planets, asteroids, and spaceships and the ‘Andes’ rollercoaster, a 97 s ride through a mountain scenery (for more details, see Figure 5 and the Appendix 1). The two rollercoaster rides were commercially available on Steam. The rollercoasters were selected for their length (to not cause physical discomfort by wearing the HMD for too long) and content (to induce variance in emotional arousal). The experience, comprising the sequence ‘Space’-break-‘Andes’, was kept constant across participants. Self-reports Questionnaires Request a detailed protocol At the beginning of the experiment, participants completed two arousal-related questionnaires: (1) the ‘Trait’ subscale of the ‘State-Trait Anxiety Inventory’ (STAI-T; Spielberger, 1983; Spielberger, 1989) and (2) the ‘Sensation Seeking’ subscale of the ‘UPPS Impulsive Behaviour Scale’ (UPPS; Schmidt et al., 2008; Whiteside and Lynam, 2001). Before and after the experiment, participants completed a customized version of the ‘Simulator Sickness Questionnaire’ (SSQ, Bouchard et al., 2017) comprising three items from the nausea (general discomfort, nausea, dizziness) and three items from the oculomotor subscale (headache, blurred vision, difficulty concentrating) to capture potential VR side effects (Sharples et al., 2008). After the experiment, participants also rated the presence and valence of their experience (the results will be reported elsewhere). Emotional arousal Request a detailed protocol After each VR experience, participants watched a 2D recording (recorded using OBS Studio, https://obsproject.com/) of their experience on a virtual screen (SteamVR’s ‘view desktop’ feature), that is, without removing the HMD. They recalled and continuously rated their emotional arousal by turning a dial (PowerMate USB, Griffin Technology, Corona, CA; sampling frequency: 50 Hz), with which they manipulated a vertical rating bar, displayed next to the video, ranging from low (0) to high (100) in 50 discrete steps (McCall et al., 2015; see Figure 1B). The exact formulation was ‘When we show you the video, please state continuously how emotionally arousing or exciting the particular moment during the VR experience was’ (German: ‘Wenn wir dir das Video zeigen, gebe bitte durchgehend an, wie emotional erregend, bzw. aufregend der jeweilige Moment während der VR Erfahrung war’). To present the playback video and the rating bar, a custom script written in Processing (v3.0) was used. Figure 1 Download asset Open asset Schematic of experimental setup. (A) The participants underwent the experience (two rollercoasters separated by a break) in immersive virtual reality (VR), while EEG was recorded. (B) They then continuously rated the level of emotional arousal with a dial viewing a replay of their experience. The procedure was completed twice, without and with head movements. Procedure Request a detailed protocol Participants came to the lab and filled in the pre-test questionnaires. After the torso and limb electrodes had been attached, participants completed a heartbeat guessing task (Schandry, 1981) to assess inter-individual differences in interoceptive accuracy (the results of peripheral physiology and interoception will be reported elsewhere). Then, the EEG cap was attached, and the HMD was carefully placed on top of it. To prevent or minimize (e.g., movement-related) artefacts, customized cushions were placed below the straps of the VR headset to reduce the contact with the EEG sensors. In addition, the VR experience took place while seated and without full body movements (participants were asked to keep their feet and arms still during the recordings). A white grid was presented in the HMD to ensure that the participants’ vision was clear. They then completed a 10 min resting-state phase (5 min eyes open, 5 min eyes closed), before experiencing the first VR episode, which consisted of the two virtual rollercoaster rides and the intermediate break: first the ‘Space’ and then, after the break, the ‘Andes’ rollercoaster. In the subsequent rating phase, they recalled and continuously rated their emotional arousal while viewing a 2D recording of their experience. Importantly, each participant completed the VR episode (plus rating) twice: once while not moving the head (nomov condition) and once while freely moving the head (mov condition) during the VR experience. The sequence of the movement conditions was counterbalanced across participants (n = 19 with nomov condition first). At the end of the experiment, participants completed two additional questionnaires (the SUS and the questionnaire on subjective feelings of presence and valence during the virtual rollercoaster rides) before they were debriefed. Data analysis Request a detailed protocol To exclude effects related to the on- or offset of the rollercoasters, data recorded during the first and the last 2.5 s of each rollercoaster were removed and the inter-individually slightly variable break was cropped to 30 s. The immersive VR experience that was analysed thus consisted of two time series of 270 s length each per participant (nomov and mov). Self-reports Questionnaires Request a detailed protocol Inter-individual differences as assessed by the trait questionnaires were not the focus of this study, and their results (together with the peripheral physiological and interoception data) will be reported elsewhere. The sum of the simulator sickness ratings before and after the experiment was compared using a two-sided paired t-test. Emotional arousal Request a detailed protocol Emotional arousal ratings were resampled to 1 Hz by averaging non-overlapping sliding windows, yielding one arousal value per second. For the classification analyses, ratings were divided by a tertile split into three distinct classes of arousal ratings (low, medium, high) per participant. For the binary classification (high vs. low arousal), the medium arousal ratings were discarded. Neurophysiology Preprocessing Request a detailed protocol EEG data were preprocessed and analyzed with custom MATLAB (RRID:SCR_001622) scripts built on the EEGLAB toolbox (RRID:SCR_007292, v13.5.4b; Delorme and Makeig, 2004). The preprocessing steps were applied separately for data recorded during the nomov and mov conditions (i.e., without and with head movement). Continuous data were downsampled to 250 Hz (via the ‘pop_resample.m’ method in EEGLAB) and PREP pipeline (v0.55.2; Bigdely-Shamlo et al., 2015) procedures were applied for detrending (1 Hz high-pass filter, Hamming windowed zero-phase sinc FIR filter, cutoff frequency (–6 dB): 0.5 Hz, filter order: 827, transition band width: 1 Hz), line-noise removal (line frequency: 50 Hz), robust referencing to average, and detection as well as spherical interpolation of noisy channels. Due to the relatively short lengths of the time series, the default fraction of bad correlation windows (parameter ‘badTimeThreshold’, used to mark bad channels) was increased to 0.05. For all other parameters, default values of PREP were kept. On average, 2.08 and 2.47 channels per subject were interpolated in the nomov and mov condition, respectively. Data remained high-pass filtered for the further steps of the analysis. Retrospective arousal ratings were added to the data sets, labelling each second of data with an associated arousal rating used as target for the later classification and regression approaches. ICA decomposition was used to identify and remove EEG artefacts caused by eye movements, blinks, and muscular activity. To facilitate the decomposition, ICA projection matrices were calculated on a subset of the data from which the noisiest parts had been removed. To this end, a copy of the continuous data was split into 270 epochs of 1 s length. Epochs containing absolute voltage values > 100 µV in at least one channel (excluding channels that reflected eye movements, i.e., EOG channels, Fp1, Fp2, F7, F8) were deleted. Extended infomax (Lee et al., 1999) ICA decomposition was calculated on the remaining parts of the data (after correcting for rank deficiency with a principal component analysis). Subjects with >90 to-be-deleted epochs (33% of the data) were discarded from further analyses (nomov: n = 5; mov: n = 10). Artefactual ICA components were semi-automatically selected using the SASICA extension (Chaumon et al., 2015) of EEGLAB and visual inspection. On average, 13.41 (nomov) and 10.31 (mov) components per subject were discarded. The remaining ICA weights were back-projected onto the continuous time series. Dimensionality reduction: SSD in the (individual) alpha frequency range Our main hypothesis was that EEG-derived power in the alpha frequency range allows the discrimination between different states of arousal. To calculate alpha power, we adopted spatio-spectral decomposition (SSD; Nikulin et al., 2011) which extracts oscillatory sources from a set of mixed signals. Based on generalized eigenvalue decomposition, it finds the linear filters that maximize the signal in a specific frequency band and minimize noise in neighbouring frequency bands. Preprocessing with SSD has been previously shown to increase classification accuracy in BCI applications (Haufe et al., 2014a). The alpha frequency range is typically fixed between 8 and 13 Hz. The individual alpha peak frequency, however, varies intra- and inter-individually, for example, with age or cognitive demand (Haegens et al., 2014; Mierau et al., 2017). To detect each participant’s individual peak of alpha oscillations for the SSD, (1) the power spectral density (PSD) of each channel was calculated using Welch’s method (segmentlength=5s∗samplingfrequency [i.e., 250 Hz] with 50% overlap) in MATLAB (pwelch function). (2) To disentangle the power contribution of the 1/f aperiodic signal from the periodic component of interest (i.e., alpha), the MATLAB wrapper of the FOOOF toolbox (v0.1.1; Haller et al., 2018; frequency range: 0–40 Hz, peak width range: 1–12 Hz, no minimum peak amplitude, threshold of two SDs above the noise of the flattened spectrum) was used. The maximum power value in the 8–13 Hz range was considered the individual alpha peak frequency αi, on which the SSD bands of interest were defined (bandpass signal αi ± 2 Hz, bandstop noise αi ± 3 Hz, bandpass noise αi ± 4 Hz). The entire procedure was separately applied to the nomov and the mov condition to account for potential peak variability (Haegens et al., 2014; Mierau et al., 2017). SSD was then computed based on these peaks. A summary of the resulting individual alpha peak frequencies can be found in Figure 2—source data 1. Figure 2 shows the averaged power spectrum across all participants and electrodes. A clearly defined peak in the alpha frequency range is discernible for both conditions (nomov, mov) as well as for states of high and low emotional arousal. Figure 2 Download asset Open asset Group averaged power spectra for the two emotional arousal levels (low, high) and head movement conditions (nomov, mov). Thick lines represent the mean log-transformed power spectral density of all participants and electrodes. Shaded areas indicate the standard deviation of the participants. High and low emotional arousal are moments that have been rated as most (top tertile) and least arousing (bottom tertile), respectively (