Emotion Prediction from Physiological Signals: A Comparison Study Between Visual and Auditory Elicitors

Abstract

Unlike visual stimuli, little attention has been paid to auditory stimuli in terms of emotion prediction with physiological signals. This paper aimed to investigate whether auditory stimuli can be used as an effective elicitor as visual stimuli for emotion prediction using physiological channels. For this purpose, a well-controlled experiment was designed, in which standardized visual and auditory stimuli were systematically selected and presented to participants to induce various emotions spontaneously in a laboratory setting. Numerous physiological signals, including facial electromyogram, electroencephalography, skin conductivity and respiration data, were recorded when participants were exposed to the stimulus presentation. Two data mining methods, namely decision rules and k-nearest neighbor based on the rough set technique, were applied to construct emotionpredictionmodelsbasedonthefeaturesextractedfromthephysiologicaldata.Experimental results demonstrated that auditory stimuli were as effective as visual stimuli in eliciting emotions in terms of systematic physiological reactivity. This was evidenced by the best prediction accuracy quantified by the F1 measure (visual: 76.2% vs. auditory: 76.1%) among six emotion categories (excited, happy, neutral, sad, fearful and disgusted). Furthermore, we also constructed culturespecific (Chinese vs. Indian) prediction models. The results showed that model prediction accuracy was not significantly different between culture-specific models. Finally, the implications of affective auditory stimuli in human‐computer interaction, limitations of the study and suggestions for further research are discussed. RESEARCH HIGHLIGHTS • Emotions can be predicted accurately based on physiological data using computational models. • Prediction accuracy was on the same level for auditory and visual stimuli. • Auditory stimuli can thus have potential in affective human–computer interaction applications.