DRER: Deep Learning-Based Driver's Real Emotion Recognizer.

Geesung Oh,Sejoon Lim,Ji Hyun Yang,Junghwan Ryu,Euiseok Jeong,Sungwook Hwang,Sangho Lee

doi:10.3390/s21062166

Geesung Oh, Sejoon Lim + Show 5 more

Open Access

https://doi.org/10.3390/s21062166

Copy DOI

Abstract

In intelligent vehicles, it is essential to monitor the driver’s condition; however, recognizing the driver’s emotional state is one of the most challenging and important tasks. Most previous studies focused on facial expression recognition to monitor the driver’s emotional state. However, while driving, many factors are preventing the drivers from revealing the emotions on their faces. To address this problem, we propose a deep learning-based driver’s real emotion recognizer (DRER), which is a deep learning-based algorithm to recognize the drivers’ real emotions that cannot be completely identified based on their facial expressions. The proposed algorithm comprises of two models: (i) facial expression recognition model, which refers to the state-of-the-art convolutional neural network structure; and (ii) sensor fusion emotion recognition model, which fuses the recognized state of facial expressions with electrodermal activity, a bio-physiological signal representing electrical characteristics of the skin, in recognizing even the driver’s real emotional state. Hence, we categorized the driver’s emotion and conducted human-in-the-loop experiments to acquire the data. Experimental results show that the proposed fusing approach achieves 114% increase in accuracy compared to using only the facial expressions and 146% increase in accuracy compare to using only the electrodermal activity. In conclusion, our proposed method achieves 86.8% recognition accuracy in recognizing the driver’s induced emotion while driving situation.

Highlights

Drivers’ emotional state affects their ability to drive [1,2]
We propose a facial expression recognition (FER) model constructed with reference to several state-of-the-art convolutional neural networks (CNNs), such as VGGNet [28], ResNet [29], ResNeXt [30] and SENet [31]
The driver’s real emotion recognizer (DRER) algorithm that we proposed is constructed by combining the FER model and sensor fusion emotion recognition (SFER) model with the best performance

Summary

Introduction

Drivers’ emotional state affects their ability to drive [1,2]. As vehicles become more intelligent, it becomes increasingly important to recognize the driver’s emotions. In the human–machine interface, facial expressions are considered important because they are useful for revealing emotions between people These methods based on facial expressions have been established as a research field called facial expression recognition (FER). When a driver frowns while driving, it may be tempting to assume that the driver is currently in an unpleasant state if the judgment is made purely based on the driver’s facial expressions. If it is the reaction of the driver’s facial muscles to the stimulus of sunlight, the driver should not be judged to be in an unpleasant state. Such micro changes normally occur when the real emotions are concealed deliberately or unconsciously

Objectives

Methods

Results

Conclusion