Towards affect‐aware vehicles for increasing safety and comfort: recognising driver emotions from audio recordings in a realistic driving study

Abstract

For vehicle safety, the in-time monitoring of the driver and assessing his/her state is a demanding issue. Frustration can lead to aggressive driving behaviours, which play a decisive role in up to one-third of fatal road accidents. Consequently, the authors present the automatic analysis of the emotional driver states of frustration, anxiety, positive and neutral. Based on experiments with normal drivers within cars in real-world (low expressivity) situations, they use speech data, as speech can be recorded with zero invasiveness and comes naturally in driving situations. A careful selection of speech features, subject data identification, hyper-parameter optimisation, and machine learning algorithms was applied for this difficult 4-emotion-class detection problem, where the literature hardly reports results above chance level. In-car assistance demands real-time computing. A very detailed analysis yields best results with relatively small random forests, and with an optimal feature set containing only 65 features (6.51% of the standard emobase feature set) which outperformed all other feature sets, producing 35.38% unweighted average recall (53.26% precision) with low computational effort, and also reducing the inevitably high confusion of ‘neutral’ with low-expressed emotions. This result is comparable to and even outperforming other reported studies of emotion recognition in the wild. Their work, therefore, triggers adaptive automotive safety applications.