Abstract
The Internet of Things (IoT) technologies such as interconnection and edge computing help emotion recognition to be applied in healthcare, smart education, etc. However, the acquisition and transmission processes may have some situations, such as lost signals and serious interference noise caused by motion, which affect the quality of the received data and limit the performance of IoT emotion detection. We collectively refer to these as invalid data. A multi-step deep (MSD) system is proposed to reliably detect multimodal emotion by the collected records containing invalid data. Semantic compatibility and continuity are utilized to filter out the invalid data. The feature from invalid modal data is replaced through the imputation method to compensate for the impact of invalid data on emotion detection. In this way, the proposed system can automatically process invalid data and improve the recognition performance. Furthermore, considering the spatiotemporal information, the features of video and physiological signals are extracted by specific deep neural networks in the MSD system. The simulation experiments are conducted on a public multimodal database, and the performance of the MSD system measured by the unweighted average recall is better than that of the traditional system. The promising results observed in the experiments verify the potential influence of the proposed system in practical IoT applications.
Highlights
The ability to perceive human emotions can be used to provide more personalized interactive products
As this study mainly focuses on data processing of emotion detection based on video and physiological signals, the rest aspects of Internet of Things (IoT) framework can solely rely on the methods described in relevant works, such as [1], [11], [33]
DATASETS AND SETTINGS 1) Remote Collaborative and Affective Interactions (RECOLA) DATASET The RECOLA database has been widely utilized for multimodal emotion recognition and has been provided for The Audio/Visual Emotion Challenge and Workshop (AVEC) since 2015 [46]–[48]
Summary
The ability to perceive human emotions can be used to provide more personalized interactive products. Due to the massive data and powerful computational capacity, the fast growth of Internet of Things (IoT) technologies helps to realize the possibility of real-time human emotion detection or perception in different scenarios. There have been many novel studies combining the IoT and affective computing to provide various emotion detection frameworks for different applications, such as healthcare services [1], battlefield environments [2] and smart homes [3]. These studies treat emotion detection as a part of the overall framework and tend. We use the phrase ‘‘invalid data’’ to refer to the missing data, and to the data whose extracted feature does not
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
