Decoding emotional changes of android-gamers using a fused Type-2 fuzzy deep neural network

Abstract

With the fastest growing popularity of gaming applications on android phone, analyzing emotion changes of steadfast android-gamers have become a study of utmost interest among most of the psychologists. Recently, some android games are producing negative impacts to the gamers; even in the worst cases the effect is becoming life-threatening too. Most of the existing research works are based on psychological view-point of exploring the impact (positive/negative) of playing android games for the child and adult age-group. However, the online recognition of emotional state changes of the android-gamers while playing video games may be relatively unexplored. To fill this void, the present study proposes a novel method of identifying the emotional state changes of android-gamers by decoding their brain signals and facial images simultaneously during playing video games. Besides above, the second novelty of the paper lies in designing a multimodal fusion method between brain signals and facial images for the said application. To address this challenge, the paper proposes a fused type-2 fuzzy deep neural network (FT2FDNN) which integrates the brain signal processing approach by a general type-2 fuzzy reasoning algorithm with the flavor of the image/video processing approach using a deep convolutional neural network. FT2FDNN uses multiple modalities to extract the similar information (here, emotional changes) simultaneously from the type-2 fuzzy and deep neural representations. The proposed fused type-2 fuzzy deep learning paradigm demonstrates promising results in classifying the emotional changes of gamers with high classification accuracy. Thus the proposed work explores a new era for future researchers.