Mist–fog-assisted real-time emotion recognition using deep transfer learning framework for smart city 4.0

Abstract

Face emotion recognition (FER) has found several applications across numerous industries. One such application area lies in the field of human–computer interaction, where it can improve user experiences by allowing systems to adjust based on users’ emotional states by leveraging the capabilities of the Internet of Things (IoT) to facilitate more responsive interactions. The increasing demand for precise and immediate emotion analysis across various applications drives the need for FER utilizing deep transfer learning (DTL). In this study, a novel Mist-Fog framework for real-time FER is introduced, which employs a DTL algorithm called Mobile-Net for quick and easy image classification on mobile and embedded devices. It addresses the challenges of operating deep neural networks (DNN) under limits on power and processing resources. By utilizing depth-wise separable convolutions, the MobileNet algorithm reduces computational complexity while maintaining reasonable accuracy on the FER-2013 dataset. A Fog-based architecture, in which a mist layer comprising edge devices like Raspberry-Pi (RPi), an ARM-based single-board computer, was considered for a smart city scenario. Further, the M/M/1/K queuing system was considered, which follows the First-Come-First-Serve (FCFS) scheduling criteria and a Markov Modulated Poisson Process (MMPP) arrival rate to evaluate the system’s performance. The effectiveness of this novel approach towards industrial computer vision paradigms is assessed pertaining to different performance criteria.