Binary neural network based real time emotion detection on an edge computing device to detect passenger anomaly

Abstract

Passenger safety in public transportation especially while riding in the form of shared cabs, and taxis are often ignored, and not much preventive protocols are devised. In the connected mobility world, emotion recognition from facial expressions is a possibility, however a faster processing and edge computing device to derive anomaly state inferences will be apt for further notifying about the safety of the passenger. FPGA implementation is a viable approach to not only implement in the embedded system automotive electronics, but also accelerate the inference results, hence making it as an ideal real time candidate for passenger anomaly state identification. For the same, a real time emotion detection system using facial features was implemented on FPGA. A Binary Neural Network (BNN) feeded by Local Binary Pattern (LBP) output was designed towards the development of an improved and faster emotion recognition system. LBP is configured as a preprocessing step to extract facial features that is passed on to the BNN layer for successful inference. The preprocessing method utilizes Viola-Jones (VJ) algorithm to extract facial data while removing other background information from the image. The LBP-BNN network is modelled using Facial Expression 2013 (FER-2013) data set for training. The custom hardware accelerator or the overlay is synthesized and the designed IP is implemented on FPGA for the inference. Inference is done using the trained model on FPGA to enable faster classified results. Emotion detection using facial expressions is classified to six states namely: angry, disgust, fear, happy, sad, and surprise. The LBP-BNN network is implemented in FPGA, to realize a real time facial emotion recognition by capturing the image of a person from a web camera interfaced to the FPGA acting as edge computing inference device, with acceptable accuracy. The image processing based emotion detection design is highly suitable for other applications including tracking of emotions for movement disorder patients in hospitals.