A Novel Facial Expression Recognition Method for Identifying and Recording Emotion

Abstract

In this study, an accurate, innovative, and real-time emotion recognition and recording system based on novel deep learning techniques is proposed. A nine-layered convolutional neural network (CNN) is built with the addition of backpropagation (BP) to increase accuracy and Haar-Cascade with Open Source Computer Vision (OpenCV) for frontal face detection. In past research, mass datasets for investigations into emotion are often collected through case studies, surveys, or naturalistic observations. Due to their subjective nature, these studies often yield biased results. Through the use of facial recognition software trained with the Adam optimizer and evaluated on Kaggle’s Fer2013 public dataset, this CNN can recognize six universal emotions – happiness, sadness, anger, fearful, surprise, and disgust – with the addition of neutral for other or no emotion. After training, this method has reached a train accuracy of 95% and a test accuracy of 53.55%. In addition, the system uses an unprecedented method that creates a graph demonstrating the short-term fluctuation of emotion on a millisecond basis. The final emotion value of each short-term evaluation will appear in a separate file to document long-term emotional status. Aside from software, a smart mirror including a camera and a LED Electronic Screen is built for real-time feedback of people’s emotional states. Every 100 millisecond, a label and an emoji will appear on the mirror to create an easily accessible and entertaining form of emotion detection.