Real-time video based emotion recognition using convolutional neural network and transfer learning

J Sujanaa

doi:10.17485/ijst/v13i31.1118

Abstract

Background/Objectives: The deep learning approaches have paved their way to construct various artificial intelligence products and the proposed system uses a convolutional neural network for detecting real-time emotions of mankind. The objective of the study is to develop a real-time application for emotion recognition using convolutional neural network and transfer learning methods. Methods/Statistical analysis: The proposed system considers happy, normal and surprised categories of emotions. The system consists of four major steps: dataset collection, training, validation, and real-time testing. The dataset is comprised of face images containing emotions such as happy,normal and surprised in the form of video frames. The face and mouth regions are detected using the Haar-Based Cascade classifier at 20 frames per second. Findings: The convolutional neural network (CNN) is trained using mouth images and the pre-trained models VGG16 and VGG19 are trained with face images. The trained model is used to detect the emotions in the live webcam video. The experimental results show that the CNN model trained using mouth images gives an accuracy of 85.71% and the pre-trained models trained with face images using transfer learning method achieves an accuracy of 77.78%. The proposed system using CNN outperforms the pre-trained models for recognizing the emotions in real-time video. Novelty/Applications:The proposed system is entirely based on the mouth region video frames and the real-time emotion recognition system is developed. This work can detect the three emotions in an unconstrained laboratory environment. Keywords: Convolutional neural network; mouth detection; pre-trained models; real-time emotion recognition; transfer learning

Highlights

The emotion recognition places a very indispensable role in inspecting human feelings and internal thoughts more precisely.The emotions and mood lead in identifying the human mind quickly
A total of 8400 mouth images and 5400 face images are used from 20 subjects (10 Male, 10 Female)
For training the convolutional neural network (CNN) architecture, 6720 mouth images are used for training and 1680 mouth images are used for validation

Summary

Introduction

The emotion recognition places a very indispensable role in inspecting human feelings and internal thoughts more precisely.The emotions and mood lead in identifying the human mind quickly. The emotion recognition places a very indispensable role in inspecting human feelings and internal thoughts more precisely. Humans may interact with society through emotions in case of the absence of verbal communication[1]. Among the other non-verbal communications, emotions play a very effective way of exchanging internal thoughts with society. Emotions of humans can be detected through distinct ways such as their verbal responses or voice tone, physical responses or through the body-languages, autonomic responses, etc. The basic types of emotions in a person are happy, normal, surprised, fear, anger, disgust or dislike and sadness. Normal, surprised, disgust and fear are easy to find out whereas other expressions like disgust, amusement, pride, contempt, and shame are very hard to find in human through facial expressions. There are varieties of applications for facial emotions recognition like student classroom behavior monitoring system, airport/railway suspicious person detection system, autism children expression detection, facial expression-based emotion chat applications, real-time person pain monitoring systems, etc.[2,3]

Objectives

Methods

Results

Conclusion