A Real-Time and Privacy-Preserving Facial Expression Recognition System Using an AI-Powered Microcontroller

Abstract

This study proposes an edge computing-based facial expression recognition system that is low cost, low power, and privacy preserving. It utilizes a minimally obtrusive cap-based system designed for the continuous and real-time monitoring of a user’s facial expressions. The proposed method focuses on detecting facial skin deformations accompanying changes in facial expressions. A multi-zone time-of-flight (ToF) depth sensor VL53L5CX, featuring an 8 × 8 depth image, is integrated into the front brim of the cap to measure the distance between the sensor and the user’s facial skin surface. The distance values corresponding to seven universal facial expressions (neutral, happy, disgust, anger, surprise, fear, and sad) are transmitted to a low-power STM32F476 microcontroller (MCU) as an edge device for data preprocessing and facial expression classification tasks utilizing an on-device pre-trained deep learning model. Performance evaluation of the system is conducted through experiments utilizing data collected from 20 subjects. Four deep learning algorithms, including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) networks, and Deep Neural Networks (DNN), are assessed. These algorithms demonstrate high accuracy, with CNN yielding the best result, achieving an accuracy of 89.20% at a frame rate of 15 frames per second (fps) and a maximum latency of 2 ms.