Hierarchical CNN‐based real‐time fatigue detection system by visual‐based technologies using MSP model

Abstract

Visual-based technologies are very useful and meaningful to driver's fatigue detection. In this study, the authors present a multi-task hierarchical CNN scheme for fatigue detection system and propose a convolutional neural network (CNN) model with multi-scale pooling (MSP-Net). `Multi-task' includes three tasks: face detection, eye and mouth state detection and fatigue detection. First, they use a pre-trained network - multi-task CNN for face detection extracting eye and mouth regions. Then, the main work of this study, eye and mouth state detection is processed by MSP-Net, which can fit multi-resolution input images captured from variant cameras excellently. For the third step, the percentage of eyelid closure over the pupil over time (PERCLOS) parameters and the frequency of open mouth (FOM) parameters are used to detect fatigue, and the FOM parameters are proposed by ourselves. Besides, they successfully port the system to the embedded platform (the NVIDIA JETSON TX2 development board) and test on real driving scene. The results show that their system performs well and is robust to complex environments and is in line with the demand of real-time system.