Fatigue driving is an important cause of traffic accidents, so it is particularly important to achieve effective fatigue driving detection. The current fatigue driving detection algorithm has the problems of poor comfort, vulnerable to external factors (light intensity, wearing masks and sunglasses), and poor real-time performance. To solve these problems, we design a system for detecting driver’s facial features (FFD-System) and an algorithm for judging driver’s fatigue state (MF-Algorithm). In FFD-System, we designed three networks M1-FDNet, M2-PENet and M3-SJNet for driver face detection, head pose estimation and eye-mouth state judgment respectively. M1-FDNet is a lightweight face detection network. We design a new loss function to allow the network to output five key face points in addition to the output face. M2-PENet converts the regression problem into a classification problem, and the head pose can be estimated only by a single face image. M3-SJNet can use the global information of eyes and mouth to determine its state. Wearing sunglasses or masks while driving can cause the driver’s eyes or mouth to be obscured, which makes it impossible to accurately detect all of the driver’s facial features. MF-Algorithm can solve this problem well. MF-Algorithm uses the output of FFD-System to calculate single eye closure time, blinking frequency, eye closure rate, mouth opening frequency and head misalignment time, and combines these parameters to comprehensively judge the driver’s fatigue state. Thus, even if a single facial feature cannot be accurately identified, our algorithm can accurately determine the driver’s fatigue status based on other features. The experimental results show that the algorithm meets the requirements of real-time detection in terms of detection speed and accuracy.