Eye State Detection Research Articles

A Real-Time Embedded System for Driver Drowsiness Detection Based on Visual Analysis of the Eyes and Mouth Using Convolutional Neural Network and Mouth Aspect Ratio.

Currently, the number of vehicles in circulation continues to increase steadily, leading to a parallel increase in vehicular accidents. Among the many causes of these accidents, human factors such as driver drowsiness play a fundamental role. In this context, one solution to address the challenge of drowsiness detection is to anticipate drowsiness by alerting drivers in a timely and effective manner. Thus, this paper presents a Convolutional Neural Network (CNN)-based approach for drowsiness detection by analyzing the eye region and Mouth Aspect Ratio (MAR) for yawning detection. As part of this approach, endpoint delineation is optimized for extraction of the region of interest (ROI) around the eyes. An NVIDIA Jetson Nano-based device and near-infrared (NIR) camera are used for real-time applications. A Driver Drowsiness Artificial Intelligence (DD-AI) architecture is proposed for the eye state detection procedure. In a performance analysis, the results of the proposed approach were compared with architectures based on InceptionV3, VGG16, and ResNet50V2. Night-Time Yawning-Microsleep-Eyeblink-Driver Distraction (NITYMED) was used for training, validation, and testing of the architectures. The proposed DD-AI network achieved an accuracy of 99.88% with the NITYMED test data, proving superior to the other networks. In the hardware implementation, tests were conducted in a real environment, resulting in 96.55% and 14 fps on average for the DD-AI network, thereby confirming its superior performance.

Cascaded learning with transformer for simultaneous eye landmark, eye state and gaze estimation

Eye tracking have garnered attention in human–machine interaction, disease monitoring, biometrics, etc. Existing investigations for eye tracking have predominantly concentrated on individual task for pupil detection or gaze estimation, overlooking the implicit relationships that exist among different tasks for eye tracking. In this work, we introduce a cascaded framework with transformer to collaboratively realize eye landmark detection, eye state detection and gaze estimation. Within our framework, we leverage Transformer to capture long dependencies with explicit eye-related structural information and implicit correlation among different tasks. Furthermore, the proposed cascade iteration framework alternatively optimize each task and boost the overall performance for pupil center, eye state and gaze estimation simultaneously. To address the problem of manual annotation, we further introduce the Control-Eye Diffusion Model (CEDM), a controllable eye image generation method conditioned on a simple contour with structure information. The proposed methods are evaluated on challenging datasets such as GI4E, BioID and MPIIGaze, and the results show that our methods outperform state-of-the-art methods in several tasks.

Efficient Eye State Detection for Driver Fatigue Monitoring Using Optimized YOLOv7-Tiny

This study refines the YOLOv7-tiny model through structured pruning and architectural fine-tuning, specifically for real-time eye state detection. By focusing on enhancing the model’s efficiency, particularly in environments with limited computational resources, this research contributes significantly to advancing driver monitoring systems, where timely and accurate detection of eye states such as openness or closure can prevent accidents caused by drowsiness or inattention. Structured pruning was utilized to simplify the YOLOv7-tiny model, reducing complexity and storage requirements. Subsequent fine-tuning involved adjustments to the model’s width and depth to further enhance processing speed and efficiency. The experimental outcomes reveal a pronounced reduction in storage size, of approximately 97%, accompanied by a sixfold increase in frames per second (FPS). Despite these substantial modifications, the model sustains high levels of precision, recall, and mean average precision (mAP). These improvements indicate a significant enhancement in both the speed and efficiency of the model, rendering it highly suitable for real-time applications where computational resources are limited.

Eye-LRCN: A Long-Term Recurrent Convolutional Network for Eye Blink Completeness Detection.

Computer vision syndrome causes vision problems and discomfort mainly due to dry eye. Several studies show that dry eye in computer users is caused by a reduction in the blink rate and an increase in the prevalence of incomplete blinks. In this context, this article introduces Eye-LRCN, a new eye blink detection method that also evaluates the completeness of the blink. The method is based on a long-term recurrent convolutional network (LRCN), which combines a convolutional neural network (CNN) for feature extraction with a bidirectional recurrent neural network that performs sequence learning and classifies the blinks. A Siamese architecture is used during CNN training to overcome the high-class imbalance present in blink detection and the limited amount of data available to train blink detection models. The method was evaluated on three different tasks: blink detection, blink completeness detection, and eye state detection. We report superior performance to the state-of-the-art methods in blink detection and blink completeness detection, and remarkable results in eye state detection.

Eye State Detection Using Frequency Features from 1 or 2-Channel EEG.

Brain-computer interfaces (BCIs) establish a direct communication channel between the human brain and external devices. Among various methods, electroencephalography (EEG) stands out as the most popular choice for BCI design due to its non-invasiveness, ease of use, and cost-effectiveness. This paper aims to present and compare the accuracy and robustness of an EEG system employing one or two channels. We present both hardware and algorithms for the detection of open and closed eyes. Firstly, we utilize a low-cost hardware device to capture EEG activity from one or two channels. Next, we apply the discrete Fourier transform to analyze the signals in the frequency domain, extracting features from each channel. For classification, we test various well-known techniques, including Linear Discriminant Analysis (LDA), Support Vector Machine (SVM), Decision Tree (DT), or Logistic Regression (LR). To evaluate the system, we conduct experiments, acquiring signals associated with open and closed eyes, and compare the performance between one and two channels. The results demonstrate that employing a system with two channels and using SVM, DT, or LR classifiers enhances robustness compared to a single-channel setup and allows us to achieve an accuracy percentage greater than 95% for both eye states.

Drowsiness Detection System Based on Machine Learning Using Eye State

Drowsiness is one of the major causes of driver-induced traffic accidents. The interactive systems developed to reduce road accidents by alerting drivers is called as Advanced Driver Assistance Systems (ADAS). The most important ADAS are Lane Departure Warning System, Front Collision Warning System and Driver Drowsiness Systems. In this study, an ADAS system based on eye state detection is presented to detect driver drowsiness. First, Viola-Jones algorithm approach is used to detect the face and eye areas in the proposed method. The detected eye region is classified as closed or open by making use of a machine learning method. Finally, the eye conditions are analyzed at time domain with PERcentage of eyelid CLOsure (PERCLOS) metric and drowsiness conditions are determined by Support Vector Machine (SVM), kNN and decision tree classifiers. The proposed methods tested on 7 real people and drowsiness states are detected at 99.77%, 94.35%, and 96.62% accuracy, respectively.

Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal.

Accidents caused by fatigue occur frequently, and numerous scholars have devoted tremendous efforts to investigate methods to reduce accidents caused by fatigued driving. Accordingly, the assessment of the spirit status of the driver through the eyes blinking frequency and the measurement of physiological signals have emerged as effective methods. In this study, a drowsiness detection system is proposed to combine the detection of LF/HF ratio from heart rate variability (HRV) of photoplethysmographic imaging (PPGI) and percentage of eyelid closure over the pupil over time (PERCLOS), and to utilize the advantages of both methods to improve the accuracy and robustness of drowsiness detection. The proposed algorithm performs three functions, including LF/HF ratio from HRV status judgment, eye state detection, and drowsiness judgment. In addition, this study utilized a near-infrared webcam to obtain a facial image to achieve non-contact measurement, alleviate the inconvenience of using a contact wearable device, and for use in a dark environment. Furthermore, we selected the appropriate RGB channel under different light sources to obtain LF/HF ratio from HRV of PPGI. The main drowsiness judgment basis of the proposed drowsiness detection system is the use of algorithm to obtain sympathetic/parasympathetic nervous balance index and percentage of eyelid closure. In the experiment, there are 10 awake samples and 30 sleepy samples. The sensitivity is 88.9%, the specificity is 93.5%, the positive predictive value is 80%, and the system accuracy is 92.5%. In addition, an electroencephalography signal was used as a contrast to validate the reliability of the proposed method.

Design of a Wearable Eye-Movement Detection System Based on Electrooculography Signals and Its Experimental Validation.

In the assistive research area, human–computer interface (HCI) technology is used to help people with disabilities by conveying their intentions and thoughts to the outside world. Many HCI systems based on eye movement have been proposed to assist people with disabilities. However, due to the complexity of the necessary algorithms and the difficulty of hardware implementation, there are few general-purpose designs that consider practicality and stability in real life. Therefore, to solve these limitations and problems, an HCI system based on electrooculography (EOG) is proposed in this study. The proposed classification algorithm provides eye-state detection, including the fixation, saccade, and blinking states. Moreover, this algorithm can distinguish among ten kinds of saccade movements (i.e., up, down, left, right, farther left, farther right, up-left, down-left, up-right, and down-right). In addition, we developed an HCI system based on an eye-movement classification algorithm. This system provides an eye-dialing interface that can be used to improve the lives of people with disabilities. The results illustrate the good performance of the proposed classification algorithm. Moreover, the EOG-based system, which can detect ten different eye-movement features, can be utilized in real-life applications.

Eye state detection based on Weight Binarization Convolution Neural Network and Transfer Learning

Detection of eye state can assist the related work in the field of computer vision such as face recognition, expression recognition, pose estimation and human–computer interaction. This paper proposes an Weight Binarization Convolution Neural Network and Transfer Learning (WBCNNTL) based eye state detection method, in which the WBCNNTL is composed of deep convolution neural network and the weight is binarized. The human eye state features can be extracted by Convolutional Neural Network (CNN) effectively, and binary network not only speeds up the computation, but also helps to reduce the storage space and fewer parameters of the model. Transfer learning applies the knowledge or patterns learned from the source domain to different but related fields or problems, which improves the training efficiency of the new model. Experiments on eye state detection are performed using Closed Eyes in the wild (CEW), FER2013 and Zhejiang University Eyeblink (ZJU) Databases, from which the experiment results show the average accuracy obtained by our method are 97.41% on CEW and are 97.15% on ZJU, the computing speed of binary network is faster than non-binary network. Moreover, our method requires less storage space due to lightweight binary model, which maintains better detection capability on CEW compared with some state-of-the-art works.

Hybrid classification model for eye state detection using electroencephalogram signals.

The electroencephalography (EEG) signal is an essential source of Brain-Computer Interface (BCI) technology implementation. The BCI is nothing but a non-muscle communication medium among the external devices and the brain. The basic concept of BCI is to enable the interaction among the neurological ill patients to others with the help of brain signals. EEG signal classification is an essential requirement for various applications such as motor imagery classification, drug effects diagnosis, emotion classification, seizure prediction/detection, eye state prediction/detection, and so on. Thus, there is a need for an efficient classification model that can deal with the EEG datasets more adequately with better classification accuracy, which will further help in developing the automatic solution for the medical domain. In this paper, we have introduced a hybrid classification model for eye state detection using electroencephalogram (EEG) signals. This hybrid classification model has been evaluated with the other traditional machine learning models, eight classification models (Prepossessed + Hypertuned) and six state-of-the-art methods to assess its appropriateness and correctness. This proposed classification model establishes a machine learning-based hybrid model for the classification of eye state using EEG signals with greater exactness. It is also capable of solving the issue of outlier detection and removal to address the class imbalance problem, which will offer the solution toward building the robotic or smart machine-based solution for social well-being.

Hierarchical CNN‐based real‐time fatigue detection system by visual‐based technologies using MSP model

Visual-based technologies are very useful and meaningful to driver's fatigue detection. In this study, the authors present a multi-task hierarchical CNN scheme for fatigue detection system and propose a convolutional neural network (CNN) model with multi-scale pooling (MSP-Net). `Multi-task' includes three tasks: face detection, eye and mouth state detection and fatigue detection. First, they use a pre-trained network - multi-task CNN for face detection extracting eye and mouth regions. Then, the main work of this study, eye and mouth state detection is processed by MSP-Net, which can fit multi-resolution input images captured from variant cameras excellently. For the third step, the percentage of eyelid closure over the pupil over time (PERCLOS) parameters and the frequency of open mouth (FOM) parameters are used to detect fatigue, and the FOM parameters are proposed by ourselves. Besides, they successfully port the system to the embedded platform (the NVIDIA JETSON TX2 development board) and test on real driving scene. The results show that their system performs well and is robust to complex environments and is in line with the demand of real-time system.

Deep Learning of EEG Time–Frequency Representations for Identifying Eye States

A new Convolutional Neural Network (CNN) architecture to classify nonstationary biomedical signals using their time–frequency representations is proposed. The present method uses the spectrogram of the biomedical signals as an input to CNN, in addition Non-negative matrix factorization (NMF) dictionary elements are used as an additional feature to improve the performance of the CNN model. Considering a number of applications involving eye state classification, such as in Parkinson’s disease detection, analysis of eye fatigue in 3D TVs, driver’s drowsiness detection, infant sleep-waking state identification, and classification of bipolar mood disorder and attention deficit hyperactivity, the proposed method was applied to Electroencephalography (EEG) data for classification of eye state. First, the spectrogram of EEG signal is obtained and used as an image input to CNN, simultaneously, the NMF feature is also fed to CNN. Further, both features are combined in fully connected layer of CNN architecture. The proposed method is compared with other existing methods for eye state detection and shows good classification accuracy with 96.16%. The prediction rate for the proposed method is 134 observations/second, which is suitable for brain–computer interface applications.

Eye and mouth state detection algorithm based on contour feature extraction

Eye and mouth state analysis is an important step in fatigue detection. An algorithm that analyzes the state of the eye and mouth by extracting contour features is proposed. First, the face area is detected in the acquired image database. Then, the eyes are located by an EyeMap algorithm through a clustering method to extract the sclera-fitting eye contour and calculate the contour aspect ratio. In addition, an effective algorithm is proposed to solve the problem of contour fitting when the human eye is affected by strabismus. Meanwhile, the value of chromatism s is defined in the RGB space, and the mouth is accurately located through lip segmentation. Based on the color difference of the lip, skin, and internal mouth, the internal mouth contour can be fitted to analyze the opening state of mouth; at the same time, another unique and effective yawning judgment mechanism is considered to determine whether the driver is tired. This paper is based on the three different databases to evaluate the performance of the proposed algorithm, and it does not need training with high calculation efficiency.

Explore technology innovation and intelligence for IoT (Internet of Things) based eyewear technology

Advances in wireless Internet and mobile communications devices have driven significant development in the Internet of Things (IoT), bringing a stream of innovative technologies and services. This study explores the technology innovation and intelligence for IoT (internet of things) based eyewear technology. This study proposes a two-stage patent analysis based on the quality function development (QFD) method which adopts customer requirement and technology viewpoints to explore key technologies. This methodology can recognize the specific technologies with development potential in the eyewear industry, and identify holders of key relevant patents. This study finds that consumers value functions including motion tracking, reminders, eye state detection, and non-eye disease detection. Key technologies with development potential for satisfying customer demand include eyewear, communications protocols, and sensors. Thus, embedding micron-scale sensors directly into contact lenses to monitor user physiological data can satisfy customer demand and is considered as emerging technology in the smart eyewear industry. Furthermore, patent portfolios of these technologies vary among different countries and regions, with the US and EU focusing on eye tracking, motion tracking, and identity verification, while China focuses on eye fatigue detection, distance measurement, and wireless frequency technologies. Visualizations of overall research results can benefit eyewear-related patent holders, eyewear manufacturers and smart wearable manufacturers to build their patent portfolio strategies on the basis of regional or country considerations.

EyeLSD a Robust Approach for Eye Localization and State Detection

Improving the safety of public roads and industrial factories requires more reliable and robust computer vision-based approaches for monitoring the eye state (open or closed) of human operators. Getting this information in real time when humans are driving cars or using hazardous machinery will help to prevent accidents and deaths. This paper proposes a new framework called EyeLSD to localize the eyes and detect their states without face detection step. For EyeLSD aims, two novel descriptors are proposed: enhanced Pyramidal Local Binary Pattern Histogram (ePLBPH) and Multi-Three-Patch LBP histogram (Multi-TPLBP). The performance of EyeLSD with ePLBPH and Multi-TPLBP is evaluated and compared against other approaches. For this evaluation three independent and public datasets were used: BioID, CAS-PEAL-R1 and ZJU datasets. The set EyeLSD, ePLBPH and Multi-TPLBP have a greater performance when compared against the state-of-the-art algorithms. The proposed approach is very stable under large range of eye appearances caused by expression, rotation, lighting, head pose, and occlusion.

Levenberg-Marquardt Neural Network for Eye States Detection Based on Electroencephalography Data

The eye state detection is one of various task toward Brain Computer Interface system. The eye state can be read in brain signals. In this paper use EEG Eye State dataset (Rosler, 2013) from UCI Machine Learning Repository Database. Dataset is consisting of continuous 14 EEG measurements in 117 seconds. The eye states were marked as “1” or “0”. “1” indicates the eye-closed and “0” the eye-open state. The proposed schemes use Multi Layer Neural Network with Levenberg Marquardt optimization learning algorithm, as classification method. Levenberg Marquardt method used to optimize the learning algorithm of neural network, because the standard algorithm has a weak convergence rate. It is need many iterations to have minimum error. Based on the analysis towards the experiment on the EEG dataset, it can be concluded that the proposed scheme can be implemented to detect the Eye State. The best accuracy gained from combination variable sigmoid function, data normalization and number of neurons are 31 (95.71%) for one hidden layer, and 98.912% for two hidden layers with number of neurons are 39 and 47 neurons and linear function.

The Classification of Eye State by Using kNN and MLP Classification Models According to the EEG Signals

What is widely used for classification of eye state to detect human’s cognition state is electroencephalography (EEG). In this study, the usage of EEG signals for online eye state detection method was proposed. In this study, EEG eye state dataset that is obtained from UCI machine learning repository database was used. Continuous 14 EEG measurements forms the basic of the dataset. The duration of the measurement is 117 seconds (each measurement has14980 sample). Weka (Waikato Environment for Knowledge Analysis) program is used for classification of eye state. Classification success was calculated by using k-Nearest Neighbors algorithm and multilayer perceptron neural networks models. The obtained success of classification methods were compared. The classification success rates were calculated for various number of neurons in the hidden layer of a multilayer perceptron neural network model. The highest classification success rate have been obtained when the number of neurons in the hidden layer was equal to 7. And it was 56.45%. The classification success rates were calculated with k-nearest neighbors algorithm for different neighbourhood values. The highest success was achieved in the classification made with kNN algorithm. In kNN models, the success rate for 3 nearest neighbor were calculated as 84.05%.

Eye State Detection Based on Embedded Linux System

Eye state detection is dramatically influenced by the position of iris, for this reason, this paper proposed an eye state detection method combined the area between the eyelid with the eyelid contour. By modifying and transplanting V4L-utils and OpenCV image processing library, video capture and display software is built on the Cortex-A8 embedded system. Through experimental verification, the embedded system can realize the acquisition, processing and display of the video stream and the eye state detection algorithm also has high accuracy.

Efficient algorithms for detection of face, eye and eye state

Eye state analysis (open or closed) is an important step in fatigue detection. In this study, an efficient algorithm for eye state detection is proposed. At first, a new face detection method is presented for noisy images that finds the face area in the input image well. Then, novel algorithms for detection of eye region and eye state are introduced. The performance of the proposed method is evaluated on four different databases namely FERET, Aberdeen, IMM and CVL which contain more than 5700 images with different descents, positions, light conditions and glasses. The results show that the new method achieves more accuracy rate than the previously presented algorithms, while it does not need training data and is also computationally efficient.

Real-time vision-based eye state detection for driver alertness monitoring

This paper presents a real-time vision-based system to detect the eye state. The system is implemented with a consumer-grade computer and an uncalibrated web camera with passive illumination. Previously established similarity measures between image regions, feature selection algorithms, and classifiers have been applied to achieve vision-based eye state detection without introducing a new methodology. From many different extracted data of 1,293 pair of eyes images and 2,322 individual eye images, such as histograms, projections, and contours, 186 similarity measures with three eye templates were computed. Two feature selection algorithms, the $$ J_{5} (\xi ) $$ criterion and sequential forward selection, and two classifiers, multi-layer perceptron and support vector machine, were intensively studied to select the best scheme for pair of eyes and individual eye state detection. The output of both the selected classifiers was combined to optimize eye state monitoring in video sequences. We tested the system with videos with different users, environments, and illumination. It achieved an overall accuracy of 96.22 %, which outperforms previously published approaches. The system runs at 40 fps and can be used to monitor driver alertness robustly.