Lip Detection Research Articles

ViLiDEx- A Lip Extraction Algorithm for Lip Reading

Technology is evolving at an immense speed every day. In the lap of technology, computer vision and machine learning are also growing fast. Many real time applications are running without human interaction just because of Computer vision and machine learning. In this paper, we are using computer vision and machine learning for lip feature extraction for Gujarati language. For this task we have created dataset GVLetters for Gujarati alphabets. We have taken videos of 24 speakers for 33 alphabets of Guajarati language. Face landmark algorithm from dlib is used for deriving ViLiDEx (Vibhavari’s algorithm for Lip Detection and Extraction). ViLiDEx is applied for 24 speakers and 5 alphabets from each class (Guttural, Palatal, Retroflex, Dental and Labial). This algorithm calculates total number of frames for each speaker, keep 20/25 frames as a dataset and removes extra frames. Depending on number of frames, frame numbers divisible by prime numbers are chosen for removal.

Lip Detection and Tracking with Geometric Constraints under Uneven Illumination and Shadows

In the modern era, recent advancement in computer vision has led to emergent attention in lip reading. Indeed, lip-reading is used to understand speech without hearing it, and the process is mentioned as a lip-reading system. To construct an automatic lip-reading system, locating the lip and defining the lip region is essential, especially under different lighting conditions, significantly impacting the robustness of the lip-reading system. Unluckily, in previous studies, lip localization under illumination and shadow consideration has not been well solved. In this paper, we extant a local region-based approach towards the lip-reading system. It consists of four significant parts, firstly detecting/localizing the human face, mouth and lip region of interest in the first video frame. Secondly, apply pre-processing to overwhelmed the inference triggered by illumination effects, shadow and teeth appearance, thirdly create contour line using sixteen key points with geometric constraint and stored the coordinates of these constraints. Finally, track the coordinates of sixteen points in the following frames. The proposed method adapts to the lip movement and is robust in contrast to the appearance of teeth, shadows, and low contrast environment. Extensive experiments show encouraging results and the proposed method's effectiveness compared to the existing methods.

A Survey of Research on Lipreading Technology

Although automatic speech recognition (ASR) technology is mature, there are still some unsolved problems, such as how to accurately identify what the speaker is saying in a noisy environment. Lipreading is a visual speech recognition technology that recognizes the speech content based on the motion characteristics of the speaker's lips without speech signals. Therefore, lipreading can detect the speaker's content in a noisy environment, even without a voice signal. This article summarizes the main research from traditional methods to deep learning methods on lipreading. Traditional lipreading methods are mainly discussed from three aspects: lip detection and extraction, lip feature extraction, and classification. Traditional feature extraction methods focus on handmade features, which are, however, not very reliable under unconstrained conditions. In recent years, traditional lipreading methods have been gradually replaced by deep learning methods. The advantage of deep learning methods is that they can learn the best features from large databases. This article analyzes typical deep learning methods in detail according to their structural characteristics, and lists existing lipreading databases, including their detailed information and the methods applied to these databases. Finally, the problems and challenges of current lipreading methods are discussed, and the future research direction has prospected.

DeepLip: block-based lip pixel detection by deep neural networks

ABSTRACTThis paper presents an effective lip pixel detection method based on blocks and deep neural networks. Since only-rough localization of a pair of lips is a trivial task, we use a rectangle that loosely bounds two lips as an input region of interest for lip detection. For each pixel in the rectangle region we generate a block whose center is at the pixel, and the pixel is classified into either a lip or non-lip pixel by exploiting the pixels in the block. Deep neural networks are trained using a sufficient number of labeled blocks obtained from a quite tractable number of labeled images. As a result, lip pixels are detected with high accuracy despite negligible labeling effort. Experimental results demonstrate the effectiveness of the presented method. We show that even single-minute training can outperform the mouth map with the best threshold.

Lip Movement Feature Detection and Classification Methods

Computerized lip reading has been one of the most actively researched areas of computer vision in recent past because of its crime fighting potential and invariance to acoustic environment. However, several factors like fast speech, bad pronunciation, and poor illumination, movement of face, moustaches and beards make lip reading difficult. In present work, we propose a solution for automatic lip contour tracking and recognizing numbers 1-10 of English language spoken by speakers using the information available from lip movements. In this method first, we are detecting Face then in that ROI Lip detection. Lip movements as the only input and without using any speech recognition system in parallel. The approach used in this work is found to significantly solve the purpose of lip reading when size of database is small. In this paper also compare LBP and HOG feature with Support Vector Machine Classification. The system is use for the modern sign in PASSWORD systems.

Robust lip detection based on histogram of oriented gradient features and convolutional neural network under effects of light and background

Detection of the lip area is essential pre-processing for several applications, such as lip reading and visual information services. In this paper, we propose a lip detection method that finds the lip area in an image using the histogram of oriented gradient (HOG) features and convolutional neural network (CNN). We find the face area from an input image, divide the face image in half, and apply sliding window detection to the bottom half of the image. We obtain the HOG feature vector from the image that corresponds to the window, and use it as the input to a pre-trained support vector machine (SVM). HOG and SVM are used for coarse detection. If SVM determines that the image is not the lip, we reapply sliding window detection. Otherwise, the image is used as input to CNN, which is employed for fine detection and to determine whether the image is the lip. If CNN determines that the image is the lip, we apply canny edge detection to the image to obtain the mouth contour. We use MATLAB to confirm the effectiveness of our method, and can find the mouth area with over 94% accuracy and over 98% precision.

Lip Detection and Lip Geometric Feature Extraction using Constrained Local Model for Spoken Language Identification using Visual Speech Recognition

Background/Objectives: The aim of our research is to guess the language of spoken utterance by using the cues from visual speech recognition i.e. from movement of lips. The first step towards this task is to detect lips form face image and then to extract various geometric features of lip shape in order to guess the utterance. Methods/Statistical Analysis: This paper presents the methodology for detecting lips from face images using constrained local model (CLM) and then extracting the geometric features of lip shape. The two steps involved in lip detection are CLM model building and CLM search. For extracting lip geometric features, twenty feature points are defined on lips and lip height, width, area are defined using these twenty feature points. Findings: CLM model is build using images from FGnet Talking face video database and tested using images from FGnet Talking face video database and also using other images. The detection accuracy is more for FGnet images as compare to other images. Feature vector defining the lip shape consists of geometric parameters like height, width and area of inner and outer lip contours. Feature vector is calculated for all test images after detecting lips from face image. So the error in detecting lips leads to the error in feature vector. This indicates the speaker dependency of visual speech recognition systems. Application/Improvements: The proposed approach is useful in visual speech recognition for lip detection and feature extraction. Minimizing the speaker dependency and generalizing the approach should be considered for further improvements.

Face Detection and Recognition in Color Images under Matlab

In this paper we describe our implementation of algorithms for face detection and recognition in color images under Matlab. For face detection, we trained a feedforward neural network to perform skin segmentation, followed by the eyes detection, face alignment, lips detection and face delimitation. The eyes were detected by analyzing the chrominance and the angle between neighboring pixels and, then, the results were used to perform face alignment. The lips were detected based on the analysis of the Red color component intensity in the lower face region. Finally, the faces were delimited using the eyes and lips positions. The face recognition involved a classifier that used the standard deviation of the difference between color matrices of the faces to identify the input face. The algorithms were run on Faces 1999 dataset. The proposed method achieved 96.9%, 89% and 94% correct detection rate of face, eyes and lips, respectively. The correctness rate of the face recognition algorithm was 70.7%.

Techniques for Skin, Face, Eye and Lip Detection using Skin Segmentation in Color Images

Face detection is a challenging and important problem in Computer Vision. In most of the face recognition systems, face detection is used in order to locate the faces in the images. There are different methods for detecting faces in images. One of these methods is to try to find faces in the part of the image that contains human skin. This can be done by using the information of human skin color. Skin detection can be challenging due to factors such as the differences in illumination, different cameras, ranges of skin colors due to different ethnicities, and other variations. Neural networks have been used for detecting human skin. Different methods have been applied to neural networks in order to increase the detection rate of the human skin. The resulting image is then used in the detection phase. The resulting image consists of several components and in the face detection phase, the faces are found by just searching those components. If the components consist of just faces, then the faces can be detected using correlation. Eye and lip detections have also been investigated using different methods, using information from different color spaces. The speed of face detection methods using color images is compared with other face detection methods.

Driver Fatigue Recognition using Skin Color Modeling

In Driver’s fatigue is a major safety concern in transportation system, because driver drowsiness and distraction have been casual factor for the large number of road accident. Fatigue reduces the driver’s perception level and decision making ability, which responsible for serious road accident. Around 22%-24% of car crash occurred by driver drowsiness. There is a way to reduce these accidents by monitoring drives fatigue and driving behaviors at the driving time by alerting the drivers, while the drivers are drowse. Face detection, eyes state measurement, lip detection, yawing detection, head tilting detection are the major visual facial symptoms for the driver fatigue detection. In this paper a modern assistive frame work has been introduced, which detected driver drowsiness based on visual features measurement. The goal of this paper has been monitored the driver driving behaviors, to detect the visual facial symptoms for safe driving in the road. Facial features symptoms have been monitored by two cameras. To detect driver distraction, the proposed algorithm has been experimented the facial fatigue expression, head tilting and lane departure. Experimental result of the proposed method has been compared with the existing methods. The experimental results show that, the proposed algorithm has good accuracy and reliable performance to reduce the road accident than the existing methods. The average accuracy of the proposed method is 92.44%.

Phoneme and Viseme based Approach for Lip Synchronization

Phoneme to Viseme mapping has great application in Visual Speech Recognition, Lip Synchronization, Talking Head Applications, movies, news reading, and film industries. Lot of work has been done in area of various face component detection and recognition. Apart from eye detection, ear detection, iris detection etc, lip tracking and lip detection is one of the favourite topics for researchers. Various algorithms and techniques have been implemented so far to achieve better and better performance. Normalized RGB colour scheme, HSV colour model, Lip detection using HUE segmentation and many more techniques have been implemented and are in the boom. All methods are having their own pros and cons. We are aiming to extract out phonemes from speech as well as we extract visual feature i.e. visemes from face by using hue and saturation values. The reason behind the selection of this algorithm is that, it performs well under various illumination conditions, which is the one of the dimension of difficulty in the area of lip detection. We are aiming to carry out the work on in-house database with varying lighting and noisy conditions.

색상 군집화를 이용한 입술탐지 알고리즘

본 논문에서는 색상 군집화를 이용한 입술탐지 알고리즘을 제안한다. RGB 색상 모델로 주어진 입력영상에서 AdaBoost 알고리즘을 이용하여 얼굴영역을 추출한 후, 얼굴영역을 Lab 컬러 모델로 변환한다. Lab 컬러 모델에서 a 성분은 입술과 유사한 색상을 잘 표현할 수 있는 반면 b 성분은 입술의 보색을 표현할 수 있기 때문에 Lab 컬러로 표현된 얼굴영역에서 a와 b 성분을 기준으로 최단 이웃(nearest neighbour) 군집화 알고리즘을 이용하여 피부 영역을 분리한 후, K-means 색상 군집화를 통해 입술 후보 영역을 추출하고, 마지막으로 기하학적 특징을 이용하여 최종적인 입술영역을 탐지하였다. 실험 결과는 제안된 방법이 강건하게 입술을 탐지함을 보인다. In this paper, we propose a robust lip detection algorithm using color clustering. At first, we adopt AdaBoost algorithm to extract facial region and convert facial region into Lab color space. Because a and b components in Lab color space are known as that they could well express lip color and its complementary color, we use a and b component as the features for color clustering. The nearest neighbour clustering algorithm is applied to separate the skin region from the facial region and K-Means color clustering is applied to extract lip-candidate region. Then geometric characteristics are used to extract final lip region. The proposed algorithm can detect lip region robustly which has been shown by experimental results.

Measuring lips control on flute-like instruments using active vision

Flute-like instruments operate under a feedback system between an air jet and a resonator. The characteristics of the air jet and its interaction with the resonator are defined by construction in some flute-like instruments such as the organ pipes, or completely determined by the musician’s control like in the Shakuhachi, panpipe, or transverse flute. In this article, a 3D lips detection system based on active vision using laser grid triangulation has been designed and implemented. Simple musical gestures from transverse flute players have been measured and analyzed, outlying the strategies used to coordinate the parameters available, such as lips to labium distance, jet speed, and jet shape, to orchestrate an adequate sound. The analysis is presented in a non-dimensional representation capable of generalizing to other members of the flute family. The measurement system developed is compared with previous strategies and reveals a promising tool to further understand the complexity of the human control in this popular family of instruments.

Real-time lip reading system for isolated Korean word recognition

This paper proposes a real-time lip reading system (consisting of a lip detector, lip tracker, lip activation detector, and word classifier), which can recognize isolated Korean words. Lip detection is performed in several stages: face detection, eye detection, mouth detection, mouth end-point detection, and active appearance model (AAM) fitting. Lip tracking is then undertaken via a novel two-stage lip tracking method, where the model-based Lucas-Kanade feature tracker is used to track the outer lip, and then a fast block matching algorithm is used to track the inner lip. Lip activation detection is undertaken through a neural network classifier, the input for which being a combination of the lip motion energy function and the first dominant shape feature. In the last step, input words are defined and recognized by three different classifiers: HMM, ANN, and K-NN. We combine the proposed lip reading system with an audio-only automatic speech recognition (ASR) system to improve the word recognition performance in the noisy environments. We then demonstrate the potential applicability of the combined system for use within hands free in-vehicle navigation devices. Results from experiments undertaken on 30 isolated Korean words using the K-NN classifier at a speed of 15fps demonstrate that the proposed lip reading system achieves a 92.67% word correct rate (WCR) for person-dependent tests, and a 46.50% WCR for person-independent tests. Also, the combined audio-visual ASR system increases the WCR from 0% to 60% in a noisy environment.

AAM 기반 얼굴 표정 인식을 위한 입술 특징점 검출 성능 향상 연구

AAM(Active Appearance Model)은 PCA(Principal Component Analysis)를 기반으로 객체의 형태(shape)와 질감(texture) 정보에 대한 통계적 모델을 통해 얼굴의 특징점을 검출하는 알고리즘으로 얼굴인식, 얼굴 모델링, 표정인식과 같은 응용에 널리 사용되고 있다. 하지만, AAM알고리즘은 초기 값에 민감하고 입력영상이 학습 데이터 영상과의 차이가 클 경우에는 검출 에러가 증가되는 문제가 있다. 특히, 입을 다문 입력얼굴 영상의 경우에는 비교적 높은 검출 정확도를 나타내지만, 사용자의 표정에 따라 입을 벌리거나 입의 모양이 변형된 얼굴 입력 영상의 경우에는 입술에 대한 검출 오류가 매우 증가되는 문제점이 있다. 이러한 문제점을 해결하기 위해 본 논문에서는 입술 특징점 검출을 통해 정확한 입술 영역을 검출한 후에 이 정보를 이용하여 AAM을 수행함으로써 얼굴 특징점 검출 정확성을 향상시키는 방법을 제안한다. 본 논문에서는 AAM으로 검출한 얼굴 특징점 정보를 기반으로 초기 입술 탐색 영역을 설정하고, 탐색 영역 내에서 Canny 경계 검출 및 히스토그램 프로젝션 방법을 이용하여 입술의 양 끝점을 추출한 후, 입술의 양 끝점을 기반으로 재설정된 탐색영역 내에서 입술의 칼라 정보와 에지 정보를 함께 결합함으로써 입술 검출의 정확도 및 처리속도를 향상시켰다. 실험결과, AAM 알고리즘을 단독으로 사용할 때보다, 제안한 방법을 사용하였을 경우 입술 특징점 검출 RMS(Root Mean Square) 에러가 4.21픽셀만큼 감소하였다. AAM(Active Appearance Model) is an algorithm to extract face feature points with statistical models of shape and texture information based on PCA(Principal Component Analysis). This method is widely used for face recognition, face modeling and expression recognition. However, the detection performance of AAM algorithm is sensitive to initial value and the AAM method has the problem that detection error is increased when an input image is quite different from training data. Especially, the algorithm shows high accuracy in case of closed lips but the detection error is increased in case of opened lips and deformed lips according to the facial expression of user. To solve these problems, we propose the improved AAM algorithm using lip feature points which is extracted based on a new lip detection algorithm. In this paper, we select a searching region based on the face feature points which are detected by AAM algorithm. And lip corner points are extracted by using Canny edge detection and histogram projection method in the selected searching region. Then, lip region is accurately detected by combining color and edge information of lip in the searching region which is adjusted based on the position of the detected lip corners. Based on that, the accuracy and processing speed of lip detection are improved. Experimental results showed that the RMS(Root Mean Square) error of the proposed method was reduced as much as 4.21 pixels compared to that only using AAM algorithm.

모바일 환경에서의 시각 음성인식을 위한 눈 정위 기반 입술 탐지에 대한 연구

음성 인식 기술은 편리한 삶을 추구하는 요즘 추세에 HMI를 위해 매력적인 기술이다. 음성 인식기술에 대한 많은 연구가 진행되고 있으나 여전히 잡음 환경에서의 성능은 취약하다. 이를 해결하기 위해 요즘은 청각 정보 뿐 아니라 시각 정보를 이용하는 시각 음성인식에 대한 연구가 활발히 진행되고 있다. 본 논문에서는 모바일 환경에서의 시각 음성인식을 위한 입술의 탐지 방법을 제안한다. 시각 음성인식을 위해서는 정확한 입술의 탐지가 필요하다. 우리는 입력 영상에서 입술에 비해 보다 찾기 쉬운 눈을 이용하여 눈의 위치를 먼저 탐지한 후 이 정보를 이용하여 대략적인 입술 영상을 구한다. 구해진 입술 영상에 K-means 집단화 알고리듬을 이용하여 영역을 분할하고 분할된 영역들 중 가장 큰 영역을 선택하여 입술의 양 끝점과 중심을 얻는다. 마지막으로, 실험을 통하여 제안된 기법의 성능을 확인하였다. Automatic speech recognition(ASR) is attractive technique in trend these day that seek convenient life. Although many approaches have been proposed for ASR but the performance is still not good in noisy environment. Now-a-days in the state of art in speech recognition, ASR uses not only the audio information but also the visual information. In this paper, We present a novel lip detection method for visual speech recognition in mobile environment. In order to apply visual information to speech recognition, we need to extract exact lip regions. Because eye-detection is more easy than lip-detection, we firstly detect positions of left and right eyes, then locate lip region roughly. After that we apply K-means clustering technique to devide that region into groups, than two lip corners and lip center are detected by choosing biggest one among clustered groups. Finally, we have shown the effectiveness of the proposed method through the experiments based on samsung AVSR database.

Lip detection by the use of neural networks

Lip detection is used in many applications such as face detection and lip reading. In this article, a method for lip detection in color images in a normalized RGB color scheme is presented. In this method, MLP neural networks are used to perform lip detection on segmented skin regions. Several combinations of chrominance components of the normalized RGB color space were used as the input to the neural networks. Two methods were used for obtaining the normalized RGB components from the RGB color scheme. These are called the maximum and intensity normalization methods, respectively. The method was tested on two Asian databases. The number of neurons in the hidden layer was determined by using a modified network-growing algorithm. It was found that the pixel intensity normalization method gave lower lip detection error than the maximum intensity normalization method regardless of the database used, and for most of the combinations of chrominance components. In addition, the combination of the g and r/g chrominance components gave the lowest lip detection error when the pixel intensity normalization method was used for both databases. The effects of the scale and facial expression on lip detection was also studied. It was found that the lip detection error decreased as the scale factor increased. As for facial expression, a laughing facial expression gave the highest lip detection error, followed by smiling and neutral expressions.

High Speed Genetic Lips Detection by Dynamic Search Domain Control

In this paper, high-speed size and orientation invariant lips detection of a talking person in an active scene using template matching and genetic algorithms is proposed. As part of the objectives, we also try to acquire numerical parameters to represent the lips. The information is very important for many applications, where high performance is required, such as audio-visual speech recognition, speaker identification systems, robot perception and personal mobile devices interfaces. The difficulty in lips detection is mainly due to deformations and geometric changes of the lips during speech and the active scene by free camera motion. In order to enhance the performance in speed and accuracy, initially, the performance is improved on a single still image, that is, the base of video processing. Our proposed system is based on template matching using genetic algorithms (GA). Only one template is prepared per experiment. The template is the closed mouth of a subject, because the application is for personal devices. In our previous study, the main problem was trade-off between search accuracy and search speed. To overcome this problem, we use two methods: scaling window and dynamic search domain control (SD-Control). We therefore focus on the population size of the GA, because it has a direct effect on search accuracy and speed. The effectiveness of the proposed system is demonstrated by performing computer simulations. We achieved a lips detection accuracy of 91.33% at an average processing time of 33.70 milliseconds per frame.

Downsized Genetic Algorithm by Automatic Search Domain Control for Lips Detection

Downsized Genetic Algorithm by Automatic Search Domain Control for Lips Detection

唇検出ニューラルネットワークと肌色識別ニューラルネットワークによるカラー画像中からの顔領域の検索法

唇検出ニューラルネットワークと肌色識別ニューラルネットワークによるカラー画像中からの顔領域の検索法