The use of tongue protrusion gestures for video-based communication

Abstract

We propose a system that tracks the mouth region in a video sequence and detects the occurrence of a tongue protrusion event. Assuming that the natural location of the tongue is inside the mouth, the tongue protrusion gesture is interpreted as an intentional communication sign that the user wishes to perform a task. The system operates in three steps: (1) mouth template segmentation, in which we initialize one template for the entire mouth, and one template for each of the left and right halves of the mouth; (2) mouth region tracking using the Normalized Correlation Coefficient (NCC), and (3) tongue protrusion event detection and interpretation. We regard the tongue protrusion transition as the event that begins when a minimum part of the tongue starts protruding from the mouth, and which ends when the protrusion is clearly visible. The left and right templates are compared to their corresponding halves for each new mouth image that has been tracked, and a left-NCC and a right-NCC are obtained for each part. By analyzing the NCCs during the tongue protrusion transition time, the left or right position of the protrusion, relative to the center of the mouth, is determined. We analyze our proposed communication method and demonstrate that it adapts easily to different users. The detection of this gesture can be used for instance as a dual-switch hand-free human-computer interface for granting control of a computer.