Toward computational understanding of sign language

Abstract

In this paper, we describe some of the current issues in computational sign language processing. Despite the seeming similarities between computational spoken language and sign language processing, signed languages have intrinsic properties that pose some very difficult problems. These include a high level of simultaneous actions, the intersection between signs and gestures, and the complexity of modeling grammatical processes. Additional problems are posed by the difficulties that computers face in extracting reliable information on the hands and the face from video images. So far, no single research group or company has managed to tackle all the hard problems and produced a real working system for analysis and recognition. We present a summary of our research into sign language recognition and how it interacts with sign language linguistics. We propose solutions to some of the aforementioned problems, and also discuss what problems are still unsolved. In addition, we summarize the current state of the art in our sign language recognition and facial expression analysis frameworks.