Application of markerless sign interpretation for Malaysian sign language

Abstract

This doctoral thesis presents a framework for machine based interpretation of Malaysian Sign Language (MSL). The goal is to explore and develop necessary core algorithms that are capable of recognizing hand signs, based on the monocular/2D visual input of a low-cost hardware (Digital Webcam), without the assistance of visual-markers on hands. Successful hand sign interpretation with the aforementioned constrained visual system is challenging. Economical hardware devices often provide noisy/cluttered raw data with limited useful information. Non-involvement of visual markers imposes the requirement of additional strategies to disambiguate occlusions and background complexities. Natural variations in hand posture/gesture performances further convolute the recognition task. Addressing these challenges require computationally heavy algorithms in image processing, machine learning and pattern recognition. Thus vision-based sign language translator frameworks have to be multifaceted and versatile, in order to address innumerable practical concerns. In this research, hand postures interpretation is achieved by statistical projection methods. Analysis of high dimensional hand shape data is a complex task. Thus reducing dimensions via known projection methods aids in alleviating some of these complexities. Locality Preserving Projection (LPP) has been employed to accomplish this goal, while Principal Component Analysis (PCA) has been used as the benchmark method. Both methods were extensively tested against a hand posture database of 24 alphabet signs from MSL. As for the hand gesture interpretation, parameter-less Gaussian process based machine learning methods are proposed, as an alternative to current status quo. A variation of the Gaussian latent variable model – Gaussian Process Dynamic Model (GPDM) – has been employed for hand gesture interpretation. This has been comparatively studied against the established Hidden Markov Model (HMM). These machine learning methods were thoroughly tested against a hand gesture database of 64 signs from MSL. Grammatical sentence construction offers both intelligible sentences, and a mean of identifying erroneous sign classifications. In this thesis, a proof-of-concept on: use of Context Free Grammar (CFG) in successful parsing of meaningful sentences based on recognised gesture words, and how it can be extended in identifying (and correcting) erroneous sign classifications, is demonstrated with 92 sentences and the output of GPDM gesture classification. The novelty and contribution of this research: this is the first undertaking of markerless vision-based translation for MSL. Both LPP and GPDM are introduced to the sign language research field as an outcome of this investigation. The preliminary study and demonstration of grammatical sentence construction for translated gesture signs, is the first of its kind to the best of author's knowledge.