Signer Adaptation Research Articles

Subunit sign modeling framework for continuous sign language recognition

A new framework named three subunit sign modeling is introduced for automatic sign language recognition. This works on continuous video sequences consisting of isolated words, signed sentences under different signer variations and illuminations. Three major issues of automatic sign language recognition is addressed namely: (i) importance of discriminative feature extraction and selection (ii) handling epenthesis movements and segmentation ambiguities (iii) automatic recognition of large vocabulary sign sentences and signer adaptation in a single subunit sign modeling framework. The proposed work has been evaluated and experimented subjectively and quantitatively with real-time signing videos gathered from different corpora and different sign languages. The results of the experiments have proven that the proposed subunit sign modeling framework remains scalable while increasing the sign vocabulary. Further, the approach is more reliable and efficient enough to adapt to real-time constraints and signer independence.

Lexicon-free fingerspelling recognition from video: Data, models, and signer adaptation

We study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL. Recognizing fingerspelling is challenging for a number of reasons: it involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work we collect and annotate a new data set of continuous fingerspelling videos, compare several types of recognizers, and explore the problem of signer variation. Our best-performing models are segmental (semi-Markov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 92% letter accuracy. The multi-signer setting is much more challenging, but with neural network adaptation we achieve up to 83% letter accuracies in this setting.

Unsupervised adaptive sign language recognition based on hypothesis comparison guided cross validation and linguistic prior filtering

Abstract Signer adaptation is important for sign language recognition systems because a fixed system cannot perform well on all kinds of signers. In supervised signer adaptation, the labeled adaptation data must be collected explicitly. To skip the data collecting process in signer adaptation, we propose a novel unsupervised adaptation method, namely the hypothesis comparison guided cross validation method. The method not only addresses the problem of the overlap between the data set to be labeled and the data set for adaptation, but also employs an additional hypothesis comparison step to decrease the noise rate of the adaptation data set. We also utilize linguistic prior knowledge to down sample the adaptation data list to further decrease the noise rate. To evaluate the effectiveness of the proposed method, the CASIIE-SL-Database is formed, which is the first specialized data set for unsupervised signer adaptation to the best of our knowledge. Experimental results show that the proposed method can achieve relative word error rate reductions of 3.93% and 4.05% respectively compared with self-teaching method and cross validation method. Though the method is proposed for signer adaptation, it can also be applied to speaker adaptation and writer adaptation directly.

A multi-class classification strategy for Fisher scores: Application to signer independent sign language recognition

Fisher kernels combine the powers of discriminative and generative classifiers by mapping the variable-length sequences to a new fixed length feature space, called the Fisher score space. The mapping is based on a single generative model and the classifier is intrinsically binary. We propose a multi-class classification strategy that applies a multi-class classification on each Fisher score space and combines the decisions of multi-class classifiers. We experimentally show that the Fisher scores of one class provide discriminative information for the other classes as well. We compare several multi-class classification strategies for Fisher scores generated from the hidden Markov models of sign sequences. The proposed multi-class classification strategy increases the classification accuracy in comparison with the state of the art strategies based on combining binary classifiers. To reduce the computational complexity of the Fisher score extraction and the training phases, we also propose a score space selection method and show that, similar or even higher accuracies can be obtained by using only a subset of the score spaces. Based on the proposed score space selection method, a signer adaptation technique is also presented that does not require any re-training.

An Adaptive Hand-Talking System for the Hearing Impaired

An adaptive Chinese hand-talking system is presented in this paper. By analyzing the 3 data collecting strategies for new users, the adaptation framework including supervised and unsupervised adaptation methods is proposed. For supervised adaptation, affinity propagation (AP) is used to extract exemplar subsets, and enhanced maximum a posteriori / vector field smoothing (eMAP/VFS) is proposed to pool the adaptation data among different models. For unsupervised adaptation, polynomial segment models (PSMs) are used to help hidden Markov models (HMMs) to accurately label the unlabeled data, then the ”labeled” data together with signerindependent models are inputted to MAP algorithm to generate signer-adapted models. Experimental results show that the proposed framework can execute both supervised adaptation with small amount of labeled data and unsupervised adaptation with large amount of unlabeled data to tailor the original models, and both achieve improvements on the performance of recognition rate. Keywords—sign language recognition, signer adaptation, eMAP/VFS, polynomial segment model.