Sign Language Data Research Articles

Maximum margin hidden Markov models for sequence classification

Discriminative learning methods are known to work well in pattern classification tasks and often show benefits compared to generative learning. This is particularly true in case of model mismatch, i.e. the model cannot represent the true data distribution. In this paper, we derive discriminative maximum margin learning for hidden Markov models (HMMs) with emission probabilities represented by Gaussian mixture models (GMMs). The focus is on single-label sequence classification where the margin objective is specified by the probabilistic gap between the true class and the most competing class. In particular, we use the extended Baum–Welch (EBW) framework to optimize this probabilistic margin embedded in a hinge loss function. Approximations of the margin objective and the derivatives are necessary. In the experiments, we compare maximum margin HMMs to generative maximum likelihood and discriminative conditional log-likelihood (CLL) HMM training. We present results of classifying trajectories of handwritten characters, Australian sign language data, digits of speech data and UCR time-series data. Maximum margin HMMs outperform in many cases CLL-HMMs. Furthermore, maximum margin HMMs achieve a significantly better performance than generative maximum likelihood HMMs.

A First for the Field of Sign Language Research

A First for the Field of Sign Language Research Research Methods in Sign Language Studies: A Practical Guide. Eleni Orfanidou, Bencie Woll, & Gary Morgan (Eds.). John Wiley & Sons, 2015. 384 pages. $35.99 (e-book), $44.95 (paperback), $99.95 (hardcover).Enactment of the Education Sciences Reform Act of 2002 bolstered education research by leading to the establishment of the Institute of Education Sciences (IES). One of the rationales for the founding of the IES was to support the quality and integrity of education research. The What Works Clearinghouse, a department within the IES, evaluates instructional programs and practices as well as education policies and research. One reason such institutions are needed is the too common practice of education researchers making causal conclusions based on nonmanipulated correlational studies (Robinson, Levin, Thomas, Pituch, & Vaughn, 2007). Researchers must be clear about the potential implications of their data because their findings should inform instruction. Education researchers, particularly inexperienced researchers, need guidance and guidelines for conducting and reporting research clearly and accurately because their findings affect classroom practices.Research handbooks are one highly useful tool for researchers, and editors Eleni Orfanidou, Bencie Woll, and Gary Morgan (and nearly 40 contributing authors) have recently provided the field of sign language research with its first: Research Methods in Sign Language Studies: A Practical Guide. Although the editors are neither explicitly seeking to address the need for clearer standards of research in this book nor directly gearing it toward education researchers, this volume serves both objectives. By providing summaries of the methodologies in sign language research in addition to instruction regarding social, linguistic, psycholinguistic, and neurolinguistic research and signed languages, the book provides guidelines on how research is being conducted in the field of sign language research that should be used to inform future research. Additionally, the field of deaf education research does not have a handbook on conducting research. Certainly, this text cannot serve as a comprehensive handbook for deaf education researchers-nor was it intended to. However, deaf education researchers do conduct sign language research to learn about language development and language usage of Deaf individuals, often focusing on preschool or school-age participants. Therefore, this book has the potential to serve a wide range of researchers.The book is divided into five parts. Part I calls attention to ethical issues relevant to sign language research. These ethical issues include implementing research methods that are Deaf-friendly by viewing members of the Deaf community as gatekeepers and offering consent forms that have been translated into participants' native signed language. Guidelines are offered on upholding ethical considerations for participants in developing countries in the same way researchers would in developed countries. Part I concludes with a discussion of good practices in field research. These include involving the Deaf community in the research; learning sign language, as well as the culture and customs of the participants; and sharing the findings through signed videos on Deaf-friendly websites or by some other means preferred by the target community.Part II focuses on methods for collecting sign language data. The authors highlight the complications of collecting data and offer sage advice, such as the best angles from which to set up video cameras for conducting interviews using a sign language-knowledge that would otherwise only be acquired through experience. There are also discussions of different methods for transcribing, analyzing, and measuring the data. In regard to transcribing, the authors discuss a multitude of methods, including glossing, phonetic transcription, Stokoe notation, and the Berkeley Transcription System. …

Building a Swiss German Sign Language avatar with JASigning and evaluating it among the Deaf community

This paper reports on the development of a system that translates German train announcements of the Swiss Federal Railways (Schweizerische Bundesbahnen, SBB) into Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) in real time and displays the result via an avatar. The system used to animate the avatar is called JASigning. Deliverables of the projects during which JASigning was developed are the main source of documentation for the system along with notes on the Web site. Not all planned features have been fully implemented: some because they are used very infrequently; others because there is insufficient linguistic research on which to base an implementation. The team of hearing and Deaf researchers identified the avatar functionality needed for the project and built a first version of the avatar. A focus group study with seven Deaf signers was then carried out to obtain feedback on how to further improve the avatar. This paper reports the evaluation results. It also discusses the workarounds introduced for features that were not yet directly available in the JASigning system. These features were not specific to train announcements. Hence, knowledge of how to achieve their designated effects in JASigning can be useful to persons working with other types of sign language data as well.

Context Shift and Indexical Variables in Sign Languages

No abstract.

Philosophy of gustatory perception strategy for hand gesture recognition

Purpose – Nature is the single and most complex system that has been always studied, and no one can compete Mother Nature, but we can learn from her, by many new methodologies through biology. The paper aims to discuss this issue. Design/methodology/approach – In this paper, being inspired by the mechanism through which our Mother Nature handling human taste, a proposed model for clustering and classification hand gesture is introduced based on human taste controlling strategy. Findings – The model can extract information from measurement data and handling it as the structure of tongue and the nervous systems of human taste recognition. Originality/value – The efficiency of proposed model is demonstrated experimentally on classifying the sign language data set; in the high recognition accuracy obtained for numbers of ASL was 95.52 percent.

Regional Variation in German Sign Language: The Role of Schools (Re-)Visited

Regional dialects in many sign languages have traditionally been linked to schools for deaf students. Focusing on three such schools in Saxony (southeast Germany), our study for the first time examines whether this link is also evident in German Sign Language (Deutsche Gebärdensprache, DGS). We compare and contrast sign language data from deaf adults and children who used to attend or are still being educated in these schools, respectively. Our study shows that the effect size between choice of variant and school attended is notably weaker in the child language data than in the adult data. Considering social and policy changes in the educational landscape, our research raises the question of whether schools for deaf students still function as cradles of German Sign Language dialects.

The Effect of New Technologies on Sign Language Research

This paper first reviews the fairly established ways of collecting sign language data. It then discusses the new technologies available and their impact on sign language research, both in terms of how data is collected and what new kinds of data are emerging as a result of technology. New data collection methods and new kinds of data are illustrated with three specific case studies.

Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures

In this paper, we compare so-called “classifier” constructions in signed languages (which we refer to as “depicting constructions”) with comparable iconic gestures produced by non-signers. We show clear correspondences between entity constructions and observer viewpoint gestures on the one hand, and handling constructions and character viewpoint gestures on the other. Such correspondences help account for both lexicalisation and de-lexicalisation processes in signed languages and how these processes are influenced by viewpoint. Understanding these processes is crucial when coding and annotating natural sign language data.

Kernel-based representation for 2D/3D motion trajectory retrieval and classification

This paper proposes a novel kernel-space representation for motion trajectories. Contrasted to most trajectory representation methods in the literature, our method is more generic in the sense that it can be applied to either 2D or 3D trajectories. In the proposed method, a trajectory is firstly projected by the Kernel Principal Component Analysis (KPCA) which can be considered as an implicit mapping to a much higher-dimensional feature space. The high dimensionality can effectively improve the accuracy in recognizing motion trajectories. Then, Nonparametric Discriminant Analysis (NDA) is used to extract the most discriminative features from the KPCA feature space. The synergistic effect of KPCA and NDA leads to better class separability and makes the proposed trajectory representation a more powerful discriminator. The experimental validation of the proposed method is conducted on the Australian Sign Language (ASL) data set. The results show that our method performs significantly better, in both trajectory classification and retrieval, than the state-of-the-art techniques.

Donkey anaphora: the view from sign language (ASL and LSF)

There are two main approaches to the problem of donkey anaphora (e.g. If John owns a donkey, he beats it). Proponents of dynamic approaches take the pronoun to be a logical variable, but they revise the semantics of quantifiers so as to allow them to bind variables that are not within their syntactic scope. Older dynamic approaches took this measure to apply solely to existential quantifiers; recent dynamic approaches have extended it to all quantifiers. By contrast, proponents of E-type analyses take the pronoun to have the semantics of a definite description (with it ≈ the donkey, or the donkey that John owns). While competing accounts make very different claims about the patterns of coindexation that are found in the syntax, these are not morphologically realized in spoken languages. But they are in sign language, namely through locus assignment and pointing. We make two main claims on the basis of ASL and LSF data. First, sign language data favor dynamic over E-type theories: in those cases in which the two approaches make conflicting predictions about possible patterns of coindexation, dynamic analyses are at an advantage. Second, among dynamic theories, sign language data favor recent ones because the very same formal mechanism is used irrespective of the indefinite or non-indefinite nature of the antecedent. Going beyond this debate, we argue that dynamic theories should allow pronouns to be bound across negative expressions, as long as the pronoun is presupposed to have a non-empty denotation. Finally, an appendix displays and explains subtle differences between overt sign language pronouns and all other pronouns in examples involving ‘disjunctive antecedents’, and suggests that counterparts of sign language loci might be found in spoken language.

Handling Movement Epenthesis and Hand Segmentation Ambiguities in Continuous Sign Language Recognition Using Nested Dynamic Programming

We consider two crucial problems in continuous sign language recognition from unaided video sequences. At the sentence level, we consider the movement epenthesis (me) problem and at the feature level, we consider the problem of hand segmentation and grouping. We construct a framework that can handle both of these problems based on an enhanced, nested version of the dynamic programming approach. To address movement epenthesis, a dynamic programming (DP) process employs a virtual me option that does not need explicit models. We call this the enhanced level building (eLB) algorithm. This formulation also allows the incorporation of grammar models. Nested within this eLB is another DP that handles the problem of selecting among multiple hand candidates. We demonstrate our ideas on four American Sign Language data sets with simple background, with the signer wearing short sleeves, with complex background, and across signers. We compared the performance with Conditional Random Fields (CRF) and Latent Dynamic-CRF-based approaches. The experiments show more than 40 percent improvement over CRF or LDCRF approaches in terms of the frame labeling rate. We show the flexibility of our approach when handling a changing context. We also find a 70 percent improvement in sign recognition rate over the unenhanced DP matching algorithm that does not accommodate the me effect.

Quantifiers and Variables: Insights from Sign Language (ASL and LSF)

In standard logical systems, quantifiers and variables are essential to express complex relations among objects. Natural language has expressions that have an analogous function: some noun phrases play the role of quantifiers (e.g. every man), and some pronouns play the role of variables (e.g. him, as in Every man likes people who admire him). Since the 1980’s, there has been a vibrant debate in linguistics about the way in which pronouns come to depend on their antecedents. According to one view, natural language is governed by a ‘dynamic’ logic which allows for dependencies that are far more flexible than those of standard (classical) logic. According to a competing view, the treatment of variables in classical logic does not have to be fundamentally revised to be applied to natural language. While the debate centers around the nature of the formal links that connect pronouns to their antecedents, these links are not overtly expressed in spoken language, and the debate has remained open. In sign language, by contrast, the connection between pronouns and their antecedents is often made explicit by pointing. We argue that data from French and American Sign Language provide crucial evidence for the dynamic approach over one of its main classical competitors; and we explore further sign language data that can help choose among competing dynamic analyses.

Understanding Video Events: A Survey of Methods for Automatic Interpretation of Semantic Occurrences in Video

Understanding video events, i.e., the translation of low-level content in video sequences into high-level semantic concepts, is a research topic that has received much interest in recent years. Important applications of this paper include smart surveillance systems, semantic video database indexing, and interactive systems. This technology can be applied to several video domains including airport terminal, parking lot, traffic, subway stations, aerial surveillance, and sign language data. In this paper, we identify the two main components of the event understanding process: abstraction and event modeling. Abstraction is the process of molding the data into informative units to be used as input to the event model. Due to space restrictions, we will limit the discussion on the topic of abstraction. See the study by Lavee et al. (Understanding video events: A survey of methods for automatic interpretation of semantic occurrences in video, Technion-Israel Inst. Technol., Haifa, Israel, Tech. Rep. CIS-2009-06, 2009) for a more complete discussion. Event modeling is devoted to describing events of interest formally and enabling recognition of these events as they occur in the video sequence. Event modeling can be further decomposed in the categories of pattern-recognition methods, state event models, and semantic event models. In this survey, we discuss this proposed taxonomy of the literature, offer a unifying terminology, and discuss popular event modeling formalisms (e.g., hidden Markov model) and their use in video event understanding using extensive examples from the literature. Finally, we consider the application domain of video event understanding in light of the proposed taxonomy, and propose future directions for research in this field.

Effort analysis in signer-independent sign gestures

A dilemma, caused by data variation, exists in research into signer-independent sign language recognition. An effective way to solve this dilemma, and thereby help push the research forward, is to understand sign language from the perspectives of human kinesics and linguistics. This paper, based on the principles of movement observation science, specifically Laban Movement Analysis (LMA), presents a summary of the factors causing sign language data variation, proposes the definition of, and a method for describing, sign language effort elements, and then provides a strategy for standardizing signer-independent sign language data. The standardized data are to be used for training and recognition. The method presented in this paper has been assessed under different experimental conditions, and the results show that the recognition accuracy is greatly increased.

Sharing sign language data online

This article describes how new technological possibilities allow sign language researchers to share and publish video data and transcriptions online. Both linguistic and technological aspects of creating and publishing a sign language corpus are discussed, and standards are proposed for both metadata and transcription categories specific to sign language data. In addition, ethical aspects of publishing video data of signers online are considered, and suggestions are offered for future corpus projects and software tools.

Facial movement analysis in ASL

In the age of speech and voice recognition technologies, sign language recognition is an essential part of ensuring equal access for deaf people. To date, sign language recognition research has mostly ignored facial expressions that arise as part of a natural sign language discourse, even though they carry important grammatical and prosodic information. One reason is that tracking the motion and dynamics of expressions in human faces from video is a hard task, especially with the high number of occlusions from the signers’ hands. This paper presents a 3D deformable model tracking system to address this problem, and applies it to sequences of native signers, taken from the National Center of Sign Language and Gesture Resources (NCSLGR), with a special emphasis on outlier rejection methods to handle occlusions. The experiments conducted in this paper validate the output of the face tracker against expert human annotations of the NCSLGR corpus, demonstrate the promise of the proposed face tracking framework for sign language data, and reveal that the tracking framework picks up properties that ideally complement human annotations for linguistic research.

Development in Signer-Independent Sign Language Recognition and the Ideas of Solving Some Key Problems

Signer-Independent sign language recognition is an unavoidable problem that must be solved in order to promote the practicality of sign language systems. In the signer-independent sign language recognition research, the lack of training data and the signer-independent sign language data variation bring a challenge to the effectivity of the existent research frame. This paper proposes a new research frame for signer-independent sign language recognition, and provides the strategy and ideas to solve the problem. Finding resolution to these problems is significant not only to the research on Chinese sign language recognition but also to other related fields.

Sign Language Processing and the Mirror Neuron System

In this paper we review evidence for frontal and parietal lobe involvement in sign language comprehension and production, and evaluate the extent to which these data can be interpreted within the context of a mirror neuron system for human action observation and execution. We present data from three literatures — aphasia, cortical stimulation, and functional neuroimaging. Generally, we find support for the idea that sign language comprehension and production can be viewed in the context of a broadly-construed frontal-parietal human action observation/execution system. However, sign language data cannot be fully accounted for under a strict interpretation of the mirror neuron system. Additionally, we raise a number of issues concerning the lack of specificity in current accounts of the human action observation/execution system.

Transcription of child sign language

This paper describes some general difficulties in analysing child sign language data with an emphasis on the process of transcription. The particular issue of capturing how signers encode simultaneity in narrative is discussed.

Transcription of child sign language: A focus on narrative

This paper describes some general difficulties in analysing child sign language data with an emphasis on the process of transcription. The particular issue of capturing how signers encode simultaneity in narrative is discussed.