Sign Language Sentences Research Articles

SENTENCE ARTICULATION IN BRAZILIAN SIGN LANGUAGE – LIBRAS

Articulation of sentences is a grammatical strategy found in any natural language. This phenomenon generates complex constructions which appear in a gradient continuum (parataxis – hypotaxis – enchainment), based on the integration of syntactical, semantical, and pragmatical levels of the language in use. The current research is based on the functionalist perspective of language. The objective of this paper is to describe the process of articulation of sentences in Brazilian Sign Language (Libras). Thereto, we aim at identifying strategies of articulating sentences in Brazilian sign language, taking into account possible complex articulations. We analyze videos taken from conversations via social networks, produced in bidirectional and a-synchronic interaction by three signing deaf of Brazilian sign language. The videos were analyzed on Elan, a multimodal program which enables simultaneous description of audio and/or video. This is an on-going research and here we discuss some initial findings. Among the results, we enhance that Libras employs different discourse strategies of articulating complex sentences: the use of juxtaposition and facial expressions to articulate complex sentences, as well as the use of lexical items to characterize different complex relations, which emphasize the richness and creativity of Brazilian sign language. Thus, complex sentences in Brazilian sign language embodies different discursive strategies to structure parataxis, hypotaxis and enchainment.

Supplemental Material for Effects of Age on American Sign Language Sentence Repetition

Supplemental Material for Effects of Age on American Sign Language Sentence Repetition

A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion

Sign language facilitates communication between hearing impaired peoples and the rest of the society. A number of sign language recognition (SLR) systems have been developed by researchers, but they are limited to isolated sign gestures only. In this paper, we propose a modified long short-term memory (LSTM) model for continuous sequences of gestures or continuous SLR that recognizes a sequence of connected gestures. It is based on splitting of continuous signs into sub-units and modeling them with neural networks. Thus, the consideration of a different combination of sub-units is not required during training. The proposed system has been tested with 942 signed sentences of Indian Sign Language (ISL). These sign sentences are recognized using 35 different sign words. The average accuracy of 72.3% and 89.5% has been recorded on signed sentences and isolated sign words, respectively.

Neural correlates of semantic and syntactic processing in German Sign Language

The study of deaf and hearing native users of signed languages can offer unique insights into how biological constraints and environmental input interact to shape the neural bases of language processing. Here, we use functional magnetic resonance imaging (fMRI) to address two questions: (1) Do semantic and syntactic processing in a signed language rely on anatomically and functionally distinct neural substrates as it has been shown for spoken languages? and (2) Does hearing status affect the neural correlates of these two types of linguistic processing? Deaf and hearing native signers performed a sentence judgement task on German Sign Language (Deutsche Gebärdensprache: DGS) sentences which were correct or contained either syntactic or semantic violations. We hypothesized that processing of semantic and syntactic violations in DGS relies on distinct neural substrates as it has been shown for spoken languages. Moreover, we hypothesized that effects of hearing status are observed within auditory regions, as deaf native signers have been shown to activate auditory areas to a greater extent than hearing native signers when processing a signed language. Semantic processing activated low-level visual areas and the left inferior frontal gyrus (IFG), suggesting both modality-dependent and independent processing mechanisms. Syntactic processing elicited increased activation in the right supramarginal gyrus (SMG). Moreover, psychophysiological interaction (PPI) analyses revealed a cluster in left middle occipital regions showing increased functional coupling with the right SMG during syntactic relative to semantic processing, possibly indicating spatial processing mechanisms that are specific to signed syntax. Effects of hearing status were observed in the right superior temporal cortex (STC): deaf but not hearing native signers showed greater activation for semantic violations than for syntactic violations in this region. Taken together, the present findings suggest that the neural correlates of language processing are partly determined by biological constraints, but that they may additionally be influenced by the unique processing demands of the language modality and different sensory experiences.

Subunit sign modeling framework for continuous sign language recognition

A new framework named three subunit sign modeling is introduced for automatic sign language recognition. This works on continuous video sequences consisting of isolated words, signed sentences under different signer variations and illuminations. Three major issues of automatic sign language recognition is addressed namely: (i) importance of discriminative feature extraction and selection (ii) handling epenthesis movements and segmentation ambiguities (iii) automatic recognition of large vocabulary sign sentences and signer adaptation in a single subunit sign modeling framework. The proposed work has been evaluated and experimented subjectively and quantitatively with real-time signing videos gathered from different corpora and different sign languages. The results of the experiments have proven that the proposed subunit sign modeling framework remains scalable while increasing the sign vocabulary. Further, the approach is more reliable and efficient enough to adapt to real-time constraints and signer independence.

Multiple Proposals for Continuous Arabic Sign Language Recognition

The deaf community relies on sign language as the primary means of communication. For the millions of people around the world who suffer from hearing loss, interaction with hearing people is quite difficult. The main objective of sign language recognition (SLR) is the development of automatic SLR systems to facilitate communication with the deaf community. Arabic SLR (ArSLR) specifically did not receive much attention until recent years. This work presents a comprehensive comparison between two different recognition techniques for continuous ArSLR, namely a Modified k-Nearest Neighbor which is suitable for sequential data and Hidden Markov Models (HMMs) techniques based on two different toolkits. Additionally, in this work, two new ArSL datasets composed of 40 Arabic sentences are collected using Polhemus G4 motion tracker and a camera. An existing glove-based dataset is employed in this work as well. The three datasets are made publicly available to the research community. The advantages and disadvantages of each data acquisition approach and classification technique are discussed in this paper. In the experimental results section, it is shown that classification accuracy for sign sentences acquired using a motion tracker are very similar the classification accuracy for sentences acquired using sensor gloves. The modified KNN solution is inferior to HMMs in terms of the computational time required for classification.

Functional Neuroanatomy of Second Language Sentence Comprehension: An fMRI Study of Late Learners of American Sign Language.

The neurobiology of sentence comprehension is well-studied but the properties and characteristics of sentence processing networks remain unclear and highly debated. Sign languages (i.e., visual-manual languages), like spoken languages, have complex grammatical structures and thus can provide valuable insights into the specificity and function of brain regions supporting sentence comprehension. The present study aims to characterize how these well-studied spoken language networks can adapt in adults to be responsive to sign language sentences, which contain combinatorial semantic and syntactic visual-spatial linguistic information. Twenty native English-speaking undergraduates who had completed introductory American Sign Language (ASL) courses viewed videos of the following conditions during fMRI acquisition: signed sentences, signed word lists, English sentences and English word lists. Overall our results indicate that native language (L1) sentence processing resources are responsive to ASL sentence structures in late L2 learners, but that certain L1 sentence processing regions respond differently to L2 ASL sentences, likely due to the nature of their contribution to language comprehension. For example, L1 sentence regions in Broca's area were significantly more responsive to L2 than L1 sentences, supporting the hypothesis that Broca's area contributes to sentence comprehension as a cognitive resource when increased processing is required. Anterior temporal L1 sentence regions were sensitive to L2 ASL sentence structure, but demonstrated no significant differences in activation to L1 than L2, suggesting its contribution to sentence processing is modality-independent. Posterior superior temporal L1 sentence regions also responded to ASL sentence structure but were more activated by English than ASL sentences. An exploratory analysis of the neural correlates of L2 ASL proficiency indicates that ASL proficiency is positively correlated with increased activations in response to ASL sentences in L1 sentence processing regions. Overall these results suggest that well-established fronto-temporal spoken language networks involved in sentence processing exhibit functional plasticity with late L2 ASL exposure, and thus are adaptable to syntactic structures widely different than those in an individual's native language. Our findings also provide valuable insights into the unique contributions of the inferior frontal and superior temporal regions that are frequently implicated in sentence comprehension but whose exact roles remain highly debated.

Research on Continuous Sign Language Sentence Recognition Algorithm Based on Weighted Key-Frame

At present, most of the dynamic sign language recognition is only for sign language words, the continuous sign language sentence recognition research and the corresponding results are less, because the segmentation of such sentence is very difficult. In this paper, a sign language sentence recognition algorithm is proposed based on weighted key-frames. Key-frames can be regarded as the basic unit of sign word, therefore, according to key frames we can get related vocabularies, and thus we can further organize these vocabularies into meaningful sentences. Such work can avoid the hard point of dividing sign language sentence directly. With the help of Kinect, i.e. motion-control device, a kind of self-adaptive algorithm of key-frame extraction based on the trajectory of sign language is brought out in the paper. After that, the key-frame is given weight according to its semantic contribution. Finally, the recognition algorithm is designed based on these weighted key-frames and thus get the continuous sign language sentence. Experiments show that the algorithm designed in this paper can realize real-time recognition of continuous sign language sentences.

Agreement or no agreement. ERP correlates of verb agreement violation in German Sign Language

ABSTRACTPrevious studies on agreement violation in sign languages report neurophysiological responses similar to those observed for spoken languages. In contrast, the two current event-related potential studies (ERP) on agreement violations in German Sign Language sentences present results that allow for an alternative explanation. In experiment A, we investigated the processing of agreement verbs ending in an unspecified location different to the location associated with the referent. Incorrect agreement verbs engendered a posterior positivity effect (220–570 ms post nonmanual cues) and a left anterior effect (300–600 ms post the subsequent sign onset). In experiment B, we investigated a violation of morphologically modified plain verbs. Incorrect plain verbs, articulated to express third person object agreement, engendered a broadly distributed positivity effect (420–730 ms post mismatch onset). We discuss the results under the perspective of enhanced costs for context updating, and argue that sign language agreement is based on phonological and pragmatic principles.

Graph theoretical analysis of functional network for comprehension of sign language

Signed languages are natural human languages using the visual-motor modality. Previous neuroimaging studies based on univariate activation analysis show that a widely overlapped cortical network is recruited regardless whether the sign language is comprehended (for signers) or not (for non-signers). Here we move beyond previous studies by examining whether the functional connectivity profiles and the underlying organizational structure of the overlapped neural network may differ between signers and non-signers when watching sign language. Using graph theoretical analysis (GTA) and fMRI, we compared the large-scale functional network organization in hearing signers with non-signers during the observation of sentences in Chinese Sign Language. We found that signed sentences elicited highly similar cortical activations in the two groups of participants, with slightly larger responses within the left frontal and left temporal gyrus in signers than in non-signers. Crucially, further GTA revealed substantial group differences in the topologies of this activation network. Globally, the network engaged by signers showed higher local efficiency (t(24)=2.379, p=0.026), small-worldness (t(24)=2.604, p=0.016) and modularity (t(24)=3.513, p=0.002), and exhibited different modular structures, compared to the network engaged by non-signers. Locally, the left ventral pars opercularis served as a network hub in the signer group but not in the non-signer group. These findings suggest that, despite overlap in cortical activation, the neural substrates underlying sign language comprehension are distinguishable at the network level from those for the processing of gestural action.

Influence of predicate sense on word order in sign languages: Intensional and extensional verbs

We present evidence for the influence of semantics on the order of subject, object, and verb in Brazilian Sign Language (Libras) sentences. While some have argued for a prevailing pattern of SVO in Libras, we find a strong tendency for this order in sentences that do not presuppose the existence of the verb’s object, but not in sentences that do, which instead favor SOV. These findings are coherent with those of a recent study on gesture. We argue that the variable influence of the relevant predicates is particularly salient in sign languages, due to the iconic nature of the visual modality.

Comprehending Sentences With the Body: Action Compatibility in British Sign Language?

Previous studies show that reading sentences about actions leads to specific motor activity associated with actually performing those actions. We investigate how sign language input may modulate motor activation, using British Sign Language (BSL) sentences, some of which explicitly encode direction of motion, versus written English, where motion is only implied. We find no evidence of action simulation in BSL comprehension (Experiments 1-3), but we find effects of action simulation in comprehension of written English sentences by deaf native BSL signers (Experiment 4). These results provide constraints on the nature of mental simulations involved in comprehending action sentences referring to transfer events, suggesting that the richer contextual information provided by BSL sentences versus written or spoken English may reduce the need for action simulation in comprehension, at least when the event described does not map completely onto the signer's own body.

Argument Linearization in a Three-Dimensional Grammar: A Typological Perspective on Word Order in Polish Sign Language (PJM)

The aim of this paper is to analyze the underlying order of major sentential constituents (the verb and its arguments) in Polish Sign Language (polski język migowy, PJM). Although the issue of sign language sentence structure has been present in the literature for more than 30 years now, no satisfactory account thereof has yet been proposed in the case of PJM. Since visual-spatial communication is not fully linear, the importance (or even presence) of basic word order has gone unnoticed in most early accounts of the linguistic properties of PJM. We analyze the issue of argument linearization on the basis of empirical material extracted from a corpus of PJM which is being compiled at the University of Warsaw, Poland. Although a thorough description has yet to be produced, our data show that PJM does have its own rules of ordering sentential constituents. We conclude that PJM should be classified as an SVO language.

Continuous Indian Sign Language Gesture Recognition and Sentence Formation

Abstract Hand gestures are a strong medium of communication for hearing impaired society. It is helpful for establishing interaction between human and computer. In this paper we proposed a continuous Indian Sign Language (ISL) gesture recognition system where both the hands are used for performing any gesture. Recognizing a sign language gestures from continuous gestures is a very challenging research issue. We solved this problem using gradient based key frame extraction method. These key frames are helpful for splitting continuous sign language gestures into sequence of signs as well as for removing uninformative frames. After splitting of gestures each sign has been treated as an isolated gesture. Then features of pre-processed gestures are extracted using Orientation Histogram (OH) with Principal Component Analysis (PCA) is applied for reducing dimension of features obtained after OH. Experiments are performed on our own continuous ISL dataset which is created using canon EOS camera in Robotics and Artificial Intelligence laboratory (IIIT-A). Probes are tested using various types of classifiers like Euclidean distance, Correlation, Manhattan distance, city block distance etc. Comparative analysis of our proposed scheme is performed with various types of distance classifiers. From this analysis we found that the results obtained from Correlation and Euclidean distance gives better accuracy then other classifiers.

Reproducing American Sign Language sentences: cognitive scaffolding in working memory.

The American Sign Language Sentence Reproduction Test (ASL-SRT) requires the precise reproduction of a series of ASL sentences increasing in complexity and length. Error analyses of such tasks provides insight into working memory and scaffolding processes. Data was collected from three groups expected to differ in fluency: deaf children, deaf adults and hearing adults, all users of ASL. Quantitative (correct/incorrect recall) and qualitative error analyses were performed. Percent correct on the reproduction task supports its sensitivity to fluency as test performance clearly differed across the three groups studied. A linguistic analysis of errors further documented differing strategies and bias across groups. Subjects' recall projected the affordance and constraints of deep linguistic representations to differing degrees, with subjects resorting to alternate processing strategies when they failed to recall the sentence correctly. A qualitative error analysis allows us to capture generalizations about the relationship between error pattern and the cognitive scaffolding, which governs the sentence reproduction process. Highly fluent signers and less-fluent signers share common chokepoints on particular words in sentences. However, they diverge in heuristic strategy. Fluent signers, when they make an error, tend to preserve semantic details while altering morpho-syntactic domains. They produce syntactically correct sentences with equivalent meaning to the to-be-reproduced one, but these are not verbatim reproductions of the original sentence. In contrast, less-fluent signers tend to use a more linear strategy, preserving lexical status and word ordering while omitting local inflections, and occasionally resorting to visuo-motoric imitation. Thus, whereas fluent signers readily use top-down scaffolding in their working memory, less fluent signers fail to do so. Implications for current models of working memory across spoken and signed modalities are considered.

Towards subject independent continuous sign language recognition: A segment and merge approach

This paper presents a segment-based probabilistic approach to robustly recognize continuous sign language sentences. The recognition strategy is based on a two-layer conditional random field (CRF) model, where the lower layer processes the component channels and provides outputs to the upper layer for sign recognition. The continuously signed sentences are first segmented, and the sub-segments are labeled SIGN or ME (movement epenthesis) by a Bayesian network (BN) which fuses the outputs of independent CRF and support vector machine (SVM) classifiers. The sub-segments labeled as ME are discarded and the remaining SIGN sub-segments are merged and recognized by the two-layer CRF classifier; for this we have proposed a new algorithm based on the semi-Markov CRF decoding scheme. With eight signers, we obtained a recall rate of 95.7% and a precision of 96.6% for unseen samples from seen signers, and a recall rate of 86.6% and a precision of 89.9% for unseen signers.

Models of linguistic facial expressions for American Sign Language animation

Facial expressions are a necessary and important part of the grammar of American Sign Language (ASL) sentences. However, current computational linguistic software cannot produce a huge portion of ASL constructions that include facial expressions. Here we summarize our research motivations, proposed plan, and potential contributions in the accessibility field. Our research aim is to formulate signer-independent models of facial movements for various types of linguistically meaningful facial expressions in ASL animation. This project summary also includes a brief description of current progress and future research plans.

Motion-sensitive cortex and motion semantics in American Sign Language

Previous research indicates that motion-sensitive brain regions are engaged when comprehending motion semantics expressed by words or sentences. Using fMRI, we investigated whether such neural modulation can occur when the linguistic signal itself is visually dynamic and motion semantics is expressed by movements of the hands. Deaf and hearing users of American Sign Language (ASL) were presented with signed sentences that conveyed motion semantics ("The deer walked along the hillside.") or were static, conveying little or no motion ("The deer slept along the hillside."); sentences were matched for the amount of visual motion. Motion-sensitive visual areas (MT+) were localized individually in each participant. As a control, the Fusiform Face Area (FFA) was also localized for the deaf participants. The whole-brain analysis revealed static (locative) sentences engaged regions in left parietal cortex more than motion sentences, replicating previous results implicating these regions in comprehending spatial language for sign languages. Greater activation was observed in the functionally defined MT+ ROI for motion than static sentences for both deaf and hearing signers. No modulation of neural activity by sentence type was observed in the FFA. Deafness did not affect modulation of MT+ by motion semantics, but hearing signers exhibited stronger neural activity in MT+ for both sentence types, perhaps due to differences in exposure and/or use of ASL. We conclude that top down modulation of motion-sensitive cortex by linguistic semantics is not disrupted by the visual motion that is present in sign language sentences.

Dissociating linguistic and non-linguistic gesture processing: Electrophysiological evidence from American Sign Language

A fundamental advance in our understanding of human language would come from a detailed account of how non-linguistic and linguistic manual actions are differentiated in real time by language users. To explore this issue, we targeted the N400, an ERP component known to be sensitive to semantic context. Deaf signers saw 120 American Sign Language sentences, each consisting of a “frame” (a sentence without the last word; e.g. BOY SLEEP IN HIS) followed by a “last item” belonging to one of four categories: a high-close-probability sign (a “semantically reasonable” completion to the sentence; e.g. BED), a low-close-probability sign (a real sign that is nonetheless a “semantically odd” completion to the sentence; e.g. LEMON), a pseudo-sign (phonologically legal but non-lexical form), or a non-linguistic grooming gesture (e.g. the performer scratching her face). We found significant N400-like responses in the incongruent and pseudo-sign contexts, while the gestures elicited a large positivity.

Negation in Turkish Sign Language

This paper presents a detailed description of manual and nonmanual markers (NMMs) in negative sentences in Turkish Sign Language (TİD). It has been argued that TİD has a manual dominant pattern for negation with a backward head tilt being the major NMM in negative sentences. By contrast, in this paper I argue that, when it comes to negation, TİD is not a strictly manual dominant language. I will consider spreading of NMMs such as headshake, brow-lowering, and brow-raising and argue that these spreading NMMs play a syntactic role in TİD negative sentences. They mark the syntactic domain of negation, either through c-command, a spec-head-relation or both. In addition, I offer a generative analysis of the syntax of negation in TİD based on the syntactic background provided for TİD and the distributional patterns of NMMs.