Visual-spatial Modality Research Articles

Reference tracking strategies of deaf adult signers in Turkish Sign Language

We investigated the reference tracking strategies among deaf adults with signing deaf parents (DoD) and adult deaf signers with non-signing hearing parents or caregivers (DoH) who had late exposure to Turkish Sign Language (TİD). Consistent with the theories of saliency and referential accessibility, regardless of their acquisition groups or parental hearing status, signers mainly used nominals and extension classifiers for introducing referents. To maintain a referent from the previous clause, the signers used zero anaphora (e.g., constructed action, agreement and plain verbs). For topic shifts or re-introduced contexts, nominals and pronouns were chiefly favored although we observed very little pronominal use. As for the effect of native acquisition from deaf caregivers, we only report limited over-redundancy for DoH signers who used zero anaphora less compared to DoD especially for introduced and maintained contexts. We conclude that DoH signers are still able to achieve native-like competency in terms of reference tracking in simple narratives but DoD signers utilize the spatial affordances of the visual-spatial modality better than DoH signers only to a certain extent.

Sign advantage: Both children and adults' spatial expressions in sign are more informative than those in speech and gestures combined.

Expressing Left-Right relations is challenging for speaking-children. Yet, this challenge was absent for signing-children, possibly due to iconicity in the visual-spatial modality of expression. We investigate whether there is also a modality advantage when speaking-children's co-speech gestures are considered. Eight-year-old child and adult hearing monolingual Turkish speakers and deaf signers of Turkish-Sign-Language described pictures of objects in various spatial relations. Descriptions were coded for informativeness in speech, sign, and speech-gesture combinations for encoding Left-Right relations. The use of co-speech gestures increased the informativeness of speakers' spatial expressions compared to speech-only. This pattern was more prominent for children than adults. However, signing-adults and children were more informative than child and adult speakers even when co-speech gestures were considered. Thus, both speaking- and signing-children benefit from iconic expressions in visual modality. Finally, in each modality, children were less informative than adults, pointing to the challenge of this spatial domain in development.

Expression of Aboutness Subject Topic Constructions in Turkish Sign Language (TİD) Narratives

In the visual-spatial modality, signers indicate old, new, or contrastive information using certain syntactic, prosodic, and morphological strategies. Even though information structure has been described extensively for many sign languages, the flow of information in the narrative discourse remains unexplored in Turkish Sign Language (TİD). This study aims to describe aboutness subject topic constructions in TİD narratives. We examined data from six adult native signers of TİD and found that TİD signers mainly used nominals for reintroduced aboutness subject topics. The optional and rare non-manual markers observed on reintroduced topics mainly included squint, brow raise, and backward head tilt. Maintained aboutness subject topics, which have higher referential accessibility, were often omitted and tracked with zero anaphora. Finally, we found that constructed action is more frequently present on the predicates of clauses with a maintained aboutness subject topic than with a reintroduced aboutness subject topic. Overall, these results indicate that the use of constructed action and nominals in aboutness subject topics correlates with referential accessibility in TİD. While the former has been observed more in maintained contexts, the latter has been observed mainly in reintroduced contexts. In addition to the syntactic and prosodic cues that may distinguish old information from new or contrastive information in narratives, we suggest that pragmatic cues such as referential accessibility may help account for the manual and non-manual articulation strategies for information structure in TİD narratives.

Cross-modal investigation of event component omissions in language development: a comparison of signing and speaking children

ABSTRACT Language development research suggests a universal tendency for children to be under- informative in narrating motion events by omitting components such as Path, Manner or Ground. However, this assumption has not been tested for children acquiring sign language. Due to the affordances of the visual-spatial modality of sign languages for iconic expression, signing children might omit event components less frequently than speaking children. Here we analysed motion event descriptions elicited from deaf children (4–10 years) acquiring Turkish Sign Language (TİD) and their Turkish-speaking peers. While children omitted all types of event components more often than adults, signing children and adults encoded more Path and Manner in TİD than their peers in Turkish. These results provide more evidence for a general universal tendency for children to omit event components as well as a modality bias for sign languages to encode both Manner and Path more frequently than spoken languages.

Deep Learning Methods for Sign Language Translation

Many sign languages are bona fide natural languages with grammatical rules and lexicons hence can benefit from machine translation methods. Similarly, since sign language is a visual-spatial language, it can also benefit from computer vision methods for encoding it. With the advent of deep learning methods in recent years, significant advances have been made in natural language processing (specifically neural machine translation) and in computer vision methods (specifically image and video captioning). Researchers have therefore begun expanding these learning methods to sign language understanding. Sign language interpretation is especially challenging, because it involves a continuous visual-spatial modality where meaning is often derived based on context. The focus of this article, therefore, is to examine various deep learning–based methods for encoding sign language as inputs, and to analyze the efficacy of several machine translation methods, over three different sign language datasets. The goal is to determine which combinations are sufficiently robust for sign language translation without any gloss-based information. To understand the role of the different input features, we perform ablation studies over the model architectures (input features + neural translation models) for improved continuous sign language translation. These input features include body and finger joints, facial points, as well as vector representations/embeddings from convolutional neural networks. The machine translation models explored include several baseline sequence-to-sequence approaches, more complex and challenging networks using attention, reinforcement learning, and the transformer model. We implement the translation methods over multiple sign languages—German (GSL), American (ASL), and Chinese sign languages (CSL). From our analysis, the transformer model combined with input embeddings from ResNet50 or pose-based landmark features outperformed all the other sequence-to-sequence models by achieving higher BLEU2-BLEU4 scores when applied to the controlled and constrained GSL benchmark dataset. These combinations also showed significant promise on the other less controlled ASL and CSL datasets.

Constituent order in Serbian Sign Language declarative clauses

Constituent order can encode grammatical relations in a language. The visual-spatial modality imbues sign languages with characteristics such as simultaneity or the use of space which raise the question of the appropriate unit of analysis in constituent order studies. In this paper, we provide empirical evidence on the order of core constituents in elicited declarative clauses for non-reversible, reversible and locative states-of-affairs in Serbian Sign Language (SZJ). Forty (near-)native deaf SZJ signers, ranging in age between 18 and 70 years old, participated in the data-collection. We consider linguistic and social factors in 810 clauses elicited for the purposes of this study. Our findings suggest that SVO is a preferred order in non-locative clauses with two overtly expressed arguments, whilst GROUND-FIGURE-LOCATIVE RELATION is the most frequent pattern in locative clauses. We argue that our results provide some support for the claim that sign language discourse can be analysed in terms of constituent order in the clause, but that other strategies typical of the visual modality such as the simultaneous expression of core constituents, and manual and non-manual features, the use of space, core argument incorporation into the form of the verb and core argument omission, complicate the traditional notion of sequential constituent order clause as a central grammatical element in SZJ and, by extension, in other sign languages. Consequently, the description of relations between core constituents calls for careful consideration and analysis of different types of data as a way of gaining a clearer insight into the nature of a sign language.

Problematizando o uso do aplicativo de tradução Hand Talk no ensino da Libras no Ensino Superior

O Hand Talk constitui-se um tradutor automático de palavras e frases da Língua Portuguesa (LP) para a Libras, realizando a tradução por meio de um agente animado (3D), denominado Hugo. Esse aplicativo encontra-se presente em diversos segmentos sociais, inclusive no meio universitário. Dessa forma, é usual acadêmicos da disciplina da Libras buscarem apoio nele, em razão da dificuldade de aprender esta língua, de modalidade visual-espacial, que por tal contém uma lógica de produção e recepção diferente da LP, que é oral-auditiva. Apesar de o aplicativo ter validade social, sendo uma tecnologia que intenta oportunizar a acessibilidade linguística entre surdos e ouvintes, utilizando diversos recursos como, na seção de tradução, a possibilidade de repetição do sinal, de controle da velocidade de sinalização assim como de girá-lo em 360° e, dessa forma, visualizar a produção do sinal por diversos ângulos, faz-se necessário considerar que ele apresenta algumas fragilidades no processo de tradução. Entre elas, a baixa expressividade facial e corporal do Hugo, o uso recorrente da datilologia, a dificuldade de tradução, em contexto, de termos polissêmicos, o que muitas vezes repercute em equívocos de tradução da LP para a Libras. Assim, ao considerar que a presença do Hand Talk na maioria dos celulares dos alunos é uma realidade, cabe ao professor da Libras orientar sobre sua real funcionalidade, otimizando suas possibilidades benéficas de uso, como o dicionário, por exemplo. De forma estratégica, pode-se aproveitar suas fragilidades de tradução para que os aprendizes percebam a importância de se seguir a lógica visual de sinalização, não buscando, por exemplo, a enunciação com base palavra-sinal, mas sim em contexto, respeitando estrutura linguística da Libras.

Dual task interference on early perceptual processing

When two tasks, Task 1 and Task 2, are conducted in close temporal proximity and a separate speeded response is required for each target (T1 and T2), T2 report performance decreases as a function of its temporal proximity to T1. This so-called psychological refractory period (PRP) effect on T2 processing is largely assumed to reflect interference from T1 response selection on T2 response selection. However, interference on early perceptual processing of T2 has been observed in a modified paradigm, which required changes in visual-spatial attention, sensory modality, task modality, and response modality across targets. The goal of the present study was to investigate the possibility of early perceptual interference by systematically and iteratively removing each of these possible non perceptual confounds, in a series of four experiments. To assess T2 visual memory consolidation success, T2 was presented for a varying duration and immediately masked. T2 report accuracy, which was taken as a measure of perceptual—encoding or consolidation—success, decreased across all experimental control conditions as T1–T2 onset proximity increased. We argue that our results, in light of previous studies, show that central processing of a first target, responsible for the classical PRP effect, also interferes with early perceptual processing of a second target. We end with a discussion of broader implications for psychological refractory period and attentional blink effects.

Aquisição de línguas de sinais e teoria linguística

A conferência ministrada pela Prof.ª Dr.ª Diane Lillo-Martin (University of Connecticut) propôs-se à apresentação do panorama de pesquisas sobre aquisição de línguas de sinais (doravante LS), realizadas em cooperação entre pesquisadores norte-americanos e brasileiros. Teve como intuito maior a reflexão de como investigações no domínio da aquisição de LS evidenciam pormenores atrelados a universais linguísticos, de modo a contribuir a hipóteses e teorias já difundidas em estudos anteriores com línguas orais (doravante LO). Em vista disso, abordaram-se tópicos de interesse às áreas de psicolinguística e estudos em aquisição de línguas, tais como questões estruturais das LS, especificamente de American Sign Language (ASL) e Língua Brasileira de Sinais (Libras); efeitos de modalidade visuo-espacial, especificidade do processo de aquisição de linguagem por crianças surdas bilíngues bimodais e implicaturas de privação linguística.

Lexical Recognition in Deaf Children Learning American Sign Language: Activation of Semantic and Phonological Features of Signs.

Children learning language efficiently process single words, and activate semantic, phonological, and other features of words during recognition. We investigated lexical recognition in deaf children acquiring American Sign Language (ASL) to determine how perceiving language in the visual-spatial modality affects lexical recognition. Twenty native- or early-exposed signing deaf children (ages 4 to 8 years) participated in a visual world eye-tracking study. Children were presented with a single ASL sign, target picture, and three competitor pictures that varied in their phonological and semantic relationship to the target. Children shifted gaze to the target picture shortly after sign offset. Children showed robust evidence for activation of semantic but not phonological features of signs, however in their behavioral responses children were most susceptible to phonological competitors. Results demonstrate that single word recognition in ASL is largely parallel to spoken language recognition among children who are developing a mature lexicon.

Situating simultaneity: an initial schematization of the lexicogrammatical rank scale of British Sign Language

A central tenet of systemic functional theory is the rank scale: an ordered representation of the part-whole relationships of units within semiotic systems. Linguists have schematized the rank scales for the lexicogrammars of English, French, Spanish and Chinese, to name a few. However, such schematization has yet to occur for languages in the visual-spatial modality (i.e. sign languages).This paper contributes to current literature by establishing a working schematization of the lexicogrammatical rank scale of British Sign Language (BSL). By taking a glottocentric perspective and with reference to systemic functional theory and BSL data, this work demonstrates that it is possible to create an organized rank scale for a language operating the visual-spatial modality as long as the productive simultaneity found within is accounted for sufficiently. This is enacted through a more detailed elaboration of the morpheme rank, so that higher ranks may be represented accurately. This study provides the foundations for similar rank scales of semiotic systems operating in the visual-spatial modality to be schematized, while also suggesting areas for further empirical investigation in both systemic functionalism and sign linguistics.

Non-Manual Expression – Sign Language as L2

Sign Language is fundamental for Deaf communication, culture and citizenship. The Brazilian Sign Language (Libras) is a complet linguistic system, of visual-spatial modality, with specificities that present a challenge for teaching and learning as L2. Non-Manual Expression is on atributte of the language for meaning attribution. Meaning attribution occurs from visual symbolic processes where non-manual expressions acquire a central role, and differs from those used in the oral language. This requires adequate educational practices and pedagogical material for the acquisition of Libras as L2. This research proposes a learning object and a methodology for teaching and learning of Libras in the form of a game that focus on non-manual expressions. The proposed methodology comprises the context, the theme of the communication situation, the utterance of the sign in Libras, along with learning fixation activities. Validation shows the importance of valuing the grammar of Libras as a pedagogical strategy that is adequate to teaching and learning non-manual expression in Libras.

Visual spatial gnosis and visual constructive activity in patients before and after embolization of intracranial aneurysms

Aim. The study was aimed at examining visual spatial gnosis and visual constructive activity in patients with cerebral aneurysms of anterior circulation before and after endovascular embolization with flow-diverter stents.Methods. Nineteen patients were examined before and after embolization of aneurysms. Patients were asked to perform 5 tests: determining the time on an “empty” dial (without numbers); setting the clock hands on an “empty” dial for the given time; drawing a human figure using the left and right hands; copying the picture of a house from the sample; copying from memory the Rey–Taylor and the Rey–Osterrieth complex figures.Results. Group differences between single measures obtained before and after the surgery were not statistically significant in any of the tests. At the same time, dependency on the hand used for drawing was nearly significant for postoperative changes in human figure drawing quality: the quality of figures drawn using the right hand became on average better after surgery, while the figures drawn using the left hand got worse. Postoperative individual changes demonstrated a “mosaic” pattern for the whole set of the tests, i.e. in most patients the results improved in some test(s) but degraded in other(s) and patterns of these changes differed between the patients.Conclusion. Postoperative changes in visual spatial gnosis and visual constructive activity are of a “mosaic” nature, supposedly due to multiple brain microlesions without certain localizations. At the same time, the differences between changes in the quality of human figures drawn using the right and left hands could be a sign of relative prevalence of the right hemisphere lesions. But in general, there were no significant postoperative changes in the brain integrative activity within the visual-spatial modality after endovascular aneurysm surgery with flow-diverter stents.Received 13 March 2018. Revised 17 April 2018. Accepted 20 April 2018.Funding: The study did not have sponsorship.Conflict of interest: The authors declare no conflict of interest.Author contributionsConception and study design: V.G. Postnov, K.Yu. OrlovData collection: V.G. Postnov, R.S. KiselevData analysis: V.G. Postnov, E.A. LevinDrafting the article: V.G. Postnov, E.A. Levin, R.S. KiselevFinal approval of the version to be published: V.G. Postnov, E.A. Levin, K.Yu. Orlov, R.S. Kiselev

Poetry in South African Sign Language: what is different?

Poetry in a sign language can make use of literary devices just as poetry in a spoken language can. The study of literary expression in sign languages has increased over the last twenty years and for South African Sign Language (SASL) such literary texts have also become more available. This article gives a brief overview of the linguistic devices sign language poetry can make use of, in particular those specific to the visual-spatial modality. As an illustration of these devices an analysis is then presented of the SASL poem Soweto by Modiegi Moime. This poem illustrates well the multi-layered meaning that can be created in sign language poetry through the use of the two hands and the non-manual components.

Processos produtivos de sinais nocionalmente Nomes na libras

Este artigo tem como objetivo apresentar considerações sobre sinais utilizados pela Língua de Sinais Brasileira (Libras) que se correlacionam nocionalmente com nomes no português brasileiro. As observações se deram com base nas características semânticas prototípicas para a definição da categoria lexical “nome”, propostas por Givón (2001). Ao considerar especialmente a modalidade visuoespacial dessa língua, imagina-se a dimensão como critério semântico de relevância, pois é através dela que são descritas diversas entidades. Sugerem-se três processos produtivos de sinais nocionalmente nomes nessa língua, com base em suas características semânticas: a representação espacial das entidades, sinais que inserem a letra inicial da palavra em português na sua forma, e a datilologia como recurso para a formação de sinais.

Viewpoint in the Visual-Spatial Modality: The Coordination of Spatial Perspective

Sign languages express viewpoint-dependent spatial relations (e.g., left, right) iconically but must conventionalize from whose viewpoint the spatial relation is being described, the signer's or the perceiver's. In Experiment 1, ASL signers and sign-naïve gesturers expressed viewpoint-dependent relations egocentrically, but only signers successfully interpreted the descriptions nonegocentrically, suggesting that viewpoint convergence in the visual modality emerges with language conventionalization. In Experiment 2, we observed that the cost of adopting a nonegocentric viewpoint was greater for producers than for perceivers, suggesting that sign languages have converged on the most cognitively efficient means of expressing left-right spatial relations. We suggest that nonlinguistic cognitive factors such as visual perspective-taking and motor embodiment may constrain viewpoint convergence in the visual-spatial modality.

O ensino da língua brasileira de sinais na graduação

This work aims at raising reflections about Brazilian sign language (LIBRAS) teaching in undergraduate courses, since the school subject LIBRAS is inserted in this academic context due to Decreto 5.626 of 22nd. December 2005. This is a text that exposes what happens in the day to day life of classroom through the notes of the ones who teach the mentioned subject. It aims to approach facts and characteristics of this insertion taking into account that sign language is a language of visual-spatial modality which is not known by many people from acedemic medium. Despite the difficulties pointed in the text, it is noticeable that teachers tend to create possibilities to overcome these difficulties, teaching this school subject in such a way that makes it possible to incorporate sign language teaching in the universities with quality.

Learning Managent Systems and Face-to-Face Teaching in Bilingual Modality (Libras/Portuguese)

Discussions related to new approaches in deaf education are highlighting the underexplored potencial of the Information and Communication Technologies for this persons. This paper presents an study carried out under Learning Management Systems (LMS) in order to subsizide the selection of a LMS to support face-to-face classes in the form of bilingual education (Libras/Portuguese). The study was conducted in the context of Instituto Federal de Educacao, Ciencia e Tecnologia de Santa Catarina (IFSC) - Campus Palhoca-Bilingue. For this, the analysis was accomplished over open source platforms and grounded in issues related to accessibility of learning management systems to deaf people. In this way, subjects related to conformance with present accessibility guidelines and recommendations, the interoperability standards and the use of captions were analyzed, besides the identification of tools supporting interaction and communication in sign language and the engaged communities aiming the improvement of the environments studied. The selected LMS must have the featured characteristics to provide the assignment of a bilingual education propose, presenting the possibilities to explore the characteristics of the visual-spatial modality of communication present in the Lingua Brasileira de Sinais (Libras), as well as the tools to stimulate the sharing and construction of knowledge in the teachers and students community. This is just the first step considering the need of development of a series of technological devices consonant with this approach.

The signing of Deaf children with autism: Lexical phonology and perspective-taking in the visual-spatial modality

The signing of Deaf children with autism: Lexical phonology and perspective-taking in the visual-spatial modality

Locative expressions in signed languages: a view from Turkish Sign Language (TİD)

Locative expressions encode the spatial relationship between two (or more) entities. In this paper, we focus on locative expressions in signed language, which use the visual-spatial modality for linguistic expression, specifically in Turkish Sign Language ( Türk İşaret Dili , henceforth TİD ). We show that TİD uses various strategies in discourse to encode the relation between a Ground entity (i.e., a bigger and/or backgrounded entity) and a Figure entity (i.e., a smaller entity, which is in the focus of attention). Some of these strategies exploit affordances of the visual modality for analogue representation and support evidence for modality-specific effects on locative expressions in sign languages. However, other modality-specific strategies, e.g., the simultaneous expression of Figure and Ground, which have been reported for many other sign languages, occurs only sparsely in TİD. Furthermore, TİD uses categorical as well as analogical structures in locative expressions. On the basis of these findings, we discuss differences and similarities between signed and spoken languages to broaden our understanding of the range of structures used in natural language (i.e., in both the visual-spatial or oral-aural modalities) to encode locative relations. A general linguistic theory of spatial relations, and specifically of locative expressions, must take all structures that might arise in both modalities into account before it can generalize over the human language faculty.