Chinese Sign Research Articles

Global-Local Enhancement Network for NMF-Aware Sign Language Recognition

Sign language recognition (SLR) is a challenging problem, involving complex manual features (i.e., hand gestures) and fine-grained non-manual features (NMFs) (i.e., facial expression, mouth shapes, etc .). Although manual features are dominant, non-manual features also play an important role in the expression of a sign word. Specifically, many sign words convey different meanings due to non-manual features, even though they share the same hand gestures. This ambiguity introduces great challenges in the recognition of sign words. To tackle the above issue, we propose a simple yet effective architecture called Global-Local Enhancement Network (GLE-Net), including two mutually promoted streams toward different crucial aspects of SLR. Of the two streams, one captures the global contextual relationship, while the other stream captures the discriminative fine-grained cues. Moreover, due to the lack of datasets explicitly focusing on this kind of feature, we introduce the first non-manual-feature-aware isolated Chinese sign language dataset (NMFs-CSL) with a total vocabulary size of 1,067 sign words in daily life. Extensive experiments on NMFs-CSL and SLR500 datasets demonstrate the effectiveness of our method.

Early Development of Chinese Sign Language in Shanghai Schools for the Deaf

The Shanghai variant of Chinese Sign Language (SCSL) is one of the main variants of Chinese sign languages, greatly influencing other sign languages, such as Hong Kong Sign Language and Singapore Sign Language. This paper is a first attempt to trace its origins and early history and deaf education in Shanghai until 1949. The data are collected in two ways: first, by delving into the archives, i.e., written records of deaf history and education in China during that time; second, by interviewing surviving deaf students who went to school before or around 1949. Our findings are as follows: (a) SCSL began in the 1920s and emerged as a distinct sign language in the 1940s. Two deaf schools were the power shaping its progress among several deaf schools established in Shanghai: Fryer deaf school and Group learning deaf school. The sign variants of these two schools form the backbone of SCSL. (b) Deaf teachers are one of the key factors that affect the early development of a sign language. Chinese deaf played a vital role in the rise and spread of SCSL in the 1930s and 1940s, as some deaf teachers opened deaf schools in Shanghai and other cities, even other countries or areas, thus helping SCSL to spread. (c) Arising in an international and multilingual environment, SCSL is characterized by traces of foreign sign languages, especially ASL, due to language contacts linked to deaf education at that time, e.g., some proper names, like XUJIAHUI, SHANGHAI-1 and some high-frequency words like water. (d) However, foreign sign languages' direct influence is negligible due to the lack of participation of deaf foreigners in deaf education in Shanghai and oralism advocated by foreign educators in relevant deaf schools. To sum up, deaf teachers for deaf schools are key to the early development and spread of SCSL.

Teaching Chinese Sign Language with a Smartphone

BackgroundThere is a large group of deaf-mutes in the world, and sign language is their major communication tool. Therefore, it’s necessary for the deaf-mutes to communicate with hearing-speech people and the hearing-speech people also have needed to understand sign language, which produces a great demand for sign language teaching. Even though there have already been a large number of books for sign language, it is low efficient to learn sign language with books, even teaching videos. To solve this problem, we develop a smartphone-based interactive Chinese sign language teaching systemfor sign language learning. MethodsThe system provides a learner with some kinds of learning modes and captures the learner’s actions from its front camera of the smartphone. Right now the system provides a vocabulary set with 1000 frequently used words, and the learner can evaluate his/her sign action by subjective or objective comparison. In the mode of word recognition, the users can play any word within the vocabulary and the system will turn the top three retrieved candidates, so it can remind the learners what the sign is. ResultsThis system provides interactive learning for a user to learn sign language high efficiently. The systemadopts an algorithm based on point cloud recognition to evaluate a user’s sign and costs about 700ms inference time for each sample, which meets the real-time requirements. ConclusionThis interactive learning system decreases the communication barriers between the deaf-mutes and hearing-speechers.

Dynamic Gesture Recognition Combining Two-stream 3D Convolution with Attention Mechanisms

Benefits from the progress of computer hardware and computing power, natural and simple dynamic gesture recognition gets a lot of attention in human-computer interaction. In view of the requirement of the accuracy of dynamic gesture recognition in human-computer interaction, a method of dynamic gesture recognition that combines Two-stream Inflated 3D (I3D) Convolution Neural Network (CNN) with the Convolutional Block Attention Module (CBAM-I3D) is proposed. In addition, relevant parameters and structures of the I3D network model are improved. In order to improve the convergence speed and stability of the model, the Batch Normalization (BN) technology is used to optimize the network, which shortens the training time of the optimized network. At the same time, experimental comparisons with various Two-stream 3D convolution methods on the open source Chinese Sign Language (CSL) recognition dataset are performed. The experimental results show that the proposed method can recognize dynamic gestures well, and the recognition rate reaches 90.76%, which is higher than other dynamic gesture recognition methods. The validity and feasibility of the proposed method are verified.

Linguistic but Not Cognitive Weaknesses in Deaf or Hard-of-Hearing Poor Comprehenders.

This study examined the reading comprehension profiles, and the related linguistic and cognitive skills, of 146 Chinese students in Grades 3-9 who are deaf or hard of hearing (d/Dhh). Employing a rigorous regression approach, the current study identified 19 unexpected poor comprehenders, 24 expected average comprehenders, and 16 unexpected good comprehenders. Compared to the expected average and unexpected good comprehenders, the unexpected poor comprehenders performed worse in broad linguistic skills (i.e., Chinese sign language comprehension, vocabulary, and segmental and suprasegmental phonological awareness), but their weaknesses in cognitive skills (i.e., working memory and executive function) were less severe. These findings suggest that weak linguistic skills are possible indicators of reading comprehension difficulties for students who are d/Dhh.

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network.

Continuous sign language recognition is a weakly supervised task dealing with the identification of continuous sign gestures from video sequences, without any prior knowledge about the temporal boundaries between consecutive signs. Most of the existing methods focus mainly on the extraction of spatio-temporal visual features without exploiting text or contextual information to further improve the recognition accuracy. Moreover, the ability of deep generative models to effectively model data distribution has not been investigated yet in the field of sign language recognition. To this end, a novel approach for context-aware continuous sign language recognition using a generative adversarial network architecture, named as Sign Language Recognition Generative Adversarial Network (SLRGAN), is introduced. The proposed network architecture consists of a generator that recognizes sign language glosses by extracting spatial and temporal features from video sequences, as well as a discriminator that evaluates the quality of the generator’s predictions by modeling text information at the sentence and gloss levels. The paper also investigates the importance of contextual information on sign language conversations for both Deaf-to-Deaf and Deaf-to-hearing communication. Contextual information, in the form of hidden states extracted from the previous sentence, is fed into the bidirectional long short-term memory module of the generator to improve the recognition accuracy of the network. At the final stage, sign language translation is performed by a transformer network, which converts sign language glosses to natural language text. Our proposed method achieved word error rates of 23.4%, 2.1% and 2.26% on the RWTH-Phoenix-Weather-2014 and the Chinese Sign Language (CSL) and Greek Sign Language (GSL) Signer Independent (SI) datasets, respectively.

Chinese fingerspelling recognition via gray-level co-occurrence matrix and fuzzy support vector machine

INTRODUCTION: Chinese deaf-mutes communicate in their native language, Chinese sign language which contains gesture language and finger language. Chinese finger language conveys information through various movements of fingers, and its expression is accurate and convenient for c

Chinese fingerspelling sign language recognition using a nine-layer convolutional neural network

INTRODUCTION: Sign language is a form of communication and exchange of ideas by people who are hearing-impaired or unable to speak. Chinese fingerspelling is an important component of Chinese sign language, which is suitable for denoting terminology and using as

Multiuser gesture recognition using sEMG signals via canonical correlation analysis and optimal transport

Myoelectric interfaces have received much attention in the field of prosthesis control, neuro-rehabilitation systems and human-computer interaction. However, when different users perform the same gesture, the electromyography (EMG) signals can vary greatly. It is essential to design a multiuser myoelectric interface that can be simply used by novel users while maintaining good gesture classification performance. To cope with this problem, canonical correlation analysis (CCA) has been used to extract the inherent user-independent properties of EMG signals generated from the same gestures from multiple users and demonstrated superior performance. In this paper, we move forward to propose a novel framework based on CCA and optimal transport (OT), termed as CCA-OT. By optimal transport, the discrepancies in data distribution between the transformed feature matrix from the training and the testing sets can be further reduced. Experimental results on the defined 13 Chinese sign language gestures performed by 10 intact-limbed subjects demonstrated that the classification rate of our proposed CCA-OT framework is significantly higher than that of the CCA-only framework with an 8.49% promotion, which shows the necessity to reduce the drift in probability distribution functions (PDFs) of the different domains. The CCA-OT framework provides a promising method for the multiuser myoelectric interface which can be easily adapted to new users. This improvement will further facilitate the widespread implementation of myoelectric control systems using pattern recognition techniques.

Neural Decoding of Chinese Sign Language With Machine Learning for Brain-Computer Interfaces.

Limb motion decoding is an important part of brain-computer interface (BCI) research. Among the limb motion, sign language not only contains rich semantic information and abundant maneuverable actions but also provides different executable commands. However, many researchers focus on decoding the gross motor skills, such as the decoding of ordinary motor imagery or simple upper limb movements. Here we explored the neural features and decoding of Chinese sign language from electroencephalograph (EEG) signal with motor imagery and motor execution. Sign language not only contains rich semantic information, but also has abundant maneuverable actions, and provides us with more different executable commands. In this paper, twenty subjects were instructed to perform movement execution and movement imagery based on Chinese sign language. Seven classifiers are employed to classify the selected features of sign language EEG. L1 regularization is used to learn and select features that contain more information from the mean, power spectral density, sample entropy, and brain network connectivity. The best average classification accuracy of the classifier is 89.90% (imagery sign language is 83.40%). These results have shown the feasibility of decoding between different sign languages. The source location reveals that the neural circuits involved in sign language are related to the visual contact area and the pre-movement area. Experimental evaluation shows that the proposed decoding strategy based on sign language can obtain outstanding classification results, which provides a certain reference value for the subsequent research of limb decoding based on sign language.

Real-time sign language recognition based on video stream

In this paper, a real-time Chinese sign language recognition system is investigated. This system can recognise deaf-mute Chinese sign language, and output the recognition results in real time throu...

Deaf Teachers' Reflections on Chinese Literacy.

Using grounded theory, the researcher posed this question in this qualitative study: What childhood literacy-learning and current literacy-teaching experiences have influenced Chinese Deaf teachers' views on literacy learning? Responses were obtained from Deaf teachers by means of videotaped interviews about their literacy-learning and literacy-teaching experiences. When the interviews, which were conducted in Chinese Sign Language (CSL) glossed to written Chinese and English, were analyzed, six themes emerged. Extracted core categories provide the unique context for a "boomerang effect" related to language and literacy through a bilingual path to literacy. Recommendations for future research using bilingual theory and practice are discussed.

Boundary-Adaptive Encoder With Attention Method for Chinese Sign Language Recognition

The sign language signal has hierarchically related information over short and long distances. Due to the intricate temporal correlation of input sequences, Chinese sign language recognition (SLR) has a modeling challenge. The conventional encoders based on recurrent networks cannot discover and leverage the hierarchical structure of sign language well. In this paper, we propose a novel encoder-decoder method based on boundary adaptive learning for Chinese SLR. The hierarchical structure of sign language signal can be encoded by the boundary-adaptive encoder (BAE) in the proposed method. In order to improve efficiency in modeling long sign language sequences, the window attention model based on location is utilized in the decoding phase, which can generate more effective weight coefficients. Besides, we use sign language subword units to realize both isolated and continuous Chinese SLR in the same sequence learning framework in our method. Theoretical analysis and experimental results demonstrate the effectiveness and superiority of the proposed method.

Application of Tensor Train Decomposition in S2VT Model for Sign Language Recognition

Sign language recognition is a conversion of sign language into text or speech, bridging the communication between the hearing and society. Recently, sequence-to-sequence video to text (S2VT) models has been employed in the field of sign language recognition as an effective method. However, more than 20 million parameters trained in S2VT models will result in a huge consumption in memory and computational resources, making it hard to be applied in mobile devices. In order to overcome this issue, we proposed to employ tensor-train decomposition in S2VT models to reduce the parameters. First, the impact of parameters of tensor-train factorization on the model performance was investigated systematically. After that, we applied tensor-train decomposition in different layers of a S2VT model to establish 6 tensor-train S2VT models for Chinese sign language recognition. The experimental results demonstrated that when the fully-connected layer and the first LSTM layer in S2VT was represented with tensor-train format, the model could obtain the best performance, remaining high accuracy and reducing parameters and memory significantly. The proposed tensor-train S2VT models can also be applied in other sequence-to-sequence problems to improve the performance.

RNN-Transducer based Chinese Sign Language Recognition

Sign Language Recognition (SLR) targets on interpreting sign language video into natural language, which largely facilitates mutual communication between the deaf and general public. SLR is usually formulated as a sequence alignment problem, wherein connectionist temporal classification (CTC) plays an important role in building effective alignment between video sequence and sentence-level labels. However, CTC-based SLR methods tend to fail if the output label sequence is longer than the input video sequence. Besides, they ignore the interdependencies between output predictions. This paper addresses these two issues and proposes a new RNN-Transducer based SLR framework, i.e., visual hierarchy to lexical sequence alignment network (H2SNet). In the framework, we design a visual hierarchy transcription network to capture the spatial appearance and temporal motion cues of sign video on multiple levels. Meanwhile, we utilize a lexical prediction network to extract effective contextual information from output predictions. RNN-Transducer is applied to learn the mapping between sequential video features and sentence-level labels. Extensive experiments validate the effectiveness and superiority of our approach over state-of-the-art methods.

Alteration of Cortical and Subcortical Structures in Children With Profound Sensorineural Hearing Loss.

Profound sensorineural hearing loss (SNHL) is an auditory disability associated with auditory and cognitive dysfunction. Due to distinct pathogenesis, some associated structural and functional changes within the brain have been investigated in previous studies, but whole-brain structural alterations are incompletely understood. We extended the exploration of neuroanatomic differences in whole-brain structure in children with profound SNHL who are primarily users of Chinese sign language (CSL). We employed surface-based morphometry (SBM) and subcortical analyses. T1-weighted magnetic resonance images of 26 children with profound SNHL and 27 age- and sex-matched children with normal hearing were analyzed. Compared with the normal control (NC) group, children with profound SNHL showed diverse structural changes in surface-based and subcortical analyses, including decreased cortical thickness in the left postcentral gyrus, superior parietal lobule, paracentral lobule, precuneus, the right transverse temporal gyri, and the middle temporal gyrus; a noticeable increase in the Local Gyrification Index (LGI) in the left precuneus and superior parietal lobule; and diverse changes in gray-matter volume (GMV) in different brain regions. Surface-based vertex analyses revealed regional contractions in the right thalamus, putamen, pallidum, and the brainstem of children with profound SNHL when compared with those in the NC group. Volumetric analyses showed decreased volumes of the right thalamus and pallidum in children with profound SNHL. Our data suggest that children with profound SNHL are associated with diffuse cerebral dysfunction to cortical and subcortical nuclei, and revealed neuroplastic reorganization in the precuneus, superior parietal lobule, and temporal gyrus. Our study provides robust evidence for changes in connectivity and structure in the brain associated with hearing loss.

Fast and robust key frame extraction method for gesture video based on high-level feature representation

In gesture video, the inner-frame difference is too subtle to be projected via low-level features, and the gesture frames, expressing semantic information, are distributed only among the tiny part of the whole video frame. This paper introduces a fast and robust key frame extraction method for gesture video, founded upon high-level feature representation to extract the gesture key frame precisely without affecting the semantic information. Firstly, a gesture video segmentation model is designed by employing SSD, which classify gesture video into the semantic scene and the static scene. And then, the 2D-DWT-based perceptual hash algorithm is studied to extract candidate static key frames. Afterward, the multi-channel gradient magnitude frequency histogram (HGMF-MC) based on improved VGG16 is developed as a new image descriptor. Finally, a key frame extraction mechanism based on HGMF-MC is proposed to generate gesture video summary of two scenes, respectively. Experiments consistently show the superiority of the proposed method on Chinese sign language, Cambridge, ChaLearn and CVRR-Hands gesture datasets. The results demonstrate that the method proposed is effective, which improves the video compression ratio and outperforms the state-of-the-art methods.

중국수어 단어형성에 관한 기술적 연구: 형태·음운론적 특성을 중심으로

This paper aims at investigating morpho-phonological structure on the modality-specific word formation of Chinese Sign Language (CSL) in terms of their modulations in the context of Movement-Position hold interaction. The three major goals are as follows: First, to demonstrate the internal structure of syllables and the existence of morphemes; second, to illustrate the characteristics of six different syllable types; third, to give a description of the phonological processes that occur in morphologically complex words. The results show that most signs that occur in six types of syllables in the CSL data are both monosyllabic and monomorphemic. However, polysyllabic monomorphemic signs and polymorphemic monosyllabic signs also exist in CSL. Finally, an array of phonological processes, including assimilation, coalescence, metathesis, and deletion is observed either on the lexical level or on the post-lexical level in connected signing.

EM-Sign: A Non-Contact Recognition Method Based on 24 GHz Doppler Radar for Continuous Signs and Dialogues

We studied continuous sign language recognition using Doppler radar sensors. Four signs in Chinese sign language and American sign language were captured and extracted by complex empirical mode decomposition (CEMD) to obtain spectrograms. Image sharpening was used to enhance the micro-Doppler signatures of the signs. To classify the different signs, we utilized an improved Yolov3-tiny network by replacing the framework with ResNet and fine-tuned the network in advance. This method can remove the epentheses from the training process. Experimental results revealed that the proposed method can surpass the state-of-the-art sign language recognition methods in continuous sign recognition with a precision of 0.924, a recall of 0.993, an F1-measure of 0.957 and a mean average precision (mAP) of 0.99. In addition, dialogue recognition in three daily conversation scenarios was performed and evaluated. The average word error rate (WER) was 0.235, 10% lower than in of other works. Our work provides an alternative form of sign language recognition and a new approach to simplify the training process and achieve a better continuous sign language recognition effect.

Dialects or languages: A corpus-based quantitative approach to lexical variation in common signs in Chinese Sign Language (CSL)

Based on the quantitative analyses of 78,532 sign tokens generated in the lexical elicitation tasks conducted with 58 deaf participants throughout Mainland China, the paper addresses the issue of regional lexical variation in Chinese Sign Language (CSL). It aims to explore the extent of lexical variation and geographical distributions of sign varieties within the Chinese Deaf community. It is the first corpus-based quantitative research concerning the CSL lexicon supported by the CSL lexical database. It will be shown that the varieties of CSL used in different cities are dialects of the same sign language rather than different sign languages. It will also be suggested that there are only two major regional dialects of CSL. This is surprising considering the large size of both the deaf population and territory within China. This is compared to previous studies concerning other sign languages, which suggest multiple distinctive regional variants within small geographical areas. Variation also exists within the two major dialects, suggesting the existence of sub-groups of regional dialects.