Sign Language Gestures Research Articles

Towards Low-Cost Sign Language Gesture Recognition Leveraging Wearables

Different from traditional gestures, sign language gestures involve a lot of finger-level gestures without wrist or arm movements. They are hard to detect using existing motion sensors-based approaches. We introduce the first low-cost sign language gesture recognition system that can differentiate fine-grained finger movements using the Photoplethysmography (PPG) and motion sensors in commodity wearables. By leveraging the motion artifacts in PPG, our system can accurately recognize sign language gestures when there are large body movements, which cannot be handled by the traditional motion sensor-based approaches. We further explore the feasibility of using both PPG and motion sensors in wearables to improve the sign language gesture recognition accuracy when there are limited body movements. We develop a gradient boost tree (GBT) model and deep neural network-based model (i.e., ResNet) for classification. The transfer learning technique is applied to ResNet-based model to reduce the training effort. We develop a prototype using low-cost PPG and motions sensors and conduct extensive experiments and collect over 7000 gestures from 10 adults in the static and body-motion scenarios. Results demonstrate that our system can differentiate nine finger-level gestures from the American Sign Language with an average recognition accuracy over 98 percent.

СЛОВАРИ КАЗАХСКОГО ЯЗЫКА КАК ОСНОВА СЕМАНТИЧЕСКОГО АНАЛИЗА В СИСТЕМЕ КОМПЬЮТЕРНОГО СУРДОПЕРЕВОДА

his article discusses the development of a semantic dictionary of the Kazakh language for a computer translation system from Kazakh to Kazakh sign language, which will take into account the semantics of the Kazakh language and the Kazakh sign language. The semantic dictionary of the Kazakh language serves as the basis of computer translation technology from the Kazakh language to the Kazakh sign language. In the future, it will allow semantic analysis of the source text. The authors of the article analyzed and selected the available dictionaries of the Kazakh language used in the development of the semantic dictionary database. Dictionaries of the Kazakh language provide an opportunity for computer-based sign language translation of the Kazakh sign language. The article also presents the possibility of using L. S. Dimskis notation to develop a dictionary of the structure of gestures of the Kazakh sign language. The prospect of its inclusion in the database of semantic dictionary is revealed. And also revealed the need for a dictionary of gestures in the development of automated sign language translation system as a whole, taking into account its effectiveness and the possibility of full practical use.

DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition

Hand gesture recognition has attracted the attention of many researchers due to its wide applications in robotics, games, virtual reality, sign language and human-computer interaction. Sign language is a structured form of hand gestures and the most effective communication way among hear-impaired people. Developing an efficient sign language recognition system to recognize dynamic isolated gestures encounters three major challenges, namely, hand segmentation, hand shape feature representation and gesture sequence recognition. Traditional sign language recognition methods utilize color-based hand segmentation algorithms to segment hands, hand-crafted feature extraction for hand shape representation and Hidden Markov Model (HMM) for sequence recognition. In this paper, a novel framework is proposed for signer-independent sign language recognition using multiple deep learning architectures comprising hand semantic segmentation, hand shape feature representation and deep recurrent neural network. The recently developed semantic segmentation method called DeepLabv3+ is trained using a set of pixel-labeled hand images to extract hand regions from each frame of the input video. Then, the extracted hand regions are cropped and scaled to a fixed size to alleviate hand scale variations. Extracting hand shape features is achieved using a single layer Convolutional Self-Organizing Map (CSOM) instead of relying on transfer learning of pre-trained deep convolutional neural networks. The sequence of extracted feature vectors are then recognized using deep Bi-directional Long Short-Term Memory (BiLSTM) recurrent neural network. BiLSTM network contains three BiLSTM layers, one fully connected and softmax layers. The performance of the proposed method is evaluated using a challenging Arabic sign language database containing 23 isolated words captured from three different users. Experimental results show that the performance of proposed framework outperforms with large margin the state-of-the-art methods for signer-independent testing strategy.

A scale space model of weighted average CNN ensemble for ASL fingerspelling recognition

A sign language recognition system facilitates communication between the deaf community and the hearing majority. This paper proposes a novel specialised convolutional neural network (CNN) model, SignNet, to recognise hand gesture signs by incorporating scale space theory to deep learning framework. The proposed model is a weighted average ensemble of CNNs – a low resolution network (LRN), an intermediate resolution network (IRN) and a high resolution network (HRN). Augmented versions of VGG-16 are used as LRN, IRN and HRN. The ensemble works at different spatial resolutions and at varying depths of CNN. The SignNet model was assessed with static signs of American Sign Language – alphabets and digits. Since there exists no sign dataset for deep learning, the ensemble performance is evaluated on the synthetic dataset which we have collected for this task. Assessment of the synthetic dataset by SignNet reported an impressive accuracy of over 92%, notably superior to the other existing models.

Spatiotemporal spectral histogramming analysis in hand gesture signature recognition

Dynamic signature recognition emerges to perfectly solve the hygiene concern due to its no-contact characteristic. Nevertheless, the recognition of dynamic texture is challenging compared with the static signature image due to their unknown spatial and temporal nature. In this work, we present a multi-view spatiotemporal approach based on spectral histogramming for hand gesture signature recognition. A Microsoft Kinect sensor is adopted to capture the motion of signing in a sequence of depth frames. The depth frame sequence is viewed from three directional sights to retrieve rich information, such as temporal changes at each spatial location, the signing motion flow of each vertical and horizontal spatial space in a temporal manner. Furthermore, the proposed approach performs feature description on different levels of locality. This function enables a multi-resolution analysis on this dynamic signature. The robustness of the proposed approach is reflected with the promising result by striking the state-of-the-art performance, as substantiated in the empirical results.

Highly Morphology-Controllable and Highly Sensitive Capacitive Tactile Sensor Based on Epidermis-Dermis-Inspired Interlocked Asymmetric-Nanocone Arrays for Detection of Tiny Pressure.

The tactile sensor lies at the heart of electronic skin and is of great importance in the development of flexible electronic devices. To date, it still remains a critical challenge to develop a large-scale capacitive tactile sensor with high sensitivity and controllable morphology in an economical way. Inspired by the interlocked microridges between the epidermis and dermis, herein, a highly sensitive capacitive tactile sensor by creating interlocked asymmetric-nanocones in poly(vinylidenefluoride-co-trifluoroethylene) film is proposed. Particularly, a facile method based on cone-shaped nanoporous anodized aluminum oxide templates is proposed to cost-effectively fabricate the highly ordered nanocones in a controllable manner and on a large scale. Finite-element analysis reveals that under vertical forces, the strain/stress can be highly strengthened and localized at the contact apexes, resulting in an amplified variation of film permittivity and thickness. Benefiting from this, the developed tactile sensor presents several conspicuous features, including the maximum sensitivity (6.583 kPa-1 ) in the low pressure region (0-100 Pa), ultralow detection limit (≈3 Pa), rapid response/recovery time (48/36 ms), excellent stability and reproducibility (10 000 cycles). These salient merits enable the sensor to be successfully applied in a variety of applications including sign language gesture detection, spatial pressure mapping, Braille recognition, and physiological signal monitoring.

Deep Gesture Video Generation With Learning on Regions of Interest

Generating videos with semantic meaning, such as gestures in sign language, is a challenging problem. The model should not only learn to generate videos with realistic appearance, but also take notice of crucial details in frames to convey precise information. In this paper, we focus on the problem of generating long-term gesture videos containing precise and complete semantic meanings. We develop a novel architecture to learn the temporal and spatial transforms in regions of interest, i.e., gesticulating hands or face in our case. We adopt a hierarchical approach for generating gesture videos, by first making predictions on future pose configurations, and then using the encoder-decoder architecture to synthesize future frames based on the predicted pose structures. We develop the scheme of action progress in our architecture to represent how far the action has been performed during its expected execution, and to instruct our model to synthesize actions with various paces. Our approach is evaluated on two challenging datasets for the task of gesture video generation. Experimental results show that our method can produce gesture videos with more realistic appearance and precise meaning than the state-of-the-art video generation approaches.

Interpretation of Formal Semantics from Hand Gesture to Text using Proficient Contour Tracing Technique

As an assorted nation with numerous religion tongues, India has attempted to embrace an official, institutionalized gesture based communication. Where as in Indo-Pakistani communication through signing, is viewed as the prevalent sort utilized in South Asia. Many who are deaf or hard of hearing rely on sign language, to communicate. However the estimation of sign language are very unsophisticated and definitions of what counts as proficiency that varies depends on many factors. There are many existing systems which use shape parameters like orientation, palm centroid ,data gloves with 5 accelerometer sensors , and optical markers which reflect infrared light to recognise hand gestures of sign language. Background subtraction techniques used in these systems are K-means clustering ,boundary counters, Eigen backgrounds using Eigen values and wireless technology and bluetooth for connecting software for transmitting recognised hand gesture signals. They are not cost effective but, the accuracy is not met to the need. Whereas, In our proposed system we concentrate mainly to convert hand gestures to text using contour tracing technique to recognise hand gestures using normal webcam. The semantics are classified by support vector machine with trained datasets. The recognised hand gestures are displayed as text. Our main objective is to resolve the problem of facing interviewer for vocally impaired individuals. This helps them to build their confidence and eradicate their inferiority complex compared to other methods

Design and Implementation of Smart System for Interaction Between Blind, Deaf and Mute People

Difficulties faced by the deaf and dumb people and the blind people to communicate among themselves is the major motivation for the proposal of this project. The project aims at bridging this communication gap by means of an cost-effective electronic device. This is done by using a prototype hand worn glove which converts hand gestures and Braille codes into speech and text, thus enabling speech-impaired and visually impaired people to effectively communicate with everyone. MEMS sensors are used to recognize the gesture sign given by the deaf and dumb people. The gesture input is sent as an audio file which is understandable to the people. The input given by the blind people is in Braille language which is converted to text using scale-invariant feature transform (SIFT) algorithm and decoded to text and voice output through Artificial Neural Networks (ANN) by means of Back Propagation using Supervised Learning. So this ensures two way communication between blind, deaf and dumb people.

Sign Language Interpreter using Kinect Motion Sensor using Machine Learning.

Sign language is the route through which deaf and dumb people usually communicate with one another. It is seen that, impaired people find a difficulty while interacting with the society. Normal individuals are not able to understand their sign language. To bridge this gap, the proposed system acts as the mediator between impaired and normal people. This system uses Kinect motion sensor to capture the signs. The Kinect motion sensor captures 3 dimentional dynamic gestures. Thus this study is proposed for extracting features of dynamic gestures of Indian Sign Language (ISL). As American Sign Language (ASL) is popular and is used in research and development field, on the other hand ISL has been recently standardized and hence ISL recognition is less explored. The propose method extracts feature from the sign and converts it to the intended textual form. The method then integrates local as well as global information of the signs. This integrated feature leads to improvizing the system performance , the system serves as an aid to disabled people. Its application includes hospitals, government sectors and some multinational companies.

Arabic Sign Language Translator

Development of systems that can recognize the gestures of Arabic Sign language (ArSL) provides a method for hearing impaired to easily integrate into society. This paper aims to develop a computational structure for an intelligent translator to recognize the isolated dynamic gestures of the ArSL. In our proposed system we build a datasets for ArSL from scratch of, we used 100-sign vocabulary from ArSL, we have applied 1500 video files for these signs. These signs were divided into five types of signs, recognizing a sign language gestures from dynamic gestures could be a difficult analysis issue. This paper solves the problem using gradient based key frame extraction technique. These key frames are useful for splitting continuous language gestures into sequence of signs for removing uninformative frames. After splitting of gestures every sign has been treated as isolated gesture. Then features of pre-processed gestures are extracted using Intensity Histogram by integrating with Gray Level Co-occurrence Matrix (GLCM) features. Experiments are performed on our own ArSL dataset and the matching between the ArSL and Arabic text is tested by Euclidian distance. The evaluation of the proposed system for the automatic recognition and translation for isolated dynamic ArSL gestures has proven to be effective and highly accurate. The experimental results show that the proposed system recognizes signs with a precision of 95.8%.

Drawing Spirits in the Sand: Performative Storytelling in the Digital Age

For First Nations people living in the central desert of Australia, the performance of oral storytelling drawing in the sand drives new agency in the cultural metamorphosis of communication practices accelerated by the proliferation of portable digital devices. Drawing on the ground sustains the proxemic and kinesthetic aspects of performative storytelling as a sign gesture system. When rendering this drawing supra-language, the people negotiate and ride the ontological divide symbolized by traditional elders in First Nations communities and digital engineers who program and code. In particular, storytelling’s chronemic encounter offsets the estrangement of the recorded event and maintains every participants’ ability to shape identity and navigate space-time relationships. Drawing storytelling demonstrates a concomitant capacity to mediate changes in tradition and spiritual systems. While the digital portals of the global arena remain open and luring, the force enabled by the chiasmic entwinement of speech, gesture and sand continues to map the frontier of First Nations identity formation and reformation.

A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion

Sign language facilitates communication between hearing impaired peoples and the rest of the society. A number of sign language recognition (SLR) systems have been developed by researchers, but they are limited to isolated sign gestures only. In this paper, we propose a modified long short-term memory (LSTM) model for continuous sequences of gestures or continuous SLR that recognizes a sequence of connected gestures. It is based on splitting of continuous signs into sub-units and modeling them with neural networks. Thus, the consideration of a different combination of sub-units is not required during training. The proposed system has been tested with 942 signed sentences of Indian Sign Language (ISL). These sign sentences are recognized using 35 different sign words. The average accuracy of 72.3% and 89.5% has been recorded on signed sentences and isolated sign words, respectively.

Higher Order Feature Extraction and Selection for Robust Human Gesture Recognition using CSI of COTS Wi-Fi Devices.

Device-free human gesture recognition (HGR) using commercial off the shelf (COTS) Wi-Fi devices has gained attention with recent advances in wireless technology. HGR recognizes the human activity performed, by capturing the reflections of Wi-Fi signals from moving humans and storing them as raw channel state information (CSI) traces. Existing work on HGR applies noise reduction and transformation to pre-process the raw CSI traces. However, these methods fail to capture the non-Gaussian information in the raw CSI data due to its limitation to deal with linear signal representation alone. The proposed higher order statistics-based recognition (HOS-Re) model extracts higher order statistical (HOS) features from raw CSI traces and selects a robust feature subset for the recognition task. HOS-Re addresses the limitations in the existing methods, by extracting third order cumulant features that maximizes the recognition accuracy. Subsequently, feature selection methods derived from information theory construct a robust and highly informative feature subset, fed as input to the multilevel support vector machine (SVM) classifier in order to measure the performance. The proposed methodology is validated using a public database SignFi, consisting of 276 gestures with 8280 gesture instances, out of which 5520 are from the laboratory and 2760 from the home environment using a 10 × 5 cross-validation. HOS-Re achieved an average recognition accuracy of 97.84%, 98.26% and 96.34% for the lab, home and lab + home environment respectively. The average recognition accuracy for 150 sign gestures with 7500 instances, collected from five different users was 96.23% in the laboratory environment.

Indonesian sign language recognition using kinect and dynamic time warping

Sign Language Recognition System (SLRS) is a system to recognise sign language and then translate them into text. This system can be developed by using a sensor-based technique. Some studies have implemented various feature extraction and classification methods to recognise sign language in the different country. However, their systems were user dependent (the accuracy was high when the trained and the tested user were the same people, but it was getting worse when the tested user was different to the trained user). Therefore in this study, we proposed a feature extraction method which is invariant to a user. We used the distance between two users’ skeleton instead of using the users’ skeleton positions because the skeleton distance is independent to the user posture. Finally, forty-five features were extracted in this proposed method. Further, we classified the features by using a classification method that is suitable with sign language gestures characteristic (time-dependent sequence data). The classification method is Dynamic Time Wrapping. For the experiment, we used twenty Indonesian sign languages from different semantic groups (greetings, questions, pronouns, places, family and others) and different gesture characteristic (static gesture and dynamic gesture). Then the system was tested by a different user with the user who did the training. The result was promising, this proposed method produced high accuracy, reach 91% which shows that this proposed method is user independent.

Multi modal spatio temporal co-trained CNNs with single modal testing on RGB–D based sign language gesture recognition

Abstract Extracting hand movements using single RGB video camera for sign language recognition is a necessary attribute in developing an automated sign language recognition system. Local spatio temporal methods has shown encouraging outcomes for hand extraction using color cues. However, the color intensities does not behave as an independent entity during video capture in real environments. This has become a roadblock in the development of sign language machine translator for processing video data in real world environments. Not surprisingly, the result is more accurate when additional information is provided in the form of depth for sign language recognition in real environments. In this paper, we make use of a multi modal feature sharing mechanism with a four-stream convolutional neural network (CNNs) for RGB – D based sign language recognition. Unlike the multi stream CNNs, where output class prediction is based on independently operated two or three modal streams due to scale variations, we propose a feature sharing multi stream CNN on multi modal data for sign language recognition. The proposed 4 – stream CNN divides into two input data groupings under the training and testing spaces. The training space uses four inputs: RGB spatial in main stream and depth spatial, RGB and depth temporal on Region of interest mapping (ROIM) stream. The testing space uses only RGB and RGB temporal data for prediction from the trained model. The ROIM stream shares the multi modal data to generate ROI maps of the human subject, which are used to regulate the feature maps in RGB stream. The scale variations in the three streams is managed by translating the depth map to fit the RGB data. Sharing of multi modal features with RGB spatial features during training has circumvented overfitting on RGB video data. To validate the proposed CNN architecture, the accuracy of the classifier is investigated with RGB-D sign language data and three benchmark action datasets. The results show a remarkable behaviour of the classifier in handling missing depth modal during testing. The robustness of the system against state – of – the – art action recognition methods is studied using contrasting datasets.

Multimodal Spatiotemporal Networks for Sign Language Recognition

Different from other human behaviors, sign language has the characteristics of limited local motion of upper limb and meticulous hand action. Some sign language gestures are ambiguous in RGB video due to the influence of lighting and background color, which affects the recognition accuracy. We propose a multimodal deep learning architecture for sign language recognition which effectively combines RGB-D input and two-stream spatiotemporal networks. Depth videos, as an effective compensation of RGB input, can supply additional distance information about the signer's hands. A novel sampling method called ARSS (Aligned Random Sampling in Segments) is put forward to select and align optimal RGB-D video frames, which improves the capacity utilization of multimodal data and reduces the redundancy. We get the hand ROI by joints information of RGB data for local focus in spatial stream. D-shift Net is proposed as depth motion feature extraction in temporal stream, which fully utilizes three dimensional motion information of the sign language. Both streams are fused by convolutional fusion layer to get complementary features. Our approach explored the multimodal information and enhanced the recognition precision. It obtains the state-the-of-art performance on the datasets of CSL (96.7%) and IsoGD (63.78%).

Real time sign language recognition using depth sensor

Communication via gestures is a visual dialect utilised by deaf and hard-of-hearing (HoH) people group. This paper proposed a system for sign language recognition utilising human skeleton data provided from Microsoft's Kinect sensor to recognising sign gestures. The Kinect sensor generates the skeleton of a human body and distinguishes 20 joints in it. The proposed method utilises 11 out of 20 joints and extracts 35 novel features per frame, based on distances, angles and velocity involving upper body joints. Multi-class support vector machine classified the 35 Indian sign gestures in real time with accuracy of 87.6%. The proposed method is robust in cluttered environment and viewpoint variation.

Real time sign language recognition using depth sensor

Communication via gestures is a visual dialect utilised by deaf and hard-of-hearing (HoH) people group. This paper proposed a system for sign language recognition utilising human skeleton data provided from Microsoft's Kinect sensor to recognising sign gestures. The Kinect sensor generates the skeleton of a human body and distinguishes 20 joints in it. The proposed method utilises 11 out of 20 joints and extracts 35 novel features per frame, based on distances, angles and velocity involving upper body joints. Multi-class support vector machine classified the 35 Indian sign gestures in real time with accuracy of 87.6%. The proposed method is robust in cluttered environment and viewpoint variation.

Real-time glove and android application for visual and audible Arabic sign language translation

Researchers can develop new systems to capture, analyze, recognize, memorize and interpret hand gestures with machine learning and sensors. Acoustic communication is a way to convey human opinions, feelings, messages, and information. Deaf and mute individuals communicate using sign language that is not understandable by everyone. Unfortunately, they face extreme difficulty in conveying their messages to others. To facilitate the communication between deaf/mute individuals and normal people, we propose a real-time prototype using a customized glove equipped with five flex and one-accelerometer sensors. These sensors are able to detect the bindings of the fingers and the movements of the hand. In addition, we developed an android mobile application to recognize the captured Arabic Sign Language (ArSL) gestures and translate them into displayed texts and audible sounds. The developed prototype is accurate, low cost and fast in response.