Thai Sign Research Articles

Deep multimodal-based finger spelling recognition for Thai sign language: a new benchmark and model composition

Deep multimodal-based finger spelling recognition for Thai sign language: a new benchmark and model composition

Using LSTM to translate Thai sign language to text in real time

Between 2019 and 2022, as the Covid-19 pandemic unfolded, numerous countries implemented lockdown policies, leading most corporate companies to permit employees to work from home. Communication and meetings transitioned to online platforms, replacing face-to-face interactions. This shift posed challenges for deaf or hearing-impaired individuals who rely on sign language, using hand gestures for communication. However, it also affected those who can hear clearly but lack knowledge of sign language. Unfortunately, many online meeting platforms lack sign language translation features. This study addresses this issue, focusing on Thai sign language. The objective is to develop a model capable of translating Thai sign language in real-time. The Long Short-Term Memory (LSTM) architecture is employed in conjunction with MediaPipe Holistic for data collection. MediaPipe Holistic captures keypoints of hand, pose, and head, while the LSTM model translates hand gestures into a sequence of words. The model’s efficiency is assessed based on accuracy, with real-time testing achieving an 86% accuracy, slightly lower than the performance on the test dataset. Nonetheless, there is room for improvement, such as expanding the dataset by collecting data from diverse individuals, employing data augmentation techniques, and incorporating an attention mechanism to enhance model accuracy.

E-Learning Model to Identify the Learning Styles of Hearing-Impaired Students

Deaf students apparently experience hardship in conventional learning; however, despite their inability to hear, nothing can stop them from reading. Although they perform impressively in memorizing the information, their literacy and reading capability still appear to be weak since they lack the chance to revise by listening and practicing repetitively. Currently, the teaching media for deaf students are quite rare and inadequate, forcing them to face difficulties in integrating new knowledge, even though most of the contents are in a form of written, printed, downloaded, or even accessible via an e-learning platform. However, it is crucial to bear in mind that each learner is different. There is evidence showing that some learners prefer particular methods of learning, also known as learning preferences or learning styles. Thus, the present study reports the sequence of learning styles obtained by using a modified VRK + TSL model that categorized students based on their learning styles. We also propose four different ways of teaching using content-adaptive learning styles, namely visual, reading/writing, kinesthetic, and Thai sign language. Based on personal preferences and the principle of universal design under synthesized learning, an e-learning model was developed to identify deaf learners’ learning styles. The objective is to provide e-learning to identify the learning styles of hearing-impaired students and to respond with up-to-date e-learning materials that can be used anywhere and at any time. These materials must support the education of deaf students. As a result, learners have increased efficiency and increased learning outcomes.

Deep transfer learning base on sequenced edge grid image technique for sign language recognition

Sign language is a visual-gestural language used by hearing impaired person, they modality the gesture to convey meaning. The main problem with sign language communication is ordinary people do not understand the sign language. Therefore, sign language is one of the challenging problems in machine learning. In this paper, researchers focus on visual-based methods and optimize the data preprocessing apply with existing sign language resources. Researchers propose an innovative technique for video processing called Sequenced Edge Grid Images (SEGI) for sign language recognition to interpret hand gesture, body movement, and facial expression. Researchers collected several of sign language data from the internet, the data including Thai sign language utilize in everyday life. The proposed technique was implemented with a convolutional neural network (CNN). The experiments showed SEGI with CNN has increases test accuracy rate with approximately 11% compared to static hand gesture images. Finally, researchers discovered a CNN structure suitable for dataset and examination data by transferring a pre-trained CNN. The fine-tuning with SEGI technique improved 99.8%, thus highest among all the methods. From the results data-preprocesses technique of dataset generation and deep transfer learning was an effective way to improve the accuracy of sign language recognition.

Adapting Fleming-Type Learning Style Classifications to Deaf Student Behavior

This study presents the development of a novel integrated data fusion and assimilation technique to classify learning experiences and patterns among deaf students using Fleming’s model together with Thai Sign Language. Data were collected from students with hearing disabilities (Grades 7–9) studying at special schools in Khon Kaen and Udon Thani, Thailand. This research used six classification algorithms with data being resynthesized and improved via the application of feature selection, and the imbalanced data corrected using the synthetic minority oversampling technique. The collection of data from deaf students was evaluated using a 10-fold validation. This revealed that the multi-layer perceptron algorithm yields the highest accuracy. These research results are intended for application in further studies involving imbalanced data problems.

The Performance of Thai Sign Language Recognition with 2D Convolutional Neural Network Based on NVIDIA Jetson Nano Developer Kit

Thai Sign Language Recognition is a Thai Sign Language learning computer recognition. The system constructs an architecture of T-SLR by TSR- 2DCNN based on NVIDIA Jetson Nano Developer Kit. It is a novelty of automatic translation TSL innovation and reveals the performance of feature extraction and classification to reduce crashed system, overloaded or automatic reboot while complicated processing occurs. The dataset contains 7 gestures in TSL, training images are 7,000 images and validation images are 700 images. The result compares with many techniques as shown that TSR-2DCNN can increase the performance of TSLR in real-time, effectiveness with an accuracy of 0.9914 and loss of 0.03537.

Superresolution Reconstruction in Automatic Thai Sign Language Feature Extraction Using Adaptive Triangulation Interpolation

Superresolution is an image processing technique for improving image quality and enhancing low-resolution images. This paper presents a novel interpolation method for increasing superresolution reconstruction effectiveness using a triangulation interpolation algorithm for automatic Thai sign language feature extraction. This approach uses three neighboring pixels for estimation. The experiment compared the superresolution reconstruction performance using the triangulation interpolation algorithm and the nearest-neighbor, SRCNN, bilinear, bicubic, GPR, and NEDI methods. The super-resolution reconstruction using the triangulation-improved interpolation technique provided the best PSNR measurements of image quality between the original and superresolution-reconstructed images. The PSNR value of the sign language image was 40.608, improving performance by 13.15%. The Thai sign language gesture recognition using 2D convolutional neural networks showed that the designed model increased the gesture recognition effectiveness with an accuracy of 0.95 and loss of 0.14. Thus, this study provides state-of-the-art superresolution reconstruction for automatic Thai sign language gesture recognition.

THAI SIGN LANGUAGE TO ENGLISH TRANSLATION USING VARIABLE HIDDEN NEURON ANN - A CASE STUDY

Sign language translation has been a major challenge in all walks of life. The current society has been more accepting of the specially abled and the government has been actively making policy changes to accommodate and assimilate the specially abled into the society. Every country has made a conscious effort to develop its own syllable set in its native language even though globally used language is American Sign Language (ASL). In this paper a method proposed by the authors for ASL is applied on Thai Sign Language and the working of the ANN model is explored.

Thai Sign Language Image Recognition for the Hearing-Impaired using Radial Inverse Force Histogram combined with Max-Min Boundary

Sign language image recognition is also a very interesting research topic. Because it can be applied to help normal people understand and use it as a communication tool for the hearing impaired. The objectives of this research were to: 1) study and analyze Thai Sign Language image recognition image data for hearing impaired, 2) develop Thai Sign Language image recognition system for hearing impaired by using new techniques, and 3) measure the efficiency of Thai sign language image recognition for hearing impaired using Radial Inverse Force Histogram and the Maximum and Minimum boundary values. The results of the research were as follows: 1) The study and analysis of image data of Thai Sign Language Image Recognition In this research, 62,694 sign language images were used, divided into 2 parts: 1) American Sign Language images, which consisted of 36 groups of images, namely 26 groups of letters (AZ) and 10 groups of numbers (0-9), and Part 2) Picture of Thai Sign Language consisting of 61 groups of images, including 44 groups of letters (ก-ฮ), 7 groups of vowels and 10 groups of numbers (0-9). Each group of pictures is rotated, enlarged, and Image promotion There were 6 sub-groups of images in various forms, divided into 2 parts: 70% of the images for training and 30% of the images for testing. 3) The results of measuring the efficiency of image recognition. It is divided into two parts: American Sign Language Image Recognition and Thai Sign Language Image Recognition. Compared with the Angular histogram method, the mean image accuracy was 0.86, the recall of the mean American sign language was 0.91, and the accuracy of the Thai Sign Language was 0.78. The recognition performance for Thai Sign Language images averaged 0.89, while the recognition efficiency was achieved when using radial inverse force histograms in combination with image similarity measurements with maximum-minimum boundary values. Accuracy for Mean American Sign Language was 0.99 and Remembrance for Mean American Sign was 1.00, while Accuracy for Mean Thai Sign Language was 0.89. Mean Remembrance for Thai Sign Language was 0.96. The results of the visual recognition performance measurement of both the American Sign Language and the Thai Sign Language images were very good compared to the Angular Histogram method.

Automatic Thai Finger Spelling Transcription

This article explores a transcription of a video recording Thai Finger Spelling (TFS)—a specific signing mode used in Thai sign language—to a corresponding Thai word. TFS copes with 42 Thai alphabets and 20 vowels using multiple and complex schemes. This leads to many technical challenges uncommon in spelling schemes of other sign languages. Our proposed system, Automatic Thai Finger Spelling Transcription (ATFS), processes a signing video in 3 stages: ALS marking video frames to easily remove any non-signing frame as well as conveniently group frames associating to the same alphabet, SR classifying a signing image frame to a sign label (or its equivalence), and SSR transcribing a series of signs into alphabets. ALS utilizes the TFS practice of signing different alphabets at different locations. SR and SSC employ well-adopted spatial and sequential models. Our ATFS has been found to achieve Alphabet Error Rate (AER) 0.256 (c.f. 0.63 of the baseline method). In addition to ATFS, our findings have disclosed a benefit of coupling image classification and sequence modeling stages by using a feature or penultimate vector for label representation rather than a definitive label or one-hot coding. Our results also assert the necessity of a smoothening mechanism in ALS and reveal a benefit of our proposed WFS, which could lead to over 15.88 % improvement. For TFS transcription, our work emphasizes the utilization of signing location in the identification of different alphabets. This is contrary to a common belief of exploiting signing time duration, which are shown to be ineffective by our data. HIGHLIGHTS Prototype of Thai finger spelling transcription (transcribing a signing video to alphabets) Utilization of signing location as cue for identification of different alphabets Disclosure of a benefit of coupling image classification and sequence modeling in signing transcription Examination of various frame smoothing techniques and their contributions to the overall transcription performance GRAPHICAL ABSTRACT

Estimation of Hidden Neuron Requisite for Predictive Conversion from Sign Language to a Formal Language

Using sign language for communication has existed since millennia and has evolved over the ages to support the hearing impaired to a great extent. There have been numerous wonderful inventions and innovations aimed at improving the living conidtions of the hearing or speech impaired using technology which was contemporary. However, till date there exists no standard that can be shown as a single universal sign language translation system. This is partially due to the research happening in small pockets and lack of technology penetration in some countries. Largely the efforts to design translation are concentrated on Americal Sign Language (ASL) and Thai Sign Language (TSL). Other than the two, Indonasian Sign Language (SIBI) is the front runner in standardising a translation system. In our previous paper we had attempted to develop an ANN based translator for ASL. The system performed as per expectations. However, since the number of hidden neurons was taken as a variable, the estimation still remained a challenge.In this paper, we have developed a system to estimate the number of hidden neurons required for the accurate translation of ASL

การพัฒนาแบบทดสอบทักษะการใช้ภาษามือไทยของเด็กที่มีความบกพร่องทางการได้ยิน (A Development of the Thai Sign Language Skills Test for Deaf Children)

Thai Abstract: การวจยนมวตถประสงค เพอ พฒนาแบบทดสอบทกษะการใชภาษามอของเดกทมความบกพรองทางการไดยน โดยตวอยางในการวจย คอ เดกทมความบกพรองทางการไดยนทศกษาอยในชนประถมศกษาปท 3 -6 จานวน 30 คนผลการวจย พบวา แบบทดสอบทกษะการใชภาษามอของเดกทมความบกพรองทางการไดยนมจานวน 10 ชด มคาความยากระหวาง.20-.80 และมคาความเทยงทง ฉบบ เทากบ.706 และมคาสหสมพนธระหวางขอคาถามกบคะแนนรวมทง ฉบบ ระหวาง.254-.499 และมคาความตรงตามสภาพ ดวยการนาคะแนนทกษะการใชภาษามอไทยของเดกไปหาความสมพนธกบเกรดเฉลยสะสมของเดก ดวยสมประสทธส หสมพนธแบบเพยรสน เทากบ.707 ทระดบนยสา คญทางสถต.01 แสดงใหเหนวาแบบทดสอบทกษะการใชภาษามอไทยของเดกทมความบกพรองทางการไดยนมความเหมาะสมทจะนาไปใชได English Abstract: The purpose of this research was developed the Thai sign language skills test for deaf children. Sample were 30 deaf children who studied in school for the deaf, Thailand. The result shown that the Thai sign language skills test comprised 10 items which difficulty index between .20-.80 reliability =.706, corrected item-total correlation: CITC between.254-.499, and concurrent validity by using Pearson’s correlation between score of Thai sign language skills and GPAX (r =.707, p<.01). The Thai sign language skills test is appropriate to be used.

A Novel String Grammar Unsupervised Possibilistic C-Medians Algorithm for Sign Language Translation Systems

Sign language is a basic method for solving communication problems between deaf and hearing people. In order to communicate, deaf and hearing people normally use hand gestures, which include a combination of hand positioning, hand shapes, and hand movements. Thai Sign Language is the communication method for Thai hearing-impaired people. Our objective is to improve the dynamic Thai Sign Language translation method with a video captioning technique that does not require prior hand region detection and segmentation through using the Scale Invariant Feature Transform (SIFT) method and the String Grammar Unsupervised Possibilistic C-Medians (sgUPCMed) algorithm. This work is the first to propose the sgUPCMed algorithm to cope with the unsupervised generation of multiple prototypes in the possibilistic sense for string data. In our experiments, the Thai Sign Language data set (10 isolated sign language words) was collected from 25 subjects. The best average result within the constrained environment of the blind test data sets of signer-dependent cases was 89–91%, and the successful rate of signer semi-independent cases was 81–85%, on average. For the blind test data sets of signer-independent cases, the best average classification rate was 77–80%. The average result of the system without a constrained environment was around 62–80% for the signer-independent experiments. To show that the proposed algorithm can be implemented in other sign languages, the American sign language (RWTH-BOSTON-50) data set, which consists of 31 isolated American Sign Language words, is also used in the experiment. The system provides 88.56% and 91.35% results on the validation set alone, and for both the training and validation sets, respectively.

Hand Gesture Recognition for Thai Sign Language in Complex Background Using Fusion of Depth and Color Video

Hand detection and gesture recognition are the active research area in the computer vision. The main purpose to develop the sign language recognition and Human Computer Interaction (HCI). This article investigates and develops the technique to recognize hand posture of Thai sign language in a complex background using fusion of depth and color video. The new technology of sensors, such as the Microsoft Kinect, recently provides the depth video which helps researchers to find the hand position in the scene. This advantage is used to segment the hand sign in the color video without the environment interference such as skin color background. The histograms of oriented gradients are used to extract the image features of hand sign. These features are then pass to the artificial neural network for training and recognition. The result showed that the proposed method is robust to detect the hand gestures in the complex background. It provides the accuracy recognition for the Thai fingerspelling of 84.05%

Thai sign language translation using Scale Invariant Feature Transform and Hidden Markov Models

Visual communication is important for a deft and/or mute person. It is also one of the tools for the communication between human and machines. In this paper, we develop an automatic Thai sign language translation system that is able to translate sign language that is not finger-spelling sign language. In particular, we utilize Scale Invariant Feature Transform (SIFT) to match a test frame with observation symbols from keypoint descriptors collected in the signature library. These keypoint descriptors are computed from several keyframes that are recorded at different times of day for several days from five subjects. Hidden Markov Models (HMMs) are then used to translate observation sequences to words. We also collect Thai sign language videos from 20 subjects for testing. The system achieves approximately 86–95% for signer-dependent on the average, 79.75% for signer-semi-independent (same subjects used in the HMM training only) on the average and 76.56% for signer-independent on the average. These results are from the constrained system in which each signer wears a shirt with long sleeves in front of dark background. The unconstrained system in which each signer does not wear a long-sleeve shirt in front of various natural backgrounds yields a good result of around 74% on the average on the signer-independent experiment. The important feature of the proposed system is the consideration of shapes and positions of fingers, in addition to hand information. This feature provides the system ability to recognize the hand sign words that have similar gestures.

A New Design of the Intelligent Sign Language Interpreters

According to the declaration of human rights (UN) and the constitution B.E. 2540 of the Kingdom of Thailand and the present constitution, the disabled in hearing have the rights to study in the educational institutes of general persons. When the disabled in hearing study in higher levels, they find the problems of the sign language interpreters and other facilities, which affect their education. So several agencies and educational institutes try to solve these problems by inventing the Thai sign language interpreter machine and Dinsow robot, which the intelligent robot can interpret the sign languages into words. But nobody has developed the smaller communicative machine which can take with oneself and communicate two ways. In this paper, the concept of the new intelligent sign language interpreters has been developed with only two pieces of equipments, that composes of the braintooth and the mobile reader work together to communicate between teacher with learners.

SignMT: An alternative language learning tool

Learning a second language is very difficult, especially, for the disabled; the disability may be a barrier to learn and to utilize information written in text form. We present the SignMT, Thai sign to Thai machine translation system, which is able to translate from Thai sign language into Thai text. In the translation process, SignMT takes into account the differences between Thai and Thai sign language in terms of both syntax and semantic to ensure the accuracy of translation. SignMT was designed to be not only an automatic interpreter but also a language learning tool. It provides meaning of each word in both text and image forms which is easy to understand by the deaf. The grammar information and the order of the sentence are presented in order to help the deaf in learning Thai, their second language. With SignMT, deaf students are less dependent on a teacher, have more freedom to experiment with their own language, and improve their knowledge and learning skill.In our experiment, SignMT was implemented to translate sentences/phrases which were collected from different sources including textbooks, cartoons, bedtime story, and newspapers. SignMT was tested and evaluated in terms of the translation accuracy and user satisfaction. The evaluation results show that the translation accuracy is acceptable, and it satisfies the users’ needs.

Intelligent Thai text – Thai sign translation for language learning

We present the Intelligent Thai text – Thai sign translation for language learning (IT 3STL). IT 3STL is able to translate Thai text into Thai sign language simply and conveniently anytime, anywhere. Thai sign language is the language of the deaf in Thailand. In the translation process, the distinction between Thai text and Thai sign language in both grammar and vocabulary are concerned in each processing step to ensure the accuracy of translation. IT 3STL was designed not only to be an automatic interpreter but also to be a language tutor assistant. It provides meaning of each word and describes the structure formation and word order of the translated sentence. With IT 3STL, the deaf and hearing-impaired are able to enhance their communication ability and to improve their knowledge and learning skills. Moreover IT 3STL has increased motivation and opportunity for them to access multimedia and e-learning. In our initial experiment, IT 3STL was implemented to translate sentences/phrases which were collected from different sources including textbooks, cartoons, bedtime story, newspapers and the public labels. IT 3STL was tested and evaluated in terms of the translation accuracy and user satisfaction. The evaluation results show that the translation accuracy and sign representation are acceptable, and it satisfies the users’ needs.

Virtual sign animated pedagogic agents to support computer education for deaf learners

In Thailand, there is a desperate need to improve the educational opportunities for deaf and hearing-impaired university-aged students. The research described herein is targeted at two aspects of this issue: 1) design and test a virtual sign animated instructional tool for the Thai sign language, and 2) investigate how instructors of university classes can most effectively teach deaf students and assess their performance, particularly by using this tool. Currently, I have completed a thorough review of the literature, constructed a 3-D human model that includes rudimentary facial expressions, and devised a framework to investigate the instructional process.

Sorting Language Archives Out: Digitization and Its Consequences

Author(s): Meeker, Stacey | Abstract: Growing awareness of possibilities created by digital technologies coupled with an increasing concern about endangered languages has led to a wide variety of language revitalization, preservation, and documentation projects. Information professionals, who have played a surprisingly small role in these activities, need to cultivate a greater understanding of the specific needs of groups engaged in digital language projects in order to mediate between digital repositories and users. A review of the data from a field study of an endangered indigenous Thai sign language, Ban Khor Sign, serves as a limit example of the complexity of the documentary interventions grouped under the rubric of “language archive.” The language archive problem demands a total information solution involving informaticists, archivists and museologists, and librarians.