Malaysian Sign Research Articles

Malaysian Sign Language Detection with Convolutional Neural Network for Disabilities

Sign language is the main form of communication used by deaf people. Most of their activities like speaking and learning involved sign language. In Malaysia, instead of a deaf person, only a few people know sign language because it's not part of the co-curriculum. Hence, normal people need to know sign language so that they can communicate with deaf people effectively. However, learning sign language requires a lot of effort and takes years of study. Therefore, this research tends to initiate the sign language image detection system that utilizes the deep learning model. The initial effort involves the development of an image detection system that can identify the alphabet of sign language from A to Z. There will be 26 alphabets that contribute to 26 classes of image detection using the Convolutional Neural Network. 85 images will be collected for each alphabet and the total data will be distributed into 80% and 20% for training and testing the CNN model, respectively. The result showed that the proposed system produced more than 95% detection accuracy of the alphabet based on finger sign gestures. For future works, the proposed system will be evaluated with a more complex dataset in terms of words.

The design of the Deaf in Touch Everywhere (DITE)TM mobile application with Deaf and interpreter communities in Malaysia

Background Ineffective communication with Deaf individuals in healthcare settings has led to poor outcomes including miscommunication, waste, and errors. To help address these challenges, we developed a mobile app, Deaf in Touch Everywhere (DITETM) which aims to connect the Deaf community in Malaysia with a pool of off-site interpreters through secure video conferencing. Objectives The aims of this study were to (a) assess the feasibility and acceptability of measuring unified theory of acceptance and use of technology (UTAUT) constructs for DITETM with the Deaf community and Malaysian sign language (BIM) interpreters and (b) seek input from Deaf people and BIM interpreters on DITETM to improve its design. Methods Two versions of the UTAUT questionnaire were adapted for BIM interpreters and the Deaf community. Participants were recruited from both groups and asked to test the DITE app features over a 2-week period. They then completed the questionnaire and participated in focus group discussions to share their feedback on the app. Results A total of 18 participants completed the questionnaire and participated in the focus group discussions. Ratings of performance expectancy, effort expectancy, facilitating conditions and behavioural intention were high across both groups, and suggestions were provided to improve the app. High levels of engagement suggest that measurement of UTAUT constructs with these groups (through a modified questionnaire) is feasible and acceptable. Conclusions The process of engaging end users in the design process provided valuable insights and will help to ensure that the DITETM app continues to address the needs of both the Deaf community and BIM interpreters in Malaysia.

Application of Machine Learning in Malaysia Sign Language Translation

Sign language serves as a vital form of communication for individuals with hearing impairments, enabling seamless interaction among those who cannot hear. It is a widely used means of communication worldwide, facilitating communication within the deaf community. In Malaysia, Malaysian Sign Language (MySL) prevails as the primary sign language employed by the deaf community. However, sign languages possess unique grammatical rules and structures, making them unfamiliar to hearing individuals, leading to potential misunderstandings and communication barriers. This project aimed at developing a sign language translator capable of translating 24 alphabets based on hand gestures. The system employs a dataset of sign language alphabet images, gathered and trained using the Teachable Machine. To evaluate the translator's performance, response time is thoroughly analyzed. The results indicate that 18 out of the 24 alphabets can be recognized within 5 seconds, displaying promising accuracy. By bridging the communication gap between deaf and hearing individuals, the findings of this study hold substantial potential to enhance interactions and foster better understanding between these two communities. The sign language translator represents a significant step towards inclusive communication and improved accessibility for individuals with hearing impairments

A Quality Assessment Methodology for Sign Language Mobile Apps Using Fusion Of Enhanced Weighted Mobile App Rating Scale (MARS) and Content Expert Standardized Criteria

Mobile sign language apps have drawn a lot of interest recently as a way to minimize communication barriers between hearing people and people with hearing impairments. However, there are issues with the criteria and standards that should be taken into account when developing these apps. This study proposes a set of development criteria for sign language mobile apps and standardizes these criteria using the Fuzzy Delphi approach. Furthermore, the Fuzzy-Weighted Zero Inconsistency (FWZIC) approach is utilized to assign weights to the criteria and establish a ranking order. An initial set of requirements is developed based on the literature review. The Fuzzy Delphi technique is used, involving a panel of experts made up of developers, sign language experts, and users of sign language mobile apps, to assess the validity and reliability of the criteria. The FWZIC technique is used to give the criterion weights and determine their ranking order in order to further improve the decision-making process. The relative relevance of each criterion is determined by the FWZIC technique, which involves expert input and makes use of their knowledge and expertise. A thorough ranking is generated by taking into account the effects of each criterion on several zones, assisting in efficient decision-making during the creation of sign language mobile apps. Six Malaysian Sign Language apps that have been shortlisted are being utilized as a proof of concept to test the idea. The result of 6 apps is obtained based on the final standard criteria, their weights, and rankings.

REAL TIME MALAYSIAN SIGN LANGUAGE TRANSLATOR – TUTURJOM

Malaysian Sign Language (MSL) is a vital language that is used as the primary means of communication between the deaf and mute and hearing communities in Malaysia. Malaysians are currently very unfamiliar with MSL, and the present platform for learning the language is scarce, not to mention the lack of efficacy of the available mobile and web learning tools. Therefore, the aim of this study is to propose software for real-time translation of MSL through a Long Short-Term Memory (LSTM) model. The following paper explains six phases of CRISP-DM in creating the application while looking through various findings and important algorithms. This article also intends to provide a clear understanding of hand gesture and pose recognition, LSTM networks and its uses in understanding sign languages.Keywords: LSTM, Malaysian sign language, CRISP-DM, gesture recognition, sign language recognition

Face Validation of Learning Malaysian Sign Language Questionnaire (L-MySL-Q): Malaysian Sign Language (MySL) pro forma as a third language for adolescents

Malaysian Sign Language (MySL) uses visual-manual modalities for delivering information and communicating involving movements of the finger, hand, arm gestures, and facial expressions. A newly-developed instrument tool: Learning-MySL-Questionnaire (L-MySL-Q) from validated questionnaires [Rogers 1995; ELQ, University of Nottingham, UK] consisting of the diffusion model to gauge willingness and readiness for students to adopt MySL as a third language undergone face validation upon completion of Introduction to MySL Pro Forma leson. Findings showed positive feedback from respondents in comparison to pre-intervention among typical children and adolescents. This comprehensive pro forma inculcates a fun learning process while inculcating patriotism among learners. Keywords: inclusive communication, Malaysian Sign Language, third language, adolescents eISSN: 2398-4287 © 2022. The Authors. Published for AMER ABRA cE-Bs by e-International Publishing House, Ltd., UK. This is an open access article under the CC BYNC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer–review under responsibility of AMER (Association of Malaysian Environment-Behaviour Researchers), ABRA (Association of Behavioural Researchers on Asians/Africans/Arabians) and cE-Bs (Centre for Environment-Behaviour Studies), Faculty of Architecture, Planning & Surveying, Universiti Teknologi MARA, Malaysia. DOI:https://doi.org/10.21834/ebpj.v7i19.3273

Focus Group Discussion (FGD) in adopting Malaysian Sign Language (MySL) as a Third Language among Typical Children and Adolescents

Children with early-onset, severe to profound deafness are more vulnerable to mental health problems than their hearing peers. Communication deprivation is a key risk factor in developmental delays. The key element of the Introduction to MySL pro forma is to create awareness and assess the willingness to adopt a new language among hearing individuals prior to embarking on inclusive education. The objectives of this paper are to explore youths and adolescents’ prior experience and attitudes communicating with persons with hearing difficulties (PWHD), MySL lesson expectations, and adolescents' experience of the MySL pro forma and their attitudes towards communicating with PWHD. Keywords: inclusive communication; Malaysian Sign Language; third language; adolescents eISSN: 2398-4287 © 2022. The Authors. Published for AMER ABRA cE-Bs by e-International Publishing House, Ltd., UK. This is an open-access article under the CC BYNC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of AMER (Association of Malaysian Environment-Behaviour Researchers), ABRA (Association of Behavioural Researchers on Asians/Africans/Arabians), and cE-Bs (Centre for Environment-Behaviour Studies), Faculty of Architecture, Planning & Surveying, Universiti Teknologi MARA, Malaysia. DOI: https://doi.org/10.21834/ebpj.v7i19.3183

Evaluating the Efficiency of CBAM-Resnet Using Malaysian Sign Language

The deaf-mutes population is constantly feeling helpless when others do not understand them and vice versa. To fill this gap, this study implements a CNN-based neural network, Convolutional Based Attention Module (CBAM), to recognise Malaysian Sign Language (MSL) in videos recognition. This study has created 2071 videos for 19 dynamic signs. Two different experiments were conducted for dynamic signs, using CBAM-3DResNet implementing ‘Within Blocks’ and ‘Before Classifier’ methods. Various metrics such as the accuracy, loss, precision, recall, F1-score, confusion matrix, and training time were recorded to evaluate the models’ efficiency. Results showed that CBAM-ResNet models had good performances in videos recognition tasks, with recognition rates of over 90% with little variations. CBAM-ResNet ‘Before Classifier’ is more efficient than ‘Within Blocks’ models of CBAM-ResNet. All experiment results indicated the CBAM-ResNet ‘Before Classifier’ efficiency in recognising Malaysian Sign Language and its worth of future research.

Adaptive Language Processing Unit for Malaysian Sign Language Synthesizer

Language Processing Unit (LPU) is a system built to process text-based data to comply with the rules of sign language grammar. This system was developed as an important part of the sign language synthesizer system. Sign language (SL) uses different grammatical rules from the spoken/verbal language, which only involves the important words that Hearing/Impaired Speech people can understand. Therefore, it needs word classification by LPU to determine grammatically processed sentences for the sign language synthesizer. However, the existing language processing unit in SL synthesizers suffers time lagging and complexity problems, resulting in high processing time. The two features, i.e., the computational time and success rate, become trade-offs which means the processing time becomes longer to achieve a higher success rate. This paper proposes an adaptive Language Processing Unit (LPU) that allows processing the words from spoken words to Malaysian SL grammatical rule that results in relatively fast processing time and a good success rate. It involves n-grams, NLP, and Hidden Markov Models (HMM)/Bayesian Networks as the classifier to process the text-based input. As a result, the proposed LPU system has successfully provided an efficient (fast) processing time and a good success rate compared to LPU with other edit distances (Mahalanobis, Levensthein, and Soundex). The system has been tested on 130 text-input sentences with several words ranging from 3 to 10 words. Results showed that the proposed LPU could achieve around 1.497ms processing time with an average success rate of 84.23% for a maximum of ten-word sentences.

A pattern recognition model for static gestures in malaysian sign language based on machine learning techniques

This work proposes a pattern recognition model for static gestures in Malaysian Sign Language (MSL) based on Machine Learning (ML) techniques. The proposed model is divided into two phases, namely, data acquisition and data processing. The first phase involves capturing the required sign data, such as the shape and orientation of the hand, to construct a sensor-based SL dataset. The dataset is collected using a DataGlove device. This device is used to measure the motions of the fingers and wrists. Sixty-four features represent each sign in the dataset. The collected sensory dataset is cleaned in the second phase by removing redundant data. Then, the features are scaled and normalised to exhibit symmetrical behaviour and eliminate outliers. Then, ten different ML techniques are utilised based on real-time data for SL gesture recognition. Experimental results confirmed the efficacy of the proposed pattern recognition model compared with previous work.

Development of Prayer Mobile Application Software for The Hearing Impaired (Deaf) Based on Malaysian Sign Language

Development of Prayer Mobile Application Software for The Hearing Impaired (Deaf) Based on Malaysian Sign Language

Design suggestions for an mHealth app to facilitate communication between pharmacists and the Deaf: perspective of the Deaf community (HEARD Project).

Deaf and hard-of-hearing (DHH) patients have trouble communicating with community pharmacists and accessing the healthcare system. This study explored the views on a proposed mobile health (mHealth) app in terms of design and features, that will be able to bridge the communication gap between community pharmacists and DHH patients. A community-based participatory research method was utilized. Two focus group discussions (FGDs) were conducted in Malaysian sign language (BIM) with a total of 10 DHH individuals. Respondents were recruited using purposive sampling. Video-recordings were transcribed and analyzed using a thematic approach. Two themes emerged: (I) challenges and scepticism of the healthcare system; and (II) features of the mHealth app. Respondents expressed fears and concerns about accessing healthcare services, and stressed on the need for sign language interpreters. There were also concerns about data privacy and security. With regard to app features, the majority preferred videos instead of text to convey information about their disease and medication, due to their lower literacy levels. For an mHealth app to be effective, app designers must ensure the app is individualised according to the cultural and linguistic diversity of the target audience. Pharmacists should also educate patients on the potential benefits of the app in terms of assisting patients with their medicine-taking.

A mobile learning application for Malaysian sign language education

Malaysian Sign language is the primary means of communication among the deaf-mute community in Malaysia for many years since it was first created in 1998 when the Malaysian Federation of Deaf was first established. Although Malaysian Sign Language is widely used among the deaf-mute community, hearing individuals often have little or no exposure to the usage of sign language due to the lack of pervasive environment for the usage of sign language. There is also a lack of effective mobile learning application that encourages and facilitates the learning of sign language. This creates a communication gap among the deaf-mute community and the hearing individuals. The objective of the project is to design a mobile application that can facilitate the learning of sign language and ultimately bridging the gap between deaf-mute communities and hearing individuals. A survey was carried out to elicit the requirements of potential users. The sample population was drawn from Bahasa Isyarat Malaysia (BIMMFD) Facebook group and Universiti Tenaga Nasional (UNITEN) students. The questionnaire consists of multiple-choice questions, open-ended question and dichotomous questions which can be logically divided into demographic questions, questions on the experience of a participant on sign language, preferred learning methods and the preferred features of the application. The findings show that there is a need for an effective and interactive mobile application for learning sign language. Most of the respondents agreed that video tutorials are their preferred learning method as it is more dynamic than learning via images or books. A majority of the respondents also have no experience in using a sign language learning application due to the lack of availability on the market. Moreover, most of the participants also show interest in using a mobile application in learning Malaysian Sign Language. Some respondents also requested the feature of a built-in dictionary. Thus, the findings validate that SLEM has its demand and potential to provide a more accessible and user-friendly means to learn Malaysian Sign Language. The application has the potential to be commercialized to industries and communities such as sign language learning institutions, the public and individuals who are related to deaf or verbally challenged individuals. This paper provides directions and guidance for further development of this educational mobile application.

Intelligent Malaysian Sign Language Translation System Using Convolutional-Based Attention Module with Residual Network.

The deaf-mutes population always feels helpless when they are not understood by others and vice versa. This is a big humanitarian problem and needs localised solution. To solve this problem, this study implements a convolutional neural network (CNN), convolutional-based attention module (CBAM) to recognise Malaysian Sign Language (MSL) from images. Two different experiments were conducted for MSL signs, using CBAM-2DResNet (2-Dimensional Residual Network) implementing “Within Blocks” and “Before Classifier” methods. Various metrics such as the accuracy, loss, precision, recall, F1-score, confusion matrix, and training time are recorded to evaluate the models' efficiency. The experimental results showed that CBAM-ResNet models achieved a good performance in MSL signs recognition tasks, with accuracy rates of over 90% through a little of variations. The CBAM-ResNet “Before Classifier” models are more efficient than “Within Blocks” CBAM-ResNet models. Thus, the best trained model of CBAM-2DResNet is chosen to develop a real-time sign recognition system for translating from sign language to text and from text to sign language in an easy way of communication between deaf-mutes and other people. All experiment results indicated that the “Before Classifier” of CBAMResNet models is more efficient in recognising MSL and it is worth for future research.

LEGAL AND POLICY BASIS FOR BAHASA ISYARAT MALAYSIA INTERPRETATION SERVICES

There is an increasing need for Malaysian Sign Language interpreters or known as Jurubahasa Isyarat Malaysia (JBIM) to meet the economic, social, and cultural needs of persons who are deaf. To resolve the continuing lack of JBIM, the present paper addresses the legal and policy basis for the establishment of a government-owned and operated service system. We review several legal and government strategic documents, namely the Persons with Disabilities Act 2008, the Local Government Act 1976, the Policy for Persons with Disabilities 2007, the Action Plan for Persons with Disabilities 2016-2022, and the National Community Policy 2018. Although Bahasa Isyarat Malaysia (BIM) is recognized by the law beyond its status as an official language of deaf Malaysians and as part of their cultural identity, our analysis of these legal and policy documents do not suggest a tangible and proper mechanism for BIM interpretation service system. We assert the BIM interpretation services to be established and provided by local governments through collaboration with other federal government agencies and deaf people’s organizations. We also discuss structural measures regarding recruitment and certification, job-related equipment, retirement fund as well as competitive wage. These four aspects are vital in ensuring the effectiveness and sustainability of the BIM interpretation services.

Based on wearable sensory device in 3D-printed humanoid: A new real-time sign language recognition system

This study proposes a new real-time sign recognition system based on a wearable sensory glove, which has 17 sensors with 65 channels. We introduce the DataGlove to recognize various, possibly complex hand gestures of the Malaysian Sign Language (MSL). With 65 data channels, DataGlove can satisfy the requirement suggested by the analysis of hand anatomy, kinematic and gestures. Four groups of sensors were tested to select the optimal sensors that can capture hand gesture information. Also, a 3D-printed humanoid arm is used to validate the sensor mounted on the glove. In an extensive set of experiments to test our system, five well-known MSL participants were chosen to perform 75 gestures, all taken from the MSL numbers, alphabet, and words. The error rate was used to check system performance. Discussion confirms that the proposed system competes well with an advanced benchmark of previous works is up 100% based on 14 criteria in term of the type of captured signals, recognised gestures, and solved issues. Results show that our system is capable of recognizing the wide range of gestures with recognition accuracies of 99%, 96% and 93.4% for numbers, alphabet letters and words of MSL, respectively. This research contributes to enhancing the lifestyle of people with disabilities, and bridge the gap between people with hearing impairment and ordinary people.

Malaysian Sign Language Mobile Learning Application : A recommendation app to communicate with hearing-impaired communities

Malaysian Sign Language (MSL) is an important language that is used as the primary communication method for the deaf communities with the others. Currently, the MSL is poorly known by the Malaysians and the existing platform of learning the sign language is inefficient, not to mention the incomplete functionality of the existing mobile learning application in the market. Hence, the purpose of developing this application is aimed to increase the knowledge and recognition of the public towards the MSL and allows them to learn the sign language more effectively. One of the features in this application is sign detection, which could analyze the image captured by phone camera into sign meaning. The application also comprised various categories of the sign for efficient learning and quiz to test user knowledge against their learned sign language. Besides, there is a feedback module for the user to express their opinions and suggestions towards the application to the developer. This application is aimed to help the public to learn MSL efficiently by selecting the category of sign they wish to learn and test themselves by using the quiz module. Besides, the application could also detect the unknown sign by capturing the image and analyze it. The application helped to raise the recognition of MSL among the public and expose the public to the sign language knowledge. It had also become a small help in breaking the barrier of communication between the deaf communities and the public.

A Real Time Malaysian Sign Language Detection Algorithm Based on YOLOv3

Sign language is a language that involves a movement of hand gestures. It is a medium for the hearing impaired person (deaf or mute) to communicate with others. However, in order to communicate with the hearing impaired person, the communicator has to have knowledge in sign language. This is to ensure that the message delivered by the hearing impaired person is understood. This project proposes a real time Malaysian sign language detection based on the Convolutional Neural Network (CNN) technique utilizing the You Only Look Once version 3 (YOLOv3) algorithm. Sign language images from web sources and recorded sign language videos by frames were collected. The images were labelled either alphabets or movements. Once the preprocessing phase was completed, the system was trained and tested on the Darknet framework. The system achieved 63 percent accuracy with learning saturation (overfitting) at 7000 iterations. Once it is successfully conducted, this model will be integrated with other platform in the future such as mobile application.

Malaysian sign language dataset for automatic sign language recognition system

Hearing impaired individuals have issues to communicate with normal people. They havetheir own language called Sign Language (SL) to express their feeling or to communicatewith others. As communication is an essential part of normal everyday life, it is particularlyimportant for deaf people to communicate as normally as possible with others. Recentadvancements in computing technologies have the potential to be applied in the field of SLrecognition. These computer-based approaches are able to translate the SL into verballanguage and vice-versa. This paper describes the development of a dataset for an automatedSL recognition system based on the Malaysian Sign Language (MSL). Implementation resultsare described.Keywords: sign language; pattern classification; database.

EFFECTIVE SIGN LANGUAGE COMMUNICATION AMONG HEARING IMPAIRED MALAYSIAN STUDENTS WITH DIFFERENT COGNITIVE STYLES

This quantitative study investigated the effects of E-SignNet on the learning of Malaysian Sign Language among 147 hearing impaired students with different cognitive styles in Malaysia. A quasi-experimental with control group pre-test and post-test design was used in this research. The independent variables were the presentation modes (SLTI, SLI and SLT) and cognitive styles. The dependent variable was the achievement scores, and the covariate was the pre-test scores. The two-way ANCOVA was performed to analyse the effects of cognitive styles (independent variable) on the achievement scores (dependent variable) among the three presentation modes. The results revealed that by using E-SignNet, there were no significant difference in achievement scores among the three presentation modes and cognitive styles.