Sign Recognition Research Articles

Hardware accelerator for high accuracy sign language recognition with residual network based on FPGAs

Hardware accelerator for high accuracy sign language recognition with residual network based on FPGAs

Study of Artificial Intelligence Aspects in Policy Formulation and Decision-Making Paradigms and Theories of Administrative Science


 
 
 The development and use of Artificial Intelligence (AI) has become a widely discussed topic, not only in Indonesia but also globally. The emergence of ChatGPT in 2022 has led to the rapid development and use of AI in various fields, such as medicine, creative industries, academics, and social entrepreneurship. The history of AI development began with the creation of the Enigma code-breaking machine during World War II by Alan Turing. Since then, AI research has continued to evolve, with the Turing Test serving as the basis for identifying the intelligence of an artificial system. The COVID-19 pandemic has also accelerated the development of AI, particularly in the use of big data for diagnosis. The introduction of ChatGPT, a generative AI program, has further increased public interest in AI. The potential of generative AI to simplify workflows in various industries and individuals is vast, and it has been used in social entrepreneurship, such as the Sign-Speak application for real-time sign language recognition and translation. The government can also use AI to formulate policies and address issues in society. However, the use of AI must be ethical and not reduce the essence of research.
 
 

Real-Time Sign Language Recognition and Translation Using Deep Learning Techniques

Sign Language Recognition (SLR) recognizes hand gestures and produces the corresponding text or speech. Despite advances in deep learning, the SLR still faces challenges in terms of accuracy and visual quality. Sign Language Translation (SLT) aims to translate sign language images or videos into spoken language, which is hampered by limited language comprehension datasets. This paper presents an innovative approach for sign language recognition and conversion to text using a custom dataset containing 15 different classes, each class containing 70-75 different images. The proposed solution uses the YOLOv5 architecture, a state-of-the-art Convolutional Neural Network (CNN) to achieve robust and accurate sign language recognition. With careful training and optimization, the model achieves impressive mAP values (average accuracy) of 92% to 99% for each of the 15 classes. An extensive dataset combined with the YOLOv5 model provides effective real-time sign language interpretation, showing the potential to improve accessibility and communication for the hearing impaired. This application lays the groundwork for further advances in sign language recognition systems with implications for inclusive technology applications.

Combinational sign language recognition

Traditional Sign Language Recognition (SLR) suffers from the scale limitation of SL datasets, which may lead to over-fitting in narrow context and application. In this paper, to solve the problem, we for the first time propose a Combinational Sign Language Recognition (CombSLR) framework, which can serve as an augmentation to extend existing datasets by combining continuous videos (called Template) and isolated videos (called Entity). The CombSLR framework is trained on combinational SL data (T & E) and applied on continuous SL data. However, due to the unknown combination location and context inconsistency between any T-E pair, naively inserting E into T is infeasible. To tackle this issue, we propose a simple yet effective method named EinT, which contains two main modules: (1) Location Candidate Prediction, to produce a reliable insertion location considering the inter-frame relationship and make the network end-to-end trainable; (2) Feature Insertion via Context Passing, to eliminate context inconsistency between T and E feature. EinT can be easily compatible with the existing SLR models to effectively implement data augmentation at the feature level during training stage. We conduct extensive experiments on multiple publicly available sign language datasets, e.g., CCLS, CSL+DEVISIGN-D and CSL-Daily+DEVISIGN-D. The experimental results show the CombSLR can significantly promote existing SLR methods, e.g., averagely improving by 15.1% on CCLS dataset and 6.4% on CSL dataset for WER metric, which demonstrates the superiority of CombSLR framework.

Frequency and Spatial Domain-Based Approaches for Recognition of Indian Sign Language Gestures

Objectives: The objective of this paper is to introduce and demonstrate an innovative approach for the recognition of Indian sign language gestures, with a focus on bridging communication gap between the deaf and hearing communities. The goal is to contribute to the development of effective tools and technologies that facilitate seamless communication between individuals using sign language and the people with no knowledge about sign language. Methods: The methodology consists of three key steps. First, data pre-processing involves resizing and contours extraction. Next, feature extraction employs Fourier descriptors for frequency domain analysis and gray-level-co-occurrence matrix for spatial domain analysis. Finally, various machine learning models including SVM, Random Forest, Logistic Regression, K-Nearest Neighbor and Naive Bayes are trained on a standard dataset. Findings: In our controlled experimental setup, we applied a diverse set of machine learning classifiers to evaluate the proposed approach for gesture recognition. Among the classifiers tested, K-Nearest Neighbors demonstrated the highest accuracy, achieving 99.82%. To validate the robustness of our approach, we employed k-fold cross-validation with 5 folds. Novelty: This study presents an innovative method for sign language recognition by employing a dual-domain fusion strategy that prominently emphasizes the frequency domain. Through the integration of Fourier descriptors, the research conducts a detailed frequency domain analysis to characterize the contour shapes of sign language gestures. The synergy with gray-level co-occurrence matrix texture features in the spatial domain analysis, contributes to the creation of a comprehensive feature vector. The proposed approach ensures a thorough exploration of gesture features, there by advancing the precision and efficacy of sign language recognition. Keywords: Indian Sign Language (ISL), Sign Language Recognition (SLR), Frequency domain, Spatial domain, Fourier descriptors, Gray level co­occurrence matrix (GLCM), K­ Fold

Real-time sign language recognition based on YOLO algorithm

This study focuses on real-time hand gesture recognition in the Turkish sign language detection system. YOLOv4-CSP based on convolutional neural network (CNN), a state-of-the-art object detection algorithm, is used to provide real-time and high-performance detection. The YOLOv4-CSP algorithm is created by adding CSPNet to the neck of the original YOLOv4 to improve network performance. A new object detection model has been proposed by optimizing the YOLOv4-CSP algorithm in order to provide more efficient detection in Turkish sign language. The model uses CSPNet throughout the network to increase the learning ability of the network. However, Proposed YOLOv4-CSP has a learning model with Mish activation function, complete intersection of union (CIoU) loss function and transformer block added. The Proposed YOLOv4-CSP algorithm has faster learning with transfer learning than previous versions. This allows the proposed YOLOv4-CSP algorithm to perform a faster restriction and recognition of static hand signals simultaneously. To evaluate the speed and detection performance of the proposed YOLOv4-CSP model, it is compared with previous YOLO series, which offers real-time detection, as well. YOLOv3, YOLOv3-SPP, YOLOv4-CSP and proposed YOLOv4-CSP models are trained with a labeled dataset consisting of numbers in Turkish Sign language, and their performances on the hand signals recognitions are compared. With the proposed method, 98.95% precision, 98.15% recall, 98.55 F1 score and 99.49% mAP results are obtained in 9.8 ms. The proposed method for detecting numbers in Turkish sign language outperforms other algorithms with both real-time performance and accurate hand sign prediction, regardless of background.

Asymmetric multi-branch GCN for skeleton-based sign language recognition

Asymmetric multi-branch GCN for skeleton-based sign language recognition

Traffic Sign Recognition Using Optimized Federated Learning in Internet of Vehicles

Traffic Sign Recognition Using Optimized Federated Learning in Internet of Vehicles

Video frame feeding approach for validating the performance of an object detection model in real-world conditions

The challenge of evaluating deep learning-based object detection models in complex traffic scenarios, characterized by changing weather and lighting conditions, is addressed in this study. Real-world testing proves time and cost-intensive, leading to the proposal of a Video Frame Feeding (VFF) approach as a solution. The proposed Video Frame Feeding approach acts as a bridge between object detection models and simulated environments, enabling the generation of realistic scenarios. Leveraging the CarMaker (CM) tool to simulate realistic scenarios, the framework utilizes a virtual camera to capture the simulated environment and feed video frames to an object identification model. The VFF algorithm, with automated validation using simulated ground truth data, enhances detection accuracy to over 95% at 30 frames per second within 130 meters. Employing the You Only Look Once (YOLO) version 4 and the German Traffic Sign Recognition Benchmark dataset, the study assesses a traffic signboard identification model across various climatic conditions. Notably, the VFF algorithm improves accuracy by 2% to 5% in challenging scenarios like foggy days and nights. This innovative approach not only identifies object detection issues efficiently but also offers a versatile solution applicable to any object detection model, promising improved dataset quality and robustness for enhanced model performance.

Traffic sign recognition using CNN and Res-Net

In the realm of contemporary applications and everyday life, the significance of object recognition and classification cannot be overstated. A multitude of valuable domains, including G-lens technology, cancer prediction, Optical Character Recognition (OCR), Face Recognition, and more, heavily rely on the efficacy of image identification algorithms. Among these, Convolutional Neural Networks (CNN) have emerged as a cutting-edge technique that excels in its aptitude for feature extraction, offering pragmatic solutions to a diverse array of object recognition challenges. CNN's notable strength is underscored by its swifter execution, rendering it particularly advantageous for real-time processing. The domain of traffic sign recognition holds profound importance, especially in the development of practical applications like autonomous driving for vehicles such as Tesla, as well as in the realm of traffic surveillance. In this research endeavour, the focus was directed towards the Belgium Traffic Signs Dataset (BTS), an encompassing repository comprising a total of 62 distinct traffic signs. By employing a CNN model, a meticulously methodical approach was obtained commencing with a rigorous phase of data pre-processing. This preparatory stage was complemented by the strategic incorporation of residual blocks during model training, thereby enhancing the network's ability to glean intricate features from traffic sign images. Notably, our proposed methodology yielded a commendable accuracy rate of 94.25%, demonstrating the system's robust and proficient recognition capabilities. The distinctive prowess of our methodology shines through its substantial improvements in specific parameters compared to pre-existing techniques. Our approach thrives in terms of accuracy, capitalizing on CNN's rapid execution speed, and offering an efficient means of feature extraction. By effectively training on a diverse dataset encompassing 62 varied traffic signs, our model showcases a promising potential for real-world applications. The overarching analysis highlights the efficacy of our proposed technique, reaffirming its potency in achieving precise traffic sign recognition and positioning it as a viable solution for real-time scenarios and autonomous systems.

AI on the Road: NVIDIA Jetson Nano-Powered Computer Vision-Based System for Real-Time Pedestrian and Priority Sign Detection

Advances in information and signal processing, driven by artificial intelligence techniques and recent breakthroughs in deep learning, have significantly impacted autonomous driving by enhancing safety and reducing the dependence on human intervention. Generally, prevailing ADASs (advanced driver assistance systems) incorporate costly components, making them financially unattainable for a substantial portion of the population. This paper proposes a solution: an embedded system designed for real-time pedestrian and priority sign detection, offering affordability and universal applicability across various vehicles. The suggested system, which comprises two cameras, an NVIDIA Jetson Nano B01 low-power edge device and an LCD (liquid crystal system) display, ensures seamless integration into a vehicle without occupying substantial space and provides a cost-effective alternative. The primary focus of this research is addressing accidents caused by the failure to yield priority to other drivers or pedestrians. Our study stands out from existing research by concurrently addressing traffic sign recognition and pedestrian detection, concentrating on identifying five crucial objects: pedestrians, pedestrian crossings (signs and road paintings separately), stop signs, and give way signs. Object detection was executed using a lightweight, custom-trained CNN (convolutional neural network) known as SSD (Single Shot Detector)-MobileNet, implemented on the Jetson Nano. To tailor the model for this specific application, the pre-trained neural network underwent training on our custom dataset consisting of images captured on the road under diverse lighting and traffic conditions. The outcomes of the proposed system offer promising results, positioning it as a viable candidate for real-time implementation; its contributions are noteworthy in advancing the safety and accessibility of autonomous driving technologies.

Research and Development of a Traffic Sign Recognition Module in Vietnam

Automatic traffic sign recognition is essential in researching and developing driver assistance systems and autonomous vehicles. This paper presents the research and development of an automated traffic sign recognition module in Vietnam. The recognition model is developed based on the deep learning model YOLOv5 and incorporates architectural modifications to reduce computational complexity, increase inference speed, and meet real-time requirements for embedded system applications. The model is trained using a custom dataset collected by the research team from real-world street environments in Vietnam, encompassing diverse locations, times, and weather conditions. The trained recognition model is deployed on the Jetson embedded system, yielding high-quality recognition results and meeting real-time recognition needs.

Hand gesture recognition in natural human-computer interaction

This paper introduces the definition of Gesture recognition. First the article gives a precise definition of gesture recognition and explains the difference between gestures and postures, then it reveals technical difficulties of Gesture recognition. These technical difficulties include four aspects. After analyzing the technology and methods of Gesture recognition and the technical difficulties, it concretely expounds gesture recognition process based on data gloves, which is widely studied before and it also introduces the computer vision based research achievement which is currently becoming a research hotspot in this field. Then it also takes Kinect and HoloLens as examples to introduce specific practical cases of gesture recognition in wearable devices. Also it outlines the application of gesture recognition technology in human-computer interaction which include but not limited to smart terminal, Game control, Robot Control, Clinical and Health, Smart home, Sign Language Recognition, Vehicle system, Interactive entertainment. Finally it reach the conclusion that the biggest challenge researchers meet is to build a powerful framework to overcome the common problems with fewer constraints to provide reliable results. And sometimes researchers need to combine multiple methods for different complex environments.

Accelerated Low Power AI System for Indian Sign Language Recognition

Accelerated Low Power AI System for Indian Sign Language Recognition

Dynamic Indian Sign Language Recognition Based on Enhanced LSTM with Custom Attention Mechanism

Dynamic Indian Sign Language Recognition Based on Enhanced LSTM with Custom Attention Mechanism

Amplifying awareness of cardiac amyloidosis and the need for early recognition

Improvements in clinical awareness of cardiac amyloidosis have led to earlier recognition of signs and symptoms, access to diagnostics and a specialist multidisciplinary team. Yet, it remains a life-limiting disease. This editorial discusses cardiac amyloidosis and how nurses can identify ‘red flags’ to facilitate earlier diagnosis.

Real-time recognition of Indonesian sign language using recurrent neural network

Hand gestures serve as a vital means of communication for deaf individuals. They often face communication challenges in their daily interactions due to the language barrier. This underscores the necessity of sign language interpreters. However, prevailing methods primarily rely on the Indonesian Sign Language System (SIBI), despite the widespread use of Indonesian Sign Language (BISINDO) for communication. Additionally, the effectiveness of these methods hinges greatly on the accuracy of feature extraction. To address this limitation, this study introduces a Recurrent Neural Network (RNN) approach for BISINDO interpretation. Data acquisition involved the use of a webcam to capture video data, subsequently transformed into frames andarrays. Collected from three respondents, the dataset comprises 3,240 videos and 97,200 array data points, encompassing letters and numbers. Among the tested parameters, training results indicate that utilizing the Adam optimizer with a learning rate of 0.0001 and 500 epochs yields the highest accuracy and minimal loss compared to other configurations. Subsequently, this modelunderwent real-time testing, conducted five times for 36 classes, achieving an accuracy of 81.67%. It is important to note that errors may arise due to similarities within hand signal language, particularly involving characters such as I, J, D, P, M, and N.

Abstract WP45: A Collaborative Approach to Improving Inpatient Stroke Code Time Target Metrics

Much effort and education has been placed on pre-hospital and emergency departments to recognize signs and symptoms of acute stroke and time metrics such as door-to-CT, door-to-needle, and door-to-device. For admitted patients with acute onset of stroke symptoms, our data analysis identified a gap in the timeliness of stroke code activation upon symptoms discovery and subsequent delays in CT initiation and treatment. We evaluated our current process and implemented action plans to enhance the current inpatient stroke code process through focused education and training of staff to improve their recognition of acute stroke, timely activation of stroke code and improved CT initiation and treatment times. In 2021, several cardiology pilot units were selected and neuroscience nurse specialists conducted staff education which focused on recognition of stroke signs and symptoms, code activation criteria and response time targets. A resource binder and checklist were created to guide staff in putting together pertinent elements for communication and documentation. Data was collected on stroke code response times and individual feedback was provided. The group meets quarterly to evaluate progress, discuss opportunities for improvement and highlight best practices and outcomes. The checklist was a useful tool in reminding staff of pertinent information to communicate to the stroke team although form completion is not consistent. In 2021-2022, 54% of stroke codes were called within 10 minutes of symptoms discovery. CTH initiation time within 25 minutes of symptoms discovery was achieved in 16.2% of patients. Data analysis showed an opportunity to improve documentation of symptoms discovery time which were missing in 13.8% of cases. This collaborative approach with the pilot units has increased awareness and focus to the inpatient stroke code process. Opportunities exist to improve time target achievements in CTH initiation, treatment, and documentation of symptoms discovery time. Communication between the stroke team and the patient’s primary team has been identified as an essential factor for continued improvement along with on-going staff education.

Abstract NS9: Nurse-Led Quality Improvement Initiative: Advancing Stroke Awareness in Communities Experiencing Racial and Ethnic Disparities

Background and Purpose: As a Joint Commission Comprehensive Stroke Center, the provision of community education is a priority goal of our program. The majority of our existing community education program takes place in locations where we receive requests to provide these sessions. Analysis of our Get With the Guidelines (GWTG) data reveals gaps in early arrival to the hospital and Emergency Medical Service (EMS) utilization among African American and Hispanic populations. Providing educational resources within these populations may help to close these gaps and foster trust in the healthcare system. The aim of this proposed nurse-led quality improvement project is to improve hospital arrival times of African American and Hispanic stroke patients living in zip codes where GWTG data have identified gaps, through provision of targeted community education. Methods: The top zip codes associated with our target outcomes for improvement will be identified using GWTG pre-data from 2022 and 2023. We will partner with our Community Outreach department. We will identify educational venues and stakeholders among African American and Hispanic populations in these zip codes. We will provide education tailored to our identified populations including early recognition of stroke signs and symptoms, activation of EMS, stroke risk factors, and opportunities for treatment. Post implementation, GWTG data will be reviewed for any changes. Results: Using GWTG data, arrival mode, last known well time, and symptom discovery time to Emergency Department arrival as it pertains to zip codes, race and ethnicity will be reviewed and analyzed. The target zip codes’ data will be compared pre- versus post-implementation of community education efforts. Discussion: Through community education and partnership with the Community Outreach department we hope to improve timeliness of arrival to hospital among disproportionately affected racial minorities in our community for early stroke evaluation and treatment consideration.

An Adam based CNN and LSTM approach for sign language recognition in real time for deaf people

Hand gestures and sign language are crucial modes of communication for deaf individuals. Since most people can't understand sign language, it's hard for a mute and an average person to talk to each other. Because of technological progress, computer vision and deep learning can now be used to count. This paper shows two ways to use deep knowledge to recognize sign language. These methods help regular people understand sign language and improve their communication. Based on American sign language (ASL), two separate datasets have been constructed; the first has 26 signs, and the other contains three significant symbols with the crucial sequence of frames or videos for regular communication. This study looks at three different models: the improved ResNet-based convolutional neural network (CNN), the long short-term memory (LSTM), and the gated recurrent unit (GRU). The first dataset is used to fit and assess the CNN model. With the adaptive moment estimation (Adam) optimizer, CNN obtains an accuracy of 89.07%. In contrast, the second dataset is given to LSTM and GRU and a comparison has been conducted. LSTM does better than GRU in all classes. LSTM has a 94.3% accuracy, while GRU only manages 79.3%. Our preliminary models' real-time performance is also highlighted.