Recognizing gestures of the ukrainian dactylic alphabet

Abstract

Sign language is a visual way of communicating used by people who are deaf or hard of hearing. It involves handshapes, facial expressions, and body movements to convey meaning. Sign language helps the deaf community interact with each other and the hearing world, allowing them to participate fully in society. According to the WHO (World Health Organization) over 5 % of the world’s population – or 430 million people — experience problems with hearing. More than 44,000 people with hearing impairments are registered with the Ukrainian Society of the Deaf, an all-Ukrainian public organization for the disabled. Therefore, it is extremely important to develop new software, available to the public, that would allow quickly and effectively learn and understand sign language. This work aims to review gesture recognition techniques and develop a system for detecting and classifying gestures of the Ukrainian dactylic alphabet. Two main approaches to gesture recognition, glove-based and computer vision-based (CV), are explained, with the latter being preferred due to its flexibility and widespread usage. The text elaborates on deep learning-based approaches, particularly LSTM networks, and the advantages they offer in automatically learning features from raw image data. The process of creating a dataset for training the gesture classification model is described, which involves recording videos of hand gestures and extracting keypoints using Google MediaPipe. The mo­del training phase is detailed, covering the architecture of the LSTM-based classifier, optimization algorithms, and loss functions. The resulting model achieves an accuracy of 98.4% on the test dataset. A program for real-time gesture recognition is developed using Python and relevant libraries. The program utilizes a webcam feed to detect and classify hand gestures, displaying the top three predicted letters of the Ukrainian dactylic alphabet. The scientific novelty of the obtained results: the paper presents a method that utilizes hand keypoints for recognizing hand gestures of the Ukrainian dactyl alphabet. Also, as part of the development of the gesture recognition system, a data set was collected, where each gesture corresponds to 50 videos of 65 frames. The practical significance of the results obtained: the model obtained as a result of the study can be used to interpret the gestures of the Ukrainian dactylic alphabet. The dataset collected for training this model can be used in other works to train or validate similar models. The paper might be of use to the ones who are interested in developing similar systems for gesture recognition.