Abstract
Abstract. Inability to use speech interfaces greatly limits the deaf and hearing impaired people in the possibility of human-machine interaction. To solve this problem and to increase the accuracy and reliability of the automatic Russian sign language recognition system it is proposed to use lip-reading in addition to hand gestures recognition. Deaf and hearing impaired people use sign language as the main way of communication in everyday life. Sign language is a structured form of hand gestures and lips movements involving visual motions and signs, which is used as a communication system. Since sign language includes not only hand gestures, but also lip movements that mimic vocalized pronunciation, it is of interest to investigate how accurately such a visual speech can be recognized by a lip-reading system, especially considering the fact that the visual speech of hearing impaired people is often characterized with hyper-articulation, which should potentially facilitate its recognition. For this purpose, thesaurus of Russian sign language (TheRusLan) collected in SPIIRAS in 2018–19 was used. The database consists of color optical FullHD video recordings of 13 native Russian sign language signers (11 females and 2 males) from “Pavlovsk boarding school for the hearing impaired”. Each of the signers demonstrated 164 phrases for 5 times. This work covers the initial stages of this research, including data collection, data labeling, region-of-interest detection and methods for informative features extraction. The results of this study can later be used to create assistive technologies for deaf or hearing impaired people.
Highlights
Automatic lip-reading, known as visual speech recognition (VSR), has received a lot of attention in the recent years for its potential use in applications of human-machine interaction, sign language recognition, audio-visual speech recognition, biometry and biomedicine (Katsaggelos et al, 2015)
The paper discusses the possibility of use additional modality to improve the accuracy and robustness of automatic Russian sign language recognition system
Such studies have not been conducted before and it is of great practical interest to investigate how good visual speech of hearing impaired people can be recognized
Summary
Automatic lip-reading, known as visual speech recognition (VSR), has received a lot of attention in the recent years for its potential use in applications of human-machine interaction, sign language recognition, audio-visual speech recognition, biometry and biomedicine (Katsaggelos et al, 2015). Deaf people use lips movements as a part of a sign language in general, even though they cannot hear acoustic speech. The lips movements of hearing impaired people are usually characterized with hyperarticulation This fact, potentially, enables the opportunity to better recognize the visual speech of such group of people, since lip movements become more pronounced. A significant amount of research has been devoted to the topic of visual speech decoding, the problem of lip reading for hearing impaired people remains an open issue in the field (Akbari et al, 2018). The focus of this research is on improving the accuracy and robustness of automatic Russian sign language recognition via adding lip-reading module to hand gesture recognition system.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
