Abstract
We studied continuous sign language recognition using Doppler radar sensors. Four signs in Chinese sign language and American sign language were captured and extracted by complex empirical mode decomposition (CEMD) to obtain spectrograms. Image sharpening was used to enhance the micro-Doppler signatures of the signs. To classify the different signs, we utilized an improved Yolov3-tiny network by replacing the framework with ResNet and fine-tuned the network in advance. This method can remove the epentheses from the training process. Experimental results revealed that the proposed method can surpass the state-of-the-art sign language recognition methods in continuous sign recognition with a precision of 0.924, a recall of 0.993, an F1-measure of 0.957 and a mean average precision (mAP) of 0.99. In addition, dialogue recognition in three daily conversation scenarios was performed and evaluated. The average word error rate (WER) was 0.235, 10% lower than in of other works. Our work provides an alternative form of sign language recognition and a new approach to simplify the training process and achieve a better continuous sign language recognition effect.
Highlights
Human–computer interactions (HCIs) have developed prosperously in recent years, allowing various non-contact and flexible methods for different industrial and daily life backgrounds
Hand gestures can be used as inputs in many HCI scenarios and hand gesture recognition (HGR) is a key component widely applied in motion sensory games; intelligent driving; and assisting deaf communities and hearing societies [1,2]
The quantitative effect of the network could be evaluated by precision, recall and mean average precision
Summary
Human–computer interactions (HCIs) have developed prosperously in recent years, allowing various non-contact and flexible methods for different industrial and daily life backgrounds. Hand gestures can be used as inputs in many HCI scenarios and hand gesture recognition (HGR) is a key component widely applied in motion sensory games; intelligent driving; and assisting deaf communities and hearing societies [1,2]. Used in these disabled communities, sign language is considered a dynamic sequence of hand gestures that conveys meaningful semantic expressions from the brain and has received extensive attention in the latest HCI studies. In [7], four dynamic words were distinguished by analyzing the video sequence with a combination of 3D residual convolutional
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
