Sign Language Recognition Using the Reuse of Estimate Results by Each Epoch

Abstract

Various researchers have proposed systems with high recognition rates for sign language recognition. One of the methods for sign language video recognition is to perform preprocessing, such as optical flow and posture estimation from images, followed by machine learning processing using models such as 3DCNN. Recognition rates tend to be particularly high for models that use information from posture estimation. In addition, many studies have recently used multiple recognition methods and fused their results to increase recognition rates further. In those studies, a Skeleton Aware Multi-modal SLR (SAM-SLR) model used the Turkish sign language dataset AUTSL to achieve very high recognition rates. The next version, SAM-SLR-v2, proposed a skeleton-aware multi-model framework that supports three sign language datasets and uses a global ensemble model for isolated signs. In particular, one of the three datasets, Turkish Sign Language, contains both RGB and Depth videos, and SAM-SLR-v2 achieved high recognition rates of 98.00% and 98.10%, respectively. We wanted to pursue the possibility of achieving even higher recognition rates. Therefore, in this study, based on this SAM-SLR, we first propose a method to increase the recognition rate by reusing the previous training models and estimate results created at each epoch when creating training models for each recognition method using the posture estimation Joint and Bone features. Next, we report the recognition rate of the Multi-stream, Modal-free Late-fusion Ensemble using the proposed method with Joint and Bone. We demonstrate that our proposed method is easy to implement and achieves improved performance because it reuses the estimate results already obtained. Our method achieved a recognition rate of 96.10% and 96.18% for Joint and Bone, respectively, and the results were reflected in the Modal-free Latefusion Ensemble, which achieved a recognition rate of 98.05% when the optimal parameters were given manually in RGB videos.