Unsupervised sign language validation process based on hand-motion parameter clustering

Abstract

Automatic sign language translation process relies mainly on dictionaries of signs to interpret the right meaning of gestures. Due to the lack of large multi sign language dictionaries covering all the aspect of sign languages, the collaborative approach to create signs becomes essential. In fact, the collaborative sign creation process based on Kinect motion capture tool requires the collaboration of non expert users to make sign language dictionaries. However, due to the availability constraint of sign language experts to validate the created signs and the huge amount of signs to be validated manually, the automatic sign language validation process becomes the most suitable solution. In this paper, we present a new automatic and unsupervised sign validation process based on machine learning techniques applied on sign replicas. Given a set of replicas (records) of the same sign created by different non expert sign language user, our main goal is to select the adequate sign records to be used to generate the closest sign signature compared to the one created by sign language expert. For this aim, we present an automatic sign selection and validation solution based on unsupervised clustering of sign motion parameters related to the different sign replicas. We conducted an experimental study to validate 300 ASL signs based on four unsupervised clustering methods, namely, Kernel PCA Kmeans, GMM, Spectral clustering and kernel Kmeans. We concluded that the use our sign validation process using Spectral clustering method allows us to select the right sign replicas to be used to generate the user sign signature. The use of our unsupervised sign validation process onto 3000 ASL sign replicas (300 sign * 10 replicas) lead us to enhance the R2 score average from 0.4830 without sign validation to 0.9123 with sign validation compared to expert sign signature.