An optimized Generative Adversarial Network based continuous sign language classification

Abstract

Classifying manual and non-manual gestures in sign language recognition is a complex and challenging task. Sign language gestures are the combination of hand, face, and body postures, which often have self-occlusions and inter-object occlusions of both the hands, hands with face, or hands with upper body postures. This paper addresses the characterization of manual and non-manual gestures in recognizing the sign language gestures from continuous video sequences. This paper introduces a novel hyperparameter based optimized Generative Adversarial Networks (H-GANs) to classify the sign gestures, and it works in three phases. In phase-I, it adapts the stacked variational auto-encoders (SVAE) and Principal Component Analysis (PCA) to get the pre-tuned data with reduced feature dimensions. In Phase-II, the H-GANs employed Deep Long Short Term Memory (LSTM) as generator and LSTM with 3D Convolutional Neural Network (3D-CNN) as a discriminator. The generator generates random sequences with noise from the real sequence of frames, and the discriminator detects and classifies the real frames of sign gestures. In Phase-III, the proposed approach employs Deep Reinforcement Learning (DRL) for hyperparameter optimization and regularization. By getting the reward points, Proximal Policy Optimization (PPO) optimizes the hyperparameters, and Bayesian Optimization (BO) regularizes the hyperparameters. The proposed H-GANs used two different large vocabulary sign corpus of continuous sign videos to evaluate the performance and efficiency of the system. The experimental results on different dimensions reveal that the H-GANs improved the accuracy and recognition rate when compared with the state-of-the-art classification methods with reduced complexity.