A practically deployable gesture recognition system is developed using a robust hand detection method implemented using a motion-based image segmentation process and a two-level bare hand classification model, which is integrated with a gesture classification system of 58 gestures using new robust features. Since detection of bare hand is affected by nonideal conditions, multiple color-texture features are analyzed in this study. In the second stage of the system, 18 new ASCII characters are introduced and analyzed along with the existing 40 characters (alphabets, numbers, and arithmetic operators). New 15 dimensional features are introduced along with the existing features to enhance the classification accuracy of the gestures. Significance of features statistically tested using one-way analysis of variance (ANOVA), Kruskal–Wallis and Friedman test, which are sequentially ranked and evaluated using incremental feature selection (IFS) method. Performance of the proposed hand detection system is observed to be 12.5% higher than the existing hand detection system under clean conditions, while 46.4% higher under the nonideal conditions. Performance of 58 gestures classification model has improved by 12.08% (Naïve Bayes), 8.86% (ELM), 10.83% (SVM), 8.02% ([Formula: see text]NN), and 6.61% (ANN) after using the new features. Majority voting-based classifier fusion method further improves the performance of the gesture recognition system by 3.88%, which is validated by Turkey’s HSD test.