Hand gesture is considered as one of the natural ways to interact with computers. The utility of hand gesture-based application is a recent trend and is an effective method for human–computer interaction. Though many static and other intelligent approaches using Machine learning (ML) are developed, still there is a marginal tradeoff between the computational cost and accuracy. In this paper, a Lightboost based Gradient boosting machine (LightGBM) is proposed for efficient hand gesture recognition. The hyper-parameters of the LightGBM are optimized with an improved memetic firefly algorithm. We have introduced a perturbation operator and incorporated it in the proposed memetic firefly algorithm for avoiding the local optimal solution in the traditional firefly algorithm. With comparative analysis among the proposed method and other competitive ML methods, the performance of the proposed method is found to be better in terms of various performance metrics such as accuracy, precision, recall, F1 score, and ROC–AUC. The proposed memetic firefly-based boosting approach is dominant over all the other considered methods with an accuracy of 99.36% and is robust for accurate hand gesture recognition.