LED Screen-Based Intelligent Hand Gesture Recognition System

Abstract

Common human–computer interaction (HCI) modes rely on various sensors or cameras for operation. This study proposes a gesture recognition method and device based on light sensing characteristics, which removes the need for external sensors with only light-emitting diode (LED) screens. The key technique is that the photoelectric sensing ability of LED is applied to detect the information change of light without interference to the display function, thus indicating the variation of gestures. This system is integrated by using six LED screen modules with the field-programmable gate array (FPGA) to control display and collect data for deep-learning analysis. Depending on both time- and frequency-domain characteristics of gestures, the recognition algorithm is implemented based on static bidirectional long short-term memory (S-Bi-LSTM). As revealed by the experiments conducted on hand gestures of different people and under the context of complex background illumination, the accuracy of gesture recognition reaches as high as 93.60%. As for dynamic gesture actions, an optimized dynamic bidirectional long short-term memory (D-Bi-LSTM) algorithm is also proposed with the method of frame segmentation. This method divides dynamic gestures into three parts to feed different classifiers, so as to enhance the accuracy of dynamic gestures recognition up to 91.67%. The device possesses all the structures of a large LED screen, which verifies the feasibility of pattern recognition for large LED screens from an experimental perspective. Without any need for traditional sensors, it is contactless, secure, and easy to promote.