Gesture Recognition Device Research Articles

Assisting autonomous precision medical rehabilitation through gesture recognition devices

With the aging of the population and the increase of various chronic diseases, there is a growing demand for effective and accessible medical rehab solutions. In order to overcome the above problems, this project innovatively adopts the gesture recognition technology to design and fabricate the realization of an autonomous accurate medical rehab gesture recognition assistance system. In terms of hardware, this paper uses Arduino as the main controller, Paj7620 as the gesture recognition device, and OLED 0.96 display as the system display. In terms of software, this paper utilizes Arduino IDE software to successfully write the code and realize all the functions. In terms of combining with actual medical rehab needs, the team chose elbow fracture rehab as a typical type of rehab, and designed a three-device obtuse triangle placement scheme for precise monitoring of rehab training. Initial simulation test results show that over 70% of participants reported an improvement in their rehabilitation outcomes, and about 90% found the system easy to use. The results of this project will directly benefit patients requiring long-term rehabilitation.

Flexible Strain Sensor-Based Data Glove for Gesture Interaction in the Metaverse: A Review

The rise of the metaverse concept has brought about widespread attention in wearable gesture recognition devices. Data gloves based on flexible strain sensors have been favoured by researchers owing to their low cost, light weight, direct and continuous monitoring of finger movements. In this review, we first compare the advantages and disadvantages of four different approaches based on the vision sensors, myoelectric sensors, inertial and magnetic sensors, and flexible strain sensors in designing data gloves, and demonstrate the superiority of the flexible strain sensor-based data glove used for metaverse applications. Next, some latest commercial data gloves are exampled and the function modules of the data gloves are presented based on the flexible strain sensors. Meanwhile, the potential applications of gesture recognition in the metaverse are summarized in diversified fields. Finally, the existing problems and development prospects of the current data gloves based on flexible strain sensors are concluded. We are optimistic that novel flexible strain sensor-based data gloves will make transformational impact to realize accurate, low-latency, and immersive gesture interaction in the metaverse.

Research on Gesture Recognition System Using Multiple Sensors Based on Earth’s Magnetic Field and 1D Convolution Neural Network

A commonly used method of gesture recognition is the use of sensor technology. Typically, technology detecting Earth’s magnetic field is used in indoor positioning, and magnetic detection technology serves as a redundant method for gesture recognition devices. In this paper, we propose a novel system that utilizes multiple sensors measuring Earth’s magnetic field to collect data and perform gesture recognition through a one-dimensional convolutional neural network algorithm. By applying the detection of Earth’s magnetic field to gesture recognition, our system significantly improves the accuracy of recognition through a one-dimensional (1D) neural network algorithm. We conducted experiments where we collected and recognized American Sign Language standard letters, and achieved an accuracy rate close to 97%. Our experimental results demonstrate that this gesture recognition system using magnetic field sensors and a one-dimensional neural network algorithm is feasible for practical applications. Furthermore, our approach reduces the complexity of the device compared to the gesture recognition method based on artificial magnetic fields, while maintaining high recognition accuracy and not limiting the user’s hand movements. This technology holds great promise for the field of human–computer interaction.

An Investigation of Translation of Text Language to Sign Language Using Machine Learning

Among the fastest-growing areas of study today is the translation of sign language, which is the most natural form of communication for those with hearing loss. Deaf individuals may be able to communicate with hearing people directly, without an intermediary, with the use of a hand gesture recognition device. The method was developed to facilitate the automatic translation of American Sign Language into text and sound. The suggested system uses a large data collection to interpret individual words and phrases in traditional American Sign Language, alleviating any fears that the user may have about using a virtual camera. Deaf and mute persons must rely on Sign Language as their only method of communication. However, a large portion of the general public is illiterate in sign language. Therefore, those who sign have a more difficult time communicating with those who don't without the help of an interpreter. The proposed technique employs data collected by Fifth Dimension Technologies (5DT) gloves to try to interpret hand movements as spoken language. The data has been classified into text words using a variety of machine learning techniques, including neural networks, decision tree classifiers, and k-nearest neighbors.

Gesture recognition device based on cross reticulated graphene strain sensors

Perception and recognition of hand gestures have received much attention for their applications in the human–computer fields. The data-glove application in gesture recognition makes it expediently to track the movements of human fingers. Whereas, the development of the data-glove for hand gesture still face numerous technical difficulties, such as the sensitivity, flexible ability, and system integration under low-cost. In this paper, a simple gesture recognition glove system was developed to characterize the sign language, which consists of cross reticulated graphene (CRG) flexible sensors, a multi-channel data acquisition module, and an artificial neural network algorithm. The designed acquisition module has several channels, so that signs from five sensors can be collected and transferred to the intelligent terminal to execute the hand gesture recognition algorithm. Besides, the recognition results suggest that the recognition rate of the neural network for 22 letters in the English alphabet can reach more than 90% and the overall recognition has an over 86% accuracy.

Research on a novel gesture recognition device

In order to effectively detect the input gesture information, a new type of capacitive sensing technology, which is of a low power consumption, low cost but high resolution non-contact sensing technology, is adopted. When our finger gets close to the metal plate, it will cause the change of the capacitance value of the metal plate to the ground. That value can be converted into 28-bit binary data for output. The microprocessor is equipped with the IIC interface for acquisition and processing of the output data. Because a variety of gestures produce different capacitance values, their output varies. After many times of training and machine learning, different gestures can be identified. The test results show the rate of different gestures recognition is over 99%, which can be applicable in human-computer interaction equipment.

Hand Gesture Recognition Using Multiple Acoustic Measurements at Wrist

This article investigates the use of acoustic signals recorded at the human wrist for hand gesture recognition. The prototype consists of 40 microphones to be worn at the wrist. The gesture recognition performance is evaluated through the identification of 36 gestures in American sign language (ASL), including 26 ASL alphabetical characters and 10 ASL numbers. The optimal area for sensor band placement (distal/proximal) is examined to reveal the location of the highest discrimination accuracy. Ten subjects are recruited to perform over ten trials for each set of hand gestures. Using mutual information-based feature selection methods, two time-domain features: difference absolute mean value and log-energy entropy, is selected from 35 commonly used time-domain features for the hand gesture classification. As a proof-of-concept, two common classification methods, linear discriminant analysis and support vector machine, is used to classify hand gestures from the acoustic measurements recorded by each subject. Results show the intrasubject average classification accuracy above 90% using the two features with all 40 microphones, while the average classification accuracy exceeding 84% is obtained using ten microphones. These results indicate that acoustic signatures from the human wrist can be utilized for hand gesture recognition, while the use of few, simple features, with low computational requirements is sufficient to characterize some hand gestures. The proposed technique demonstrates a promising means for developing a low-cost wearable hand gesture recognition device using microphones.

Gesture Classification for a Hand Controller Device Using Neural Networks.

This paper presents a convolutional neural network-based classification of the hand flexion and extension gestures used in wrist recovery after injury. The hand gesture recognition device used in our study is the Leap Motion controller. The Leap Motion device's inability to accurately differentiate the left hand from the right hand when performing hand rotation gestures was eliminated by introducing hand and thumb direction vectors into the database used to train the neural network. A 3D environment was created for the introduction of the data describing the gesture into the database. A classification accuracy of 95% was achieved for the hand flexion and extension gesture divided into three levels for each hand. The populated database may also be used to classify other gestures involving hand rotation.

Intuitive gesture-based control system with collision avoidance for robotic manipulators

PurposeThis paper aims to present an intuitive control system for robotic manipulators that pairs a Leap Motion, a low-cost optical tracking and gesture recognition device, with the ability to record and replay trajectories and operation to create an intuitive method of controlling and programming a robotic manipulator. This system was designed to be extensible and includes modules and methods for obstacle detection and dynamic trajectory modification for obstacle avoidance.Design/methodology/approachThe presented control architecture, while portable to any robotic platform, was designed to actuate a six degree-of-freedom robotic manipulator of our own design. From the data collected by the Leap Motion, the manipulator was controlled by mapping the position and orientation of the human hand to values in the joint space of the robot. Additional recording and playback functionality was implemented to allow for the robot to repeat the desired tasks once the task had been demonstrated and recorded.FindingsExperiments were conducted on our custom-built robotic manipulator by first using a simulation model to characterize and quantify the robot’s tracking of the Leap Motion generated trajectory. Tests were conducted in the Gazebo simulation software in conjunction with Robot Operating System, where results were collected by recording both the real-time input from the Leap Motion sensor, and the corresponding pose data. The results of these experiments show that the goal of accurate and real-time control of the robot was achieved and validated our methods of transcribing, recording and repeating six degree-of-freedom trajectories from the Leap Motion camera.Originality/valueAs robots evolve in complexity, the methods of programming them need to evolve to become more intuitive. Humans instinctively teach by demonstrating the task to a given subject, who then observes the various poses and tries to replicate the motions. This work aims to integrate the natural human teaching methods into robotics programming through an intuitive, demonstration-based programming method.

Floating autostereoscopic display with in situ interaction

AbstractFloating three‐dimensional (3D) display implements direct interaction between human hands and virtual 3D images, which offers natural and effective augmented reality interaction. In this study, we propose a novel floating autostereoscopic display, combining head tracking lenticular display with an image projection system, to offer the observers with an accurate 3D image floating in midair without any optical elements between observers and the virtual 3D image. Combined with a gesture recognition device, the proposed system can achieve in situ augmented reality interaction with the floating 3D image. A distortion correction method is developed to achieve 3D display with accurate spatial information. Moreover, a coordinate calibration method is designed to improve the accuracy in the in situ interaction. Experiments were performed to prove the feasibility of the proposed system, and the good results show the potential of human‐computer interaction in medicine and life sciences.

Implementation of Hand Gesture Recognition Device Applicable to Smart Watch Based on Flexible Epidermal Tactile Sensor Array.

Ever since the development of digital devices, the recognition of human gestures has played an important role in many Human-Computer interface applications. Various wearable devices have been developed, and inertial sensors, magnetic sensors, gyro sensors, electromyography, force-sensitive resistors, and other types of sensors have been used to identify gestures. However, there are different drawbacks for each sensor, which affect the detection of gestures. In this paper, we present a new gesture recognition method using a Flexible Epidermal Tactile Sensor based on strain gauges to sense deformation. Such deformations are transduced to electric signals. By measuring the electric signals, the sensor can estimate the degree of deformation, including compression, tension, and twist, caused by movements of the wrist. The proposed sensor array was demonstrated to be capable of analyzing the eight motions of the wrist, and showed robustness, stability, and repeatability throughout a range of experiments aimed at testing the sensor array. We compared the performance of the prototype device with those of previous studies, under the same experimental conditions. The result shows our recognition method significantly outperformed existing methods.

Mirror Therapy Using Gesture Recognition for Upper Limb Function, Neck Discomfort, and Quality of Life After Chronic Stroke: A Single-Blind Randomized Controlled Trial.

BackgroundMirror therapy for stroke patients was reported to be effective in improving upper-extremity motor function and daily life activity performance. In addition, game-based virtual reality can be realized using a gesture recognition (GR) device, and various tasks can be presented. Therefore, this study investigated changes in upper-extremity motor function, quality of life, and neck discomfort when using a GR device for mirror therapy to observe the upper extremities reflected in the mirror.Material/MethodsA total of 36 subjects with chronic stroke were randomly divided into 3 groups: GR mirror therapy (n=12), conventional mirror therapy (n=12), and control (n=12) groups. The GR therapy group performed 3D motion input device-based mirror therapy, the conventional mirror therapy group underwent general mirror therapy, and the control group underwent sham therapy. Each group underwent 15 (30 min/d) intervention sessions (3 d/wk for 5 weeks). All subjects were assessed by manual function test, neck discomfort score, and Short-Form 8 in pre- and post-test.ResultsUpper-extremity function, depression, and quality of life in the GR mirror therapy group were significantly better than in the control group. The changes of neck discomfort in the conventional mirror therapy and control groups were significantly greater than in the GR mirror therapy group.ConclusionsWe found that GR device-based mirror therapy is an intervention that improves upper-extremity function, neck discomfort, and quality of life in patients with chronic stroke.

A wearable hand gesture recognition device based on acoustic measurements at wrist.

This paper investigates hand gesture recognition from acoustic measurements at wrist for the development of a low-cost wearable human-computer interaction (HCI) device. A prototype with 5 microphone sensors on human wrist is benchmarked in hand gesture recognition performance by identifying 36 gestures in American Sign Language (ASL). Three subjects were recruited to perform over 20 trials for each set of hand gestures, including 26 ASL alphabets and 10 ASL numbers. Ten features were extracted from the signal recorded by each sensor. Support Vector Machine (SVM), Decision Tree (DT), K-Nearest Neighbors (kNN), and Linear Discriminant Analysis (LDA) were compared in classification performance. Among which, LDA offered the highest average classification accuracy above 80%. Based on these preliminary results, our proposed technique has exhibited a promising means for developing a low-cost HCI.

A Wearable Gesture Recognition Device for Detecting Muscular Activities Based on Air-Pressure Sensors

Recognition of human gestures plays an important role in a number of human-interactive applications, such as mobile phones, health monitoring systems, and human-assistive robots. Electromyography (EMG) is one of the most common and intuitive methods used for detecting gestures based on muscle activities. The EMG, however, is in general, too sensitive to environmental disturbances, such as electrical noise, electromagnetic signals, humidity, and so on. In this paper, a new method for recognizing the muscular activities is proposed based on air-pressure sensors and air-bladders. The muscular activity is detected by measuring the change of the air pressure in an air-bladder contacting the interested muscle(s). Since the change of the air pressure can be more robustly measured compared with the change of electric signals appeared on the skin, the proposed sensing method is useful for mobile devices due to its great signal-to-noise ratio (SNR) and fast response time. The principle and applications of the proposed sensing method are introduced in this paper. The performance of the proposed method is evaluated in terms of linearity, repeatability, wear-comfort, etc., and is also verified by comparing it with an EMG signal and a motion sensor.

Advancing the Playing Field. A look at how virtual reality can change the future of anatomy teaching learning process

Play combines all three aspects of learning: visual, auditory, and kinesthetic. Visual learners have a preference for seeing. Auditory learners’ best learn through listening. kinesthetic learners prefer to learn via experience—moving, touching, and doing. Through play, children learn a set of skills such as social skills, creativity, handto‐eye coordination, problem solving, and imagination. It is argued that these skills are better learned through play rather than through flashcards or academic drills. An element of “play” is an essential condition for perennial learning to occur in all individuals. So why should we leave this tool behind while learning medicine? Commonly, lectures are designed to favor one or maybe two types of learning styles, usually leaving gaps within medical education. Play is more than simply having a good time; it is about working out rules, understanding constraints, being absorbed by novelty, using imagination and innovating. It is also about extending, modifying and augmenting learning activities. This project brings an overt element of play to the representation of fundamental anatomical concepts and thus engages students more strongly in the learning process. The “Microsoft Kinect™ gesture recognition device” enables users to simultaneously stimulate all three learning styles at once leading to the most profound learning experience. With the Kinect you experience an immersive 3D environment, with auditory commands coupled with kinesthetic control. Theoretically providing a learning machine that can be utilized in labs, tutorials, lectures, and even at home. This project brings the element of play to the representation of fundamental anatomical concepts, and thus engages students more strongly in the learning process.

An interrogative visualization environment for large-scale engineering simulations

An interrogative visualization environment is described for the interactive display and querying of large datasets. The environment combines a web-based intelligent agent facility with a visualization engine. The intelligent agent facility (IAF) incorporates a rule-based expert system for natural-language understanding, voice and text input facilities, a hierarchical clickable command list, an interface for multimodal devices such as menu-based wireless handheld devices and gesture recognition devices, and human-like avatars acting as virtual assistants. The IAF interacts with, and controls, the visualization engine through a TCP/IP network socket interface. The environment enables multiple users using a variety of interaction modes and devices to effectively browse through large amounts of data, focus on and query interesting features, and more easily comprehend and make use of the data. Application of the environment to the visualization of engineering simulations is described.