Leap Motion Controller Research Articles

Augmented and virtual reality for physics: Experience of Kazakhstan secondary educational institutions

AbstractCurrently, computer‐based training systems, such as virtual laboratories (VLs), are becoming more common not only in universities but also in secondary schools. For physics, VLs are of great importance, as they allow simulating processes that cannot be seen or reproduced. Technologies such as augmented reality (AR) and virtual reality (VR) help us to create such physical visualizations. This study presents a VL, which allows users to solve various physical practical tasks or problems with the help of three‐dimensional process visualizations performed in three different ways, namely desktop VR, AR, and VR using a Leap Motion controller. This study describes in detail the functionality and implementation of a VL. As an experiment, three schools were provided with the necessary equipment and software. Fifty pupils of the 9th grade used the developed VL for half of the academic year, after which they took part in a comprehensive survey. About 88% of them found the laboratory useful for studying physics. The children were satisfied with its functionality and usability. Interviews with teachers helped us to understand which of the three visualization methods is preferable for them and children in the present environment.

An Experiential Learning Approach to Learning Manual Communication Through a Virtual Reality Environment

There is a pressing need for effective pedagogical methods of manual languages, as evident in the decline of manual languages, such as New Zealand Sign Language (NZSL). Despite being recognized as one of New Zealand's official languages, recent censuses have shown that fluent NZSL signers have been steadily decreasing. There is a cultural responsibility to preserve such languages, yet the combination of barriers to acquisition and the limited availability of effective teaching methods are standing in the way. In light of this, this article proposes a computer-assisted sign language learning system that incorporates virtual reality and validation-based feedback as tools to implement the experiential learning model. An implementation in the form of virtual reality NZSL is presented, targeting the set of NZSL alphabet. Results show that a vision-based classification method, using the Leap Motion controller, is a scalable, accurate, and usable solution for feedback-assisted manual language learning. A formative usability evaluation of $n=10$ participants showed promising results for engagement, confidence, and memory retention. The results indicate that virtual reality technology is uniquely situated as an innovative medium for the self-directed acquisition of manual languages. It is hoped this article inspires technology researchers to pursue collaborations with the deaf/Deaf community to design and develop pedagogical technology solutions for manual communication.

가상 환경에서의 손동작을 사용한 물체 조작에 대한 시각적 피드백 시스템

With the recent development of various kinds of virtual reality devices, there has been an active research effort to increase the sense of reality by recognizing the physical behavior of users rather than by classical user input methods. Among such devices, the Leap Motion controller recognizes the user

Использование технологии виртуальной реальности для разработки виртуальной физической лаборатории «Изучение момента инерции шара

The article discusses the use of virtual reality technology in education. The main goal of the study isto provide students with a convenient, useful and interesting source of knowledge for studying physics.The analysis of the use of various innovative developments in the field of education was conducted. It isshown that virtual reality is a promising tool for educational use, allowing you to create a safe, affordableand flexible environment for studying various processes. While working in virtual reality, a user receivesall the necessary information in an unusual, but very visual form. In the article the authors present asoftware application using virtual reality technology for studying the moment of inertia of a ball. Theapplication is made in the form of laboratory work with the possibility to conduct a virtual experiment.For clarity, the article provides screenshots of the application scenes; the application logic is given in theform of a UML activity diagram. The cross-platform Unity 3D was chosen as the development environment,which implements the principle of WORA (write once, run anywhere). Thanks to this principle,once written application can be built for personal computers, mobile devices, helmets of mixed realityand many other platforms. As a virtual reality device, the Leap Motion controller was used. The applicationfunctionality was written in the C # programming language. Graphic models were created usingSubstance Painter. The developed application can be used by high school students of secondary andspecialized schools, students of technical specialties of higher educational institutions, as well as anyinterested users.

Gesture Classification for a Hand Controller Device Using Neural Networks.

This paper presents a convolutional neural network-based classification of the hand flexion and extension gestures used in wrist recovery after injury. The hand gesture recognition device used in our study is the Leap Motion controller. The Leap Motion device's inability to accurately differentiate the left hand from the right hand when performing hand rotation gestures was eliminated by introducing hand and thumb direction vectors into the database used to train the neural network. A 3D environment was created for the introduction of the data describing the gesture into the database. A classification accuracy of 95% was achieved for the hand flexion and extension gesture divided into three levels for each hand. The populated database may also be used to classify other gestures involving hand rotation.

Hand gesture control and design of a rotary pneumatic soft actuator using leap motion sensor

The development and testing of a soft rotary actuator for gesture-based control is described in this paper. The soft rotary actuators are fabricated via a molding process and connected to a rotary joint as an antagonist and agonist pair, which gives rise to the clockwise and counter-clockwise rotation of the joint. The range of movement of the soft rotary actuator is 0°–90° for positive pressure actuation and 0° to − 16° for negative pressure actuation. The flexible robotic system is controlled using the leap motion controller, which is a gesture-based device. Gesture commands executed include circle, swipe, screen tap, and key tap gestures to produce clockwise, counter-clockwise, stop and start movements of the rotary actuator. The advantage of the novel soft rotary actuator is in its better integration and compatibility with natural and biological interfaces. The value of a gesture-based control with a soft actuator is the ease of interfacing and integrating the soft device with the control interface as well as its combined use for hand and finger rehabilitation using the soft rotary actuator.

Upper extremity rehabilitation using video games in cerebral palsy: a randomized clinical trial.

The aim of the present study was to compare the effects of neurodevelopmental therapy (NDT)-based upper extremity rehabilitation and video game-based therapy (VGBT) using Nintendo® wii and leap motion controller (LMC) games on upper extremity function in patients with cerebral palsy (CP). The 30 patients included in the present study were randomized to two groups: VGBT group (VGBT using Nintendo® Wii and LMC games) and control group (NDT-based upper extremity rehabilitation). Both groups trained 3days a week for 8weeks. Manual dexterity was evaluated using the "Minnesota Manual Dexterity Test (MMDT)", functional ability using the "Childhood Health Assessment Questionnaire (CHAQ)" and the "Duruoz Hand Index (DEI)" and grip and pinch strengths using a dynamometer. Following treatment, significant changes were found, MMDT, grip and pinch strength, CHAQ, and DHI scores in both groups (p < 0.05); however, VGBT group was statistically superior to group II with respect to changes in MMDT (p < 0.05). VGBT using Nintendo®Wii and LMC games had slightly superior effects on manual dexterity in patients with CP while compared with NDT-based upper extremity rehabilitation. Futhermore, the effects of both treatment programs on grip strengths and functional ability were similar beneficial.

Tone Classification Matches Kodàly Handsign with the K-Nearest Neighbor Method at Leap Motion Controller

&lt;p&gt;Hands can produce a variety of poses in which each pose can have a meaning or purpose that can be used as a form of communication determined according to a general agreement or who communicate. Hand pose can be used as human interaction with the computer is faster, intuitive, and in line with the natural function of the human body called Handsign. One of them is Kodàly Handsign, made by a Hungarian composer named Zoltán Kodály, which is a concept in music education in Hungary. This hand sign is used in interactive angklung performances in determining the tone that will be played by the K-Nearest Neighbor (KNN) algorithm classification process based on hand poses. This classification process is performed on the extracted data from Leap Motion Controller, which takes Pitch, Roll, and Yaw values based on basic aircraft principle. The results of the research were conducted five times with the value of k periodically 1,3,5,7,9 with test data consisting pose of 874 Do', 702 Si, 913 La, 612 Sol, 661 Fa, 526 Mi, 891 Re, and 1004 Do punctuation on 21099 training data. The test results can recognize hand poses with the optimal k value k=1 with an accuracy level of 94.87%.&lt;/p&gt;

Chronological pattern indexing: An efficient feature extraction method for hand gesture recognition with Leap Motion

Recently, Hand-Gesture-Recognition (HGR) systems has appreciably change the way of interaction between humans and computers thanks to advanced sensor technologies like the Leap-Motion-Controller (LMC). Despite the success achieved by many state-of-the-art methods, they have not worked on the rich temporal information existing in the sequential hand gesture data and characterizing the discriminative representation of different hand gesture classes. In this paper, we suggest a novel Chronological-Pattern-Indexing (CPI) approach which encodes the temporal orders of patterns for hand gesture time series data acquired by the LMC sensor. We extract a set of temporal patterns from different optimized projections. Then, we compare their temporal order and we encode the whole sequence with the index of the first coming pattern. We repeat these steps until we generate an efficient feature vector modeling the chronological dynamics of the hand gesture. The experiments demonstrate the potential of the proposed CPI approach for HGR systems.

User Experience in Excavator Simulator using Leap Motion Controller in Virtual Reality Environment

The risk that could happen in operating heavy machinery, such as excavator, can be avoided by doing special training to the operator. The operator of an excavator can learn the use of each joystick and lever in excavator by using a virtual reality without worrying about any damage. A virtual reality controller, such as Leap Motion Controller, can improve human computer interaction. To get a result of user experience in an excavator simulator, a survey of user agreement level and user satisfaction is required. The result of this research is none of the statements in the survey has an average score of 4.

Multimodal 3D American sign language recognition for static alphabet and numbers using hand joints and shape coding

American sign language recognition is still a research focus in computer vision community. Recently, most researches mainly extract low-level features for hand gesture recognition. These approaches perform poorly on recognizing gestures posed like a fist. In this paper, we propose a novel multimodal framework for sign language recognition system which exploits the Leap Motion Controller (LMC) and a webcam. We compute two sets of features. The first set is the angles at hand joints acquired by the LMC sensor. When, hand poses like a fist, the positions of the thumb joints captured by the LMC are not very precise. So, we should incorporate the second set of features extracted from the hand shape contour provided by a webcam. In this paper, we introduce a new mid-level feature, called Contour Segment Code (CSC), to represent hand shape contour. The proposed shape representation, first, extracts meaningful landmarks from the hand shape contour. CSC then encodes different segments of the hand contour into a code based on the shape landmarks. The extracted landmarks precisely determine the hand direction. The proposed method is tested by creating a very challenging dataset composed of 64,000 samples. Our experiments study the performance of the LMC and characteristics of CSC in different scenarios. The experimental results demonstrate the privileged performance of the proposed method against the systems which use depth images.

Supination/pronation movement quantification using stereoscopic vision based system towards Parkinson’s Disease assessment – A pilot study

Parkinson's disease is a neurodegenerative disorder that degrades the motor performance of patients. Specific motor symptoms, including tremor and bradykinesia, are clinically evaluated through the Unified Parkinson's disease Rating Scale (UPDRS), but with limited reproducibility and accuracy due to its intrinsic qualitative nature.A free-field and non-contact movement assessment method, based on stereoscopic imaging (Leap Motion controller) has been explored to precisely quantify supination/pronation (SUPRO) hand movements. A bench experiment was designed using a hand phantom to test its linearity. The Leap Motion controller presented a linear response considering rotation angles from −45° to 45° (r2 = 0.995).A serious game was developed to guide the data acquisition procedure, with a protocol similar to that of UPDRS, in 31 healthy subjects. The SUPRO hand movement was characterized in terms of root mean square values, predominant frequency, total spectral power and number of zero crossings. The control group achieved root mean square values of 9.60 ± 2.87 rad/s, a predominant frequency of 2.43 ± 0.52 Hz, a total spectral power of 927 ± 571 (rad/s)2 and a number of zero crossings of 275 ± 101. Also, a patient with Parkinson's disease was tested with a similar fashion, obtaining significantly different parameters.

Validity of a Fully-Immersive VR-Based Version of the Box and Blocks Test for Upper Limb Function Assessment in Parkinson's Disease.

In recent decades, gaming technology has been accepted as a feasible method for complementing traditional clinical practice, especially in neurorehabilitation; however, the viability of using 3D Virtual Reality (VR) for the assessment of upper limb motor function has not been fully explored. For that purpose, we developed a VR-based version of the Box and Blocks Test (BBT), a clinical test for the assessment of manual dexterity, as an automated alternative to the classical procedure. Our VR-based BBT (VR-BBT) integrates the traditional BBT mechanics into gameplay using the Leap Motion Controller (LMC) to capture the user’s hand motion and the Oculus Rift headset to provide a fully immersive experience. This paper focuses on evaluating the validity of our VR-BBT to reliably measure the manual dexterity in a sample of patients with Parkinson’s Disease (PD). For this study, a group of twenty individuals in a mild to moderate stage of PD were recruited. Participants were asked to perform the physical BBT (once) and our proposed VR-BBT (twice) system, separately. Correlation analysis of collected data was carried out. Statistical analysis proved that the performance data collected by the VR-BBT significantly correlated with the conventional assessment of the BBT. The VR-BBT scores have shown a significant association with PD severity measured by the Hoehn and Yahr scale. This fact suggests that the VR-BBT could be used as a reliable indicator for health improvements in patients with PD. Finally, the VR-BBT system presented high usability and acceptability rated by clinicians and patients.

Comparison study of user experience between virtual reality controllers, leap motion controllers, and senso glove for anatomy learning systems in a virtual reality environment

Virtual Reality (VR) can be used as an assistive system to learn with immersive visualization and interactive control. Anatomy is a branch of biology that is difficult to learn using textbooks because of its limitations in displaying anatomical objects in two-dimensional media. VR can display anatomical objects in the form of a three-dimensional virtual world. Besides being assisted by its immersive appearance, the interactivity learning system can also be improved by using the appropriate VR control device. In this study, three controls were used with different features and functions, namely VIVE Controller, Leap Motion Controller, and Senso Glove, and a user experience study was conducted. The user was asked to use the anatomy learning system with three different control devices, and then the user was asked to fill out a survey form, which uses the Liker scale. The survey focuses on the level of acceptability and user satisfaction in the usability factor, ease of learning, the suitability of movement, suitability of display, and haptic feedback provided by each VR controllers. From this study, it was found that the VIVE Controller is superior compared to the Leap Motion Controller and Senso Glove in terms of usability, ease of learning, movement suitability, and display suitability. The senso glove is superior in terms of haptic feedback satisfaction.

The Use of Virtual Reality Controllers and Comparison Between Vive, Leap Motion and Senso Gloves Applied in The Anatomy Learning System

The availability of control devices that use hand and finger movements in virtual reality systems has an essential role in increasing the level of interaction in the virtual reality system. Hand and finger movements in a virtual reality system can be displayed either in the form of a controller that resembles its original shape or in the form of a virtual hand. The translation motion and rotation of the virtual reality controller can be used as input to the system. In addition, the virtual reality controllers allow the use of buttons or gestures as triggers on the system. Input in the form of translational and rotational movements on the virtual reality controller and its triggers can be applied to anatomical learning systems in a virtual reality environment. Virtual reality controller has a role as a system control tools. Different virtual reality controllers have different levels of feasibility to the system, similarity when used, and ease of learning. Therefore a comparison was performed on three virtual reality controllers that have different usage methods and features. The three tools chosen in this study are the VIVE Controller, Leap Motion Controller, and Senso Glove. These three tools have different ways of using and feature, including differences in touching the user or not, triggers in the form of buttons or gestures, and whether or not haptic feedback is available. The analysis of the use and its comparison between these three virtual reality controllers are the outcome of this study.

Hand gesture recognition in real world scenarios using approximate string matching

New interaction paradigms combined with emerging technologies have produced the creation of diverse Natural User Interface (NUI) devices in the market. These devices enable the recognition of body gestures allowing users to interact with applications in a more direct, expressive, and intuitive way. In particular, the Leap Motion Controller (LMC) device has been receiving plenty of attention from NUI application developers because it allows them to address limitations on gestures made with hands. Although this device is able to recognize the position of several parts of the hands, developers are still left with the difficult task of recognizing gestures. For this reason, several authors approached this problem using machine learning techniques. We propose a classifier based on Approximate String Matching (ASM). In short, we encode the trajectories of the hand joints as character sequences using the K-means algorithm and then we analyze these sequences with ASM. It should be noted that, when using the K-means algorithm, we select the number of clusters for each part of the hands by considering the Silhouette Coefficient. Furthermore, we define other important factors to take into account for improving the recognition accuracy. For the experiments, we generated a balanced dataset including different types of gestures and afterwards we performed a cross-validation scheme. Experimental results showed the robustness of the approach in terms of recognizing different types of gestures, time spent, and allocated memory. Besides, our approach achieved higher performance rates than well-known algorithms proposed in the current state-of-art for gesture recognition.

The Impact of a Novel Immersive Virtual Reality Technology Associated with Serious Games in Parkinson's Disease Patients on Upper Limb Rehabilitation: A Mixed Methods Intervention Study.

Background: Parkinson’s disease is a neurodegenerative disorder that causes impaired motor functions. Virtual reality technology may be recommended to optimize motor learning in a safe environment. The objective of this paper was to evaluate the effects of a novel immersive virtual reality technology used for serious games (Oculus Rift 2 plus leap motion controller—OR2-LMC) for upper limb outcomes (muscle strength, coordination, speed of movements, fine and gross dexterity). Another objective was to obtain qualitative data for participants’ experiences related to the intervention. Methods: A mixed methods intervention (embedded) study was used, with a qualitative design after a technology intervention (quantitative design). The intervention and qualitative design followed international guidelines and were integrated into the method and reporting subheadings. Results: Significant improvements were observed in strength (p = 0.028), fine (p = 0.026 to 0.028) and gross coordination dexterity, and speed movements (p = 0.039) in the affected side, with excellent compliance (100%) and a high level of satisfaction (3.66 ± 0.18 points out of the maximum of 4). No adverse side effects were observed. Qualitative findings described patients’ perspectives regarding OR2-LMC treatment, facilitators and barriers for adherence, OR2-LMC applications, and treatment improvements. Conclusions: The intervention showed positive results for the upper limbs, with elements of discordance, expansion, and confirmation between qualitative and quantitative results.

Dynamic Hand Gesture Recognition Based on a Leap Motion Controller and Two-Layer Bidirectional Recurrent Neural Network.

Dynamic hand gesture recognition is one of the most significant tools for human–computer interaction. In order to improve the accuracy of the dynamic hand gesture recognition, in this paper, a two-layer Bidirectional Recurrent Neural Network for the recognition of dynamic hand gestures from a Leap Motion Controller (LMC) is proposed. In addition, based on LMC, an efficient way to capture the dynamic hand gestures is identified. Dynamic hand gestures are represented by sets of feature vectors from the LMC. The proposed system has been tested on the American Sign Language (ASL) datasets with 360 samples and 480 samples, and the Handicraft-Gesture dataset, respectively. On the ASL dataset with 360 samples, the system achieves accuracies of 100% and 96.3% on the training and testing sets. On the ASL dataset with 480 samples, the system achieves accuracies of 100% and 95.2%. On the Handicraft-Gesture dataset, the system achieves accuracies of 100% and 96.7%. In addition, 5-fold, 10-fold, and Leave-One-Out cross-validation are performed on these datasets. The accuracies are 93.33%, 94.1%, and 98.33% (360 samples), 93.75%, 93.5%, and 98.13% (480 samples), and 88.66%, 90%, and 92% on ASL and Handicraft-Gesture datasets, respectively. The developed system demonstrates similar or better performance compared to other approaches in the literature.

Recognition of sign language hand gestures using leap motion sensor based on threshold and ANN models

Hand gesture recognition is a topic that is still investigated by many scientists for numerous useful aspects. This research investigated hand gestures for sign language number zero to nine. The hand gesture recognition was based on finger direction patterns. The finger directions were detected by a Leap Motion Controller. Finger direction pattern modeling was based on two methods: threshold and artificial neural network. Threshold model 1 contained 15 rules based on the range of finger directions on each axis. Threshold model 2 was developed from model 1 based on the behavior of finger movements when the subject performed hand gestures. The ANN model of the system was designed with four neurons at the output layer, 15 neurons at the input layer, seven neurons at the first hidden layer and 5 neurons at the second hidden layer. The artificial neural network used the logsig as the activation function. The result shows that the first threshold model has the lowest accuracy because the rule is too complicated and rigid. The threshold model 2 can improve the threshold model, but it still needs development to reach better accuracy. The ANN model gave the best result among the developed model with 98% accuracy. LMC produces useful biometric data for hand gesture recognition.

Estimation of Hand Motion from Piezoelectric Soft Sensor Using Deep Recurrent Network

Soft sensors are attracting significant attention in human–machine interaction due to their high flexibility and adaptability. However, estimating motion state from these sensors is difficult due to their nonlinearity and noise. In this paper, we propose a deep learning network for a smart glove system to predict the moving state of a piezoelectric soft sensor. We implemented the network using Long-Short Term Memory (LSTM) units and demonstrated its performance in a real-time system based on two experiments. The sensor’s moving state was estimated and the joint angles were calculated. Since we use moving state in the sensor offset calculation and the offset value is used to estimate the angle value, the accurate moving state estimation results in good performance for angle value estimation. The proposed network performed better than the conventional heuristic method in estimating the moving state. It was also confirmed that the calculated values successfully mimic the joint angles measured using a leap motion controller.