Leap Motion Device Research Articles

NeuralPMG: A Neural Polyphonic Music Generation System Based on Machine Learning Algorithms

The realm of music composition, augmented by technological advancements such as computers and related equipment, has undergone significant evolution since the 1970s. In the field algorithmic composition, however, the incorporation of artificial intelligence (AI) in sound generation and combination has been limited. Existing approaches predominantly emphasize sound synthesis techniques, with no music composition systems currently employing Nicolas Slonimsky’s theoretical framework. This article introduce NeuralPMG, a computer-assisted polyphonic music generation framework based on a Leap Motion (LM) device, machine learning (ML) algorithms, and brain-computer interface (BCI). ML algorithms are employed to classify user’s mental states into two categories: focused and relaxed. Interaction with the LM device allows users to define a melodic pattern, which is elaborated in conjunction with the user’s mental state as detected by the BCI to generate polyphonic music. NeuralPMG was evaluated through a user study that involved 19 students of Electronic Music Laboratory at a music conservatory, all of whom are active in the music composition field. The study encompassed a comprehensive analysis of participant interaction with NeuralPMG. The compositions they created during the study were also evaluated by two domain experts who addressed their aesthetics, innovativeness, elaboration level, practical applicability, and emotional impact. The findings indicate that NeuralPMG represents a promising tool, offering a simplified and expedited approach to music composition, and thus represents a valuable contribution to the field of algorithmic music composition.

Comparison of Data Classification Results in Serious Gaming for Rehabilitation of Rheumatoid Arthritis.

Rheumatoid arthritis is a common disease which affects the joints of the wrist, fingers, feet and in the end the daily activities. Nowadays, gestures and virtual reality are used in many activities supporting recovery, games, learning as technology is present more and more in different fields. This paper presents results related to the grip movement detected by a Leap Motion device using binary classification and machine learning algorithms. We used 2 models to compare the results: Naïve Bayes and Random Forest Classifier. The metrics for comparison were: accuracy, precision, recall and f1-score. Also, we create a confusion matrix for a clear visualization of the results. We used 5000 data to train the algorithm and 1500 data to test. The accuracy and the precision were bigger than 97% in all the cases.

Usability Comparison between 2D and 3D Control Methods for the Operation of Hovering Objects

This paper experimentally analyzed the cognitive load of users based on different methods of operating hovering objects, such as drones. The traditional gamepad-type control method (2D) was compared with a control method that mapped the movement directions of the drone to the natural manipulation gestures of the user using a Leap Motion device (3D). Twenty participants operated the drone on an obstacle course using the two control methods. The drone’s trajectory was measured using motion-capture equipment with a reflective marker. The distance traveled by the drone, operation time, and trajectory smoothness were calculated and compared between the two control methods. The results showed that when the drone’s movements were mapped to the user’s natural directional gestures, the drone’s 3D movements were perceived as more natural and smoother. A more intuitive drone control method can reduce cognitive load and minimize operational errors, making it more user friendly and efficient. However, due to the users’ lack of familiarity with Leap Motion, it resulted in longer distance and time and lower subjective satisfaction; therefore, a more improved 3D control method over Leap Motion is needed to address the limitations.

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%.

Target selection method on the occluded and distant object in handheld augmented reality

<span lang="EN-US">Existing interaction techniques within handheld augmented reality (AR) have frequently used touchscreen input (pure two-dimensional (2D) pointing and clicking) from the handheld device's display for target selection on the virtual object. However, performing accurate target selection on a distant target object becomes challenging as the target object will appear smaller when the distance increases. Aside from that, the difficulty increases in performing target selection when another virtual object obscures the distant virtual object. Therefore, this study aims to present a target selection method to perform the target selection. We enable the raycasting technique with real hand gesture for the target selection method on the occluded and distant object in handheld AR. The leap motion device is mounted at the back of the handheld device to track the real hand gesture. The markerless tracking technology of simultaneous localization and mapping (SLAM) is implemented to enable the AR environment. Based on the results, the aim of this study was achieved.</span>

Interaction Design of Rural Brand IP Image Based on Processing Algorithmic Art

Processing-based programming interactive art has gradually become a research hotspot. To improve users' immersion and experience, this paper proposes a rural brand IP interactive design method based on a Processing algorithm. While solving the contradiction between the graphical perceptual thinking of rural brand IP and the logic of computer language architecture, it also interconnects with Arduino, Kinect, Leap Motion, and other devices. It further provides ideas for multi-sensing interactive rural brand IP design.

Interaction Design of Rural Brand IP Image Based on Processing Algorithmic Art

Processing-based programming interactive art has gradually become a research hotspot. To improve users' immersion and experience, this paper proposes a rural brand IP interactive design method based on a Processing algorithm. While solving the contradiction between the graphical perceptual thinking of rural brand IP and the logic of computer language architecture, it also interconnects with Arduino, Kinect, Leap Motion, and other devices. It further provides ideas for multi-sensing interactive rural brand IP design.

Comparison of Data Classification Results for Leap Motion Recovery Gestures.

Static and dynamic gestures are frequently used in activities supporting learning, recovery healthcare, engineering, and 3D games to increase the interactivity between man and machine. The gestures are detected via hardware devices and data is processed using different software methods. This paper presents the manner of detection and interpretation of two gestures, a hand rotation gesture and a palm closing and opening gesture, using the Leap Motion device. These two dynamic gestures are very often used in hand recovery exercises. For comparing the two gestures we use data classification methods, Support Vector Machine (SVM) and Multilayer Perceptron (MLP). The data for the gesture classification were 80% training data and 20% testing data. The metrics for comparison are precision, recall, F1-score, and the total number of testing cases (support). The SVM classifier gives an accuracy of 99.4% and the MLP classifier a 96.2%. We built two confusion matrices for better visualizing the results.

Non-Contact Hand Movement Analysis for Optimal Configuration of Smart Sensors to Capture Parkinson's Disease Hand Tremor.

Parkinson’s disease affects millions worldwide with a large rise in expected burden over the coming decades. More easily accessible tools and techniques to diagnose and monitor Parkinson’s disease can improve the quality of life of patients. With the advent of new wearable technologies such as smart rings and watches, this is within reach. However, it is unclear what method for these new technologies may provide the best opportunity to capture the patient-specific severity. This study investigates which locations on the hand can be used to capture and monitor maximal movement/tremor severity. Using a Leap Motion device and custom-made software the volume, velocity, acceleration, and frequency of Parkinson’s (n = 55, all right-handed, majority right-sided onset) patients’ hand locations (25 joints inclusive of all fingers/thumb and the wrist) were captured simultaneously. Distal locations of the right hand, i.e., the ends of fingers and the wrist showed significant trends (p < 0.05) towards having the largest movement velocities and accelerations. The right hand, compared with the left hand, showed significantly greater volumes, velocities, and accelerations (p < 0.01). Supplementary analysis showed that the volumes, acceleration, and velocities had significant correlations (p < 0.001) with clinical MDS-UPDRS scores, indicating the potential suitability of using these metrics for monitoring disease progression. Maximal movements at the distal hand and wrist area indicate that these locations are best suited to capture hand tremor movements and monitor Parkinson’s disease.

A comparative analysis of two immersive virtual reality systems in the integration and visualization of natural hand interaction

It is generally accepted that the use of natural interaction provides a positive impact in Virtual Reality (VR) applications. Therefore, it is important to understand what is the best way to integrate and visualize this feature in VR. For this reason, this paper presents a comparative study of the integration of natural hand interaction in two immersive VR systems: a Cave Audio Visual Experience (CAVE) system –where users’ real hands are visible– and a non-see-through Head-Mounted Display (HMD) system –where only a virtual representation of the hands is possible–. In order to test the suitability of using this type of interaction in a CAVE and compare it to an HMD, we raise six research questions related to task performance, usability and perception differences regarding natural hand interaction with these two systems. To answer these questions, we designed an experiment where users have to complete a pick-and-place task with virtual balls and a text-typing task with virtual keyboards. In both systems, the same tracking technology, based on a Leap Motion device, was used. To the best of our knowledge this is the first academic work addressing a comparison of this type. Objective and subjective data were collected during the experiments. The results show that the HMD has a performance, preference and usability advantage over the CAVE with respect to the integration of natural hand interaction. Nevertheless, the results also show that the CAVE system can be, as well, successfully used in combination with an optical hand tracking device.

Development, Control Adjustment, and Gesture Recognition of a Quadrotor Helicopter

Quadrotor helicopters (quadcopters) have become popular in recent years; because they are simple to operate and steady, they are used in aerial photography, competitive flying, and search-and-rescue missions. In addition, wireless sensors have enabled gesture recognition, and therefore, this study investigated the use of gesture recognition to control a quadcopter. A quadcopter was built using Arduino NANO and MPU-6050 modules, which provided insights into the flight principles of a quadrotor system; furthermore, a proportional integrative derivative controller was modified for steady flight. A Leap Motion device was used to understand the logic behind hand gestures and test the success rate of each gesture, and was then combined with a Parrot AR.Drone 2.0 to achieve gesture recognition controls and uncover problems with connections between hand gestures. These problems were addressed by changing the code in the gesture control program. The results of the completed gesture control experiment demonstrated high gesture recognition performance as well as the ability to meet the requirements for controlling a quadcopter.

Research on a method of creating digital shadow puppets based on parameterized templates

Shadow play is an ancient traditional folk art in China and a precious Chinese artistic treasure. However, with the continuous diversification of entertainment forms, traditional shadow play art is gradually fading out of sight and even facing the possibility of having no successor. A fast method of creating digital shadow puppets based on parameterized templates is proposed in this paper. Adopting this method can transform the materials of a shadow character designed by an artist into a digital shadow puppet that can be interactively controlled by hand gestures. This paper classifies digital shadow puppets based on an analysis and summary of traditional shadow play art and designs the corresponding parameterized template for each type. In addition, a control method for shadow puppets based on the Leap Motion device is designed and implemented in this paper in accordance with common human hand gestures and the requirements for controlling shadow puppets. Based on the above work, we design and implement a platform for digital shadow story creation and performance and test the proposed method on this platform. The experimental results show that the method proposed in this paper can quickly create digital shadow puppets that can be interactively controlled and allow shadow plays to be performed based on gesture control. The research ideas and achievements of this paper help promote the development of digital shadow play technology and have great significance to the inheritance and development of traditional shadow play art.

Convolutional Neural Networks Based Motion Data Optimization Networks forLeap Motion

<p indent=0mm>It is necessary to improve the capture accuracy and precision of Leap Motion equipment, and solve its inherent problems such as finger self-occlusion and instable sampling frequency, firstly, a multi-modal synchronous hand motion capture scheme is proposed based on Leap Motion and motion capture devices, and the dataset is captured correspondingly. Secondly, a convolutional-neural-network-based hand motion data optimization method for Leap Motion is presented. The proposed network is trained to learn the mapping from Leap Motion data domain to motion capture data domain with the synchronous dataset. Finally, a coplanar constraint for human fingers is proposed, which makes the outputs maintain stable hand skeleton structure. The 967 550 frames of synchronous hand motion data are captured in 6 categories from 15 volunteers. Experiments are designed for validating of the finger coplanar constraint, testing the consistency of the optimized motion data, and the comparison with BRA (bidirectional recurrent autoencoder) and EBD (encoder-bidirectional-decoder) methods. Experiments indicate that proposed method supports for capturing hand motion data with fixed sample frequency and MoCap-like precision using a Leap Motion. Furthermore, the proposed method can obtain smoother and more stable results than BRA and EBD. The method is also applied in rehabilitation games, which significantly reduces the number of errors of hand interaction motion recognition.

A Study Protocol for Checking Efficacy of Leap Motion Device on Gross Hand Dexterity in Sub-Acute Stroke Patients

Introduction: Stroke survivors have limited everyday living tasks, often because of dexterous issues.Videogame-based training (VBT) with the appearance of virtual reality helps to improve the role of theupper limbs. The leap Motion controller can track both hands and fingers’ fine movements. The studiesthus demonstrates the efficacy of the leap motion system on gross hand dexterity in patients with sub-acutestroke.Method: The research has been designed as an experimental study. The total of 40 participants will be takenfrom AVBRH, sawangi Meghe for study as per inclusion and exclusion criteria. With intervention the spanof the study will be 6 months. Leap Motion-based, augmented reality training will be provided to patientsfor half hour, every single day, Five days of the week a month. Pre and post, box and block test and Systemusability scale will be taken. Those two will be the patient’s major outcome.Discussion:-This study protocol aims to assess the efficacy of the leap motion controller on the rehabilitationof gross hand dexterity in patients with stroke. The study’s expected outcome will concentrate on theevaluation of the usability of VBT using the Leap Motion Controller (LMC) to train gross hand dexterity instroke patients’ early recovery process.

Design and Evaluation of Anthropomorphic Robotic Hand for Object Grasping and Shape Recognition

We developed an anthropomorphic multi-finger artificial hand for a fine-scale object grasping task, sensing the grasped object’s shape. The robotic hand was created using the 3D printer and has the servo bed for stand-alone finger movement. The data containing the robotic fingers’ angular position are acquired using the Leap Motion device, and a hybrid Support Vector Machine (SVM) classifier is used for object shape identification. We trained the designed robotic hand on a few monotonous convex-shaped items similar to everyday objects (ball, cylinder, and rectangular box) using supervised learning techniques. We achieve the mean accuracy of object shape recognition of 94.4%.

Virtual 3D environment for exploring the spatial ability of students

BackgroundSpatial ability is an unique type of intelligence; it can be distinguished from other forms of intelligence and plays an essential role in an individual's success in many academic fields, particular in this era of technology. Instruction-assisted 3D technology can display stereo graphics and promote students' understanding of the geometrical structure and characteristics of graphics. Spatial ability includes several aspects. Few software programs are available for training different aspects of spatial ability for senior high school students. This study aims to explore an effective method for training the spatial ability for senior high school students, and to promote the development of students' independent inquiry ability. MethodsFirst, an inquiry design strategy to improve the spatial ability of students is proposed. Based on this strategy, unity3D was used to develop a 3D inquiry environment that can use leap motion to complete a gesture interaction. Finally, researchers carried out experience-based activities and issued user experience questionnaires to participants to verify the application effect of the spatial ability inquiry environment and used interviews to understand the user experience of participants exploring the leap motion device in a 3D inquiry environment. Results32 learners participated in the experiment. learners have a high score for perceived usefulness and willingness to use. Compared with the perceived ease of use and perceived usefulness, the average score of the application effect is relatively low. In terms of willingness to use, most of the learners expressed their willingness to use a similar inquiry environment for spatial ability training in the future. ConclusionsThe spatial ability inquiry environment can help learners better understand different concepts. The users showed a strong willingness to continue using the device. The device also updates the teaching concept to a certain extent and emphasizes the dominant position of a student.

SFINGE 3D: A novel benchmark for online detection and recognition of heterogeneous hand gestures from 3D fingers’ trajectories

In recent years gesture recognition has become an increasingly interesting topic for both research and industry. While interaction with a device through a gestural interface is a promising idea in several applications especially in the industrial field, some of the issues related to the task are still considered a challenge. In the scientific literature, a relevant amount of work has been recently presented on the problem of detecting and classifying gestures from 3D hands’ joints trajectories that can be captured by cheap devices installed on head-mounted displays and desktop computers. The methods proposed so far can achieve very good results on benchmarks requiring the offline supervised classification of segmented gestures of a particular kind but are not usually tested on the more realistic task of finding gestures execution within a continuous hand tracking session.In this paper, we present a novel benchmark, SFINGE 3D, aimed at evaluating online gesture detection and recognition. The dataset is composed of a dictionary of 13 segmented gestures used as a training set and 72 trajectories each containing 3–5 of the 13 gestures, performed in continuous tracking, padded with random hand movements acting as noise. The presented dataset, captured with a head-mounted Leap Motion device, is particularly suitable to evaluate gesture detection methods in a realistic use-case scenario, as it allows the analysis of online detection performance on heterogeneous gestures, characterized by static hand pose, global hand motions, and finger articulation.We exploited SFINGE 3D to compare two different approaches for the online detection and classification, one based on visual rendering and Convolutional Neural Networks and the other based on geometry-based handcrafted features and dissimilarity-based classifiers. We discuss the results, analyzing strengths and weaknesses of the methods, and deriving useful hints for their improvement.

Gesture Classification for a Hand Controller Device Using Neural Networks.

This paper presents a convolutional neural network-based classification of the hand flexion and extension gestures used in wrist recovery after injury. The hand gesture recognition device used in our study is the Leap Motion controller. The Leap Motion device's inability to accurately differentiate the left hand from the right hand when performing hand rotation gestures was eliminated by introducing hand and thumb direction vectors into the database used to train the neural network. A 3D environment was created for the introduction of the data describing the gesture into the database. A classification accuracy of 95% was achieved for the hand flexion and extension gesture divided into three levels for each hand. The populated database may also be used to classify other gestures involving hand rotation.

A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion

Due to the recent development of machine learning and sensor innovations, hand gesture recognition systems become promisingfor the digital entertainment field. In this paper, we propose a dynamic hand gesture recognition approach using touchless hand motions over a Leap Motion device. First, we analyze the sequential time series data gathered from Leap Motion using Long Short-Term Memory (LSTM) recurrent neural networks for recognition purposes. We exploit basic unidirectional LSTM and bidirectional LSTM separately. Then, we propound novel architecture by combining the aforementioned models with additional components to give a final prediction network, named Hybrid Bidirectional Unidirectional LSTM (HBU-LSTM). The suggested network improves the model performance significantly by considering the spatial and temporal dependencies between the Leap Motion data and the network layers during the forward and backward pass. The recognition models are examined on two available benchmark datasets, named the LeapGestureDB dataset and the RIT dataset. Experiments demonstrate the potential of the proposed HBU-LSTM network for dynamic hand gesture recognition, with an average recognition rate reaching approximately 90%. Our suggested approach reaches superior performance, in terms of accuracy and computational complexity, over some existing methods for hand gesture recognition.

An Interactive Real-Time Cutting Technique for 3D Models in Mixed Reality

This work describes a Mixed Reality application useful to modify and cut virtual objects. A digital simulation of surgical operations is presented. Following this approach, surgeons can test all the designed solutions of the preoperative stage in a Mixed Reality environment. High precision in surgery applications can be achieved thanks to the new methodology. The presented solution is hands free and does not need the use of a mouse or computer’s keyboard: it is based on HoloLens, Leap Motion device and Unity. A new cutting algorithm has been developed in order to handle multiple objects and speed up the cut with complex meshes and preserve geometry quality. A case study presents the cut of several bones in order to simulate surgeon’s operations. A reduction in cut time compared to the original method is noticed, together with a high flexibility of the tool and a good fidelity of the geometry. Moreover, all the object fragments generated from the algorithm are available for manipulation and new cuts.