Leap Motion Research Articles

Analysis of the Leap Motion Controller Workspace for HRI Gesture Applications

The Leap Motion Controller is a sensor for precise hand tracking; it is a device used for human interaction with computer systems via gestures. The study presented in this paper evaluates its workspace in real-world conditions. An exact replica of a human operator’s hand was used to measure the sensor’s precision, and therefore determine its hand tracking abilities in varying positions above the sensor. The replica was moved randomly across the workspace defined by the manufacturer, and precision was measured in each position. The hand model was placed in the furthest distances from the sensor to find every position where the sensor was still capable of tracking. We found the dimensions of the workspace in some cases exceeded the datasheet values; in other cases, the real workspace was smaller than the proclaimed one. We also computed precision in all positions, which shows tracking reliability. This study serves researchers developing HMI and HRI algorithms as a reference for the real dimensions of the Leap Motion Controller workspace as it provides extra and more precise information compared to the datasheet.

Robust Identification System for Spanish Sign Language Based on Three-Dimensional Frame Information

Nowadays, according to the World Health Organization (WHO), of the world's population suffers from a hearing disorder that makes oral communication with other people challenging. At the same time, in an era of technological evolution and digitization, designing tools that could help these people to communicate daily is the base of much scientific research such as that discussed herein. This article describes one of the techniques designed to transcribe Spanish Sign Language (SSL). A Leap Motion volumetric sensor has been used in this research due to its capacity to recognize hand movements in 3 dimensions. In order to carry out this research project, an impaired hearing subject has collaborated in the recording of 176 dynamic words. Finally, for the development of the research, Dynamic Time Warping (DTW) has been used to compare the samples and predict the input with an accuracy of 95.17%.

Optimal Signal Processing for Steady Control of a Robotic Arm Suppressing Hand Tremors for EOD Applications

Teleoperated robotics in recent years has proven to be valuable support in EOD tasks; a remarkable improvement in the systems that control these robots has been the Natural User Interfaces (NUI); however, the research that implements this type of system does not focus on the stability of the robotic arm movements, necessary for this type of applications due to the danger of working with explosives. In this paper, we propose the implementation of an Optimal Signal Processing for a NUI interface based on the Leap Motion (LM) controller. The main objective of this research is to correctly identify the intentional movements of the operator, achieve high stability of the robotic gripper and suppress the physiological tremors from the hand of the operator, considering not to increase the mental workload and not decrease the usability of the system. The signal processing proposed in this paper is composed of three filtering algorithms: Kalman, FIR, and moving average with a threshold. In addition, the obtained results are compared with the most representative processing of recent research using LM for robotic arm control. To evaluate and validate the proposed signal processing, a target path tracking test, a stability analysis of the robotic gripper, and a performance analysis in the execution of Pick and Place tasks, NASA-TLX and SUS questionnaires are developed. Finally, the proposed Optimal Signal processing is implemented in the DOBOT-MAGICIAN and tested by police officers of the EOD Unit-Arequipa (UDEX-AQP); the results indicate a reduction of the average Vibration of 31.61% and the Target Path Tracking error of 67.57%.

Neurodata Tracker: Software for computational assessment of hand motor skills based on optical motion capture in a virtual environment.

Deficits affecting hand motor skills negatively impact the quality of life of patients. The NeuroData Tracker platform has been developed for the objective and precise evaluation of hand motor deficits. We describe the design and development of the platform and analyse the technological feasibility and usability in a relevant clinical setting. A software application was developed in Unity (C#) to obtain kinematic data from hand movement tracking by a portable device with two cameras and three infrared sensors (leap motion®). Four exercises were implemented: (a) wrist flexion-extension (b) finger-grip opening-closing (c) finger spread (d) fist opening-closing. The most representative kinematic parameters were selected for each exercise. A script in Python was integrated in the platform to transform real-time kinematic data into relevant information for the clinician. The application was tested in a pilot study comparing the data provided by the tool from ten healthy subjects without any motor impairment and ten patients diagnosed with a stroke with mild to moderate hand motor deficit. The NeuroData Tracker allowed the parameterization of kinematics of hand movement and the issuance of a report with the results. The comparison of the data obtained suggests the feasibility of the tool for detecting differences between patients and healthy subjects. This new platform based on optical motion capturing provides objective measurement of hand movement allowing quantification of motor deficits. These findings require further validation of the tool in larger trials to verify its usefulness in the clinical setting.

An Innovative Arabic Text Sign Language Translator

In recent years, several software and hardware solutions have been proposed for object detection, motion tracking, and gesture identification. However, these solutions failed to efficiently identify appropriate gestures and track their motion in a stipulated time range. To overcome the above-mentioned challenges, we propose a novel sign language translator application, which uses the MediaPipe Holistic model along with Long Short-Term Memory (LSTM), integrated with Neural Network (NN) model to build the proposed translator model. This model will pick up the gestures from different angles quickly and accurately to translate them into appropriate text. This paper has been divided into two parts, the first part involves research and data collection, and the second part involves training and testing the collected data for real-time implementation. This model has been trained and tested with real-time data collected from the Leap motion controller and cameras. The model provides appreciable results.

A hand motion capture method based on infrared thermography for measuring fine motor skills in biomedicine

Many biomedical applications require fine motor skill assessments; however, real-time and contactless fine motor skill assessments are not typically implemented. In this study, we followed the 2D-to-3D pipeline principle and proposed a transformer-based spatial–temporal network to accurately regress 3D hand joint locations by inputting infrared thermal video for eliminating need of multiple cameras or RGB-D devices. We also developed a dataset composed of infrared thermal videos and ground truth annotations for training. The label represents a set of 3D joint locations from infrared optical trackers, which is considered the gold standard for clinical applications. To demonstrate their potential, the proposed method was used to measure the finger motion angle, and we investigated its accuracy by comparing the proposal with the Azure Kinect system and Leap Motion system. On the proposed dataset, the proposed method achieved a 3D hand pose mean error of less than 14 mm and outperforms the other deep learning methods. When the error thresholds were larger than approximately 35 mm, our method first to achieved excellent performance (>80%) in terms of the fraction of good frames. For the finger motion angle calculation task, the proposed and commercial systems had comparable inter-system reliability (ICC2,1 ranging from 0.81 to 0.83) and excellent validity (Pearson’s r-values ranging from 0.82 to 0.86). We believe that the proposed approaches can capture hand motion and measure finger motion angles and can be used in different biomedicine scenarios as an effective evaluation tool for fine motor skills.

Application of virtual reality for peritoneal dialysis exchange learning in patients with end-stage renal disease and cognitive impairment.

Cognitive impairment is not uncommon in patients with end-stage renal disease and can make it more difficult for these patients to carry out peritoneal dialysis (PD) on their own. Their attempts to do so may result in adverse consequences such as peritonitis. PD exchange is a complex procedure demanding knowledge and skill which requires close supervision and guidance by a renal nurse specialist. In this study, a non-immersive virtual reality (VR) training program using a Leap motion hand tracking device was developed to facilitate patients' understanding and learning of the PD exchange procedure before attempting real task practice. This study was a two-center single-blinded randomized controlled trial on 23 incident PD patients. Patients in the experimental group received 8 sessions of VR training, while patients in the control were provided with printed educational materials. The results showed that there were significant differences between the two groups in performance of the overall PD exchange sequence, especially on the crucial steps. VR had a patient satisfaction rate of 89%, and all patients preferred to have the VR aid incorporated in PD training. Our findings conclude VR can be a useful aid in the training and reinforcement of PD exchange procedures, with distinct merits of being free from restrictions of time, space, and manpower.

Design, manufacture, and control of a laparoscopic robot via Leap Motion sensors

Minimally invasive surgeries are less likely to cause an infection, bleeding, or postoperative complications. They can, however, only be accomplished with sufficient skill and experience on the part of the surgeon. The present study mainly aims to provide the context for the remote control of laparoscopic devices and improve the performance of minimally invasive surgeries so that all patients can have access to skilled surgeons beyond geographical limitations. The leap motion controller first acquired the fine movements of the surgeons' hands, the position and gesture of the fingers, and an awareness of the alterations of the corresponding angles and coordinates, which were required in every instant. A 5-DOF robot is built and controlled using data from the Leap Motion sensor to control the laparoscopic grasper. These data, which represent the surgeon's hand coordinates, are wirelessly input to the Raspberry Pi board via TCP and then transmitted to the robot's actuators after being substituted into the corresponding kinematic equations. Encoders embedded in the joints of the robot are used for closed-loop control using a PID controller with a feed-forward compensator. The average accuracy of the prototype robot was calculated to be better than 4° by computing the RMSE of every joint. Moreover, the accuracy of the lead screw mechanism was obtained to be 0.4141 mm in opening and 0.6162 mm in closing the grasper jaws. Additionally, in experimental tests, the Leap Motion sensor's error was discovered to be up to 5 mm, and the viability of lowering this error by using Kalman filtering was investigated.

POSSIBILITY OF AUTOMATED ERGONOMIC EVALUATIONS IN VIRTUAL REALITY ENVIRONMENTS

The aim of the present research was to create a method for detecting the position and motions of the wrist and fingers at work, to design and create an experimental device to measure and evaluate the training task. Therefore, the issue of finger motions was searched, and the evaluation method was proposed. The experimental device used an optical device for motion monitoring, the so-called Leap Motion, which exports the measured data to a virtual environment created in the Unity3D development environment. This environment enables to read the required partial turning of the individual elements of the hand using the script. The training task consisted of a simple assembly. The evaluation is presented in this article.

A Pilot Study of Compensatory Strategies for Reach-to-Grasp-Pen in Patients with Stroke.

Coordinated reaching and grasping movements may be impaired in patients with poststroke hemiplegia. Patients frequently adopt compensatory strategies, which require investigation. This pilot study used kinematic parameters to examine compensatory strategies by assessing the reach-to-grasp-pen movements in patients with stroke and unaffected participants. Twelve patients with stroke with mild impairment (45.16 ± 12.62 years, 2.41 ± 1.97 months), twelve with moderate impairment (50.41 ± 12.92 years, 3.83 ± 3.58 months), and ten healthy individuals (20.6 ± 0.69 years) performed a reach-to-grasp-pen task. Kinematics parameters of upper limb and fingers, such as movement time, number of movement units, index of curvature, spectral arc length, trunk forward transition, trunk lateral transition, elbow extension, shoulder flexion, shoulder abduction, trunk rotation, arm-plane angle, the joint angles of interphalangeal joints of the thumb, index, middle, ring, and little fingers were examined in the study. These parameters were evaluated with two Microsoft Azure Kinect and Leap Motion, which belong to markerless motion capture systems. Patients with stroke showed longer reaching movement time, less smooth movement trajectories, and more trunk rotation (P < 0.05). In patients with stroke, the metacarpophalangeal joint (MCP) and proximal interphalangeal joint (PIP) of the thumb were flexed in the starting position; the MCP and PIP joints of the index finger in the stroke group were more extended during pen grasp; the range of motion of the MCP of the middle finger and the PIP joints of the middle, ring, and little fingers became greater, suggesting a larger peak aperture (P < 0.05). The more significant extension was observed in the index finger at the end of the grasp, suggesting inadequate flexion (P < 0.05). In clinical practice, the reach-to-grasp-pen task using markless sensing technology can effectively distinguish patients with stroke from healthy individuals and evaluate the recovery and compensation strategies of upper limb and hand functions. It can potentially become an evaluation tool in hospital and community scenes. Accurate identification of abnormal trunk, arm, and finger strategies is crucial for therapists to develop targeted upper limb treatment methods and evaluate treatment effects.

Classification of Activities of Daily Living Based on Grasp Dynamics Obtained from a Leap Motion Controller.

Stroke is one of the leading causes of mortality and disability worldwide. Several evaluation methods have been used to assess the effects of stroke on the performance of activities of daily living (ADL). However, these methods are qualitative. A first step toward developing a quantitative evaluation method is to classify different ADL tasks based on the hand grasp. In this paper, a dataset is presented that includes data collected by a leap motion controller on the hand grasps of healthy adults performing eight common ADL tasks. Then, a set of features with time and frequency domains is combined with two well-known classifiers, i.e., the support vector machine and convolutional neural network, to classify the tasks, and a classification accuracy of over 99% is achieved.

Wireless Control of Six-axis Robot Arms by Leap Motion Sensor

Wireless Control of Six-axis Robot Arms by Leap Motion Sensor

Rehabilitation of hand through leap motion control on burn injury patients

Burn injuries, especially severe burns, triggers immune and inflammation response, disturbances of metabolism and distributive type of shock, all of this can be said as hard to treat and may lead to cause multiple organ failure. The fact that the injury not only effects physical health but also mental health his mental health and quality of life is critical. As a result, people with burn injuries cannot be deemed recovered if wounds have healed; rather, burn injuries cause long-term fundamental changes that must be focused in order to improve quality of life. As a result, burn care providers encounter a variety of issues, including managing acute and critical care, long-term care, and rehabilitation. Hand is the most commonly injured body part out of all accidents that are work related , ranks third of all workplace injuries and necessitating surgical and lengthy recovery. Hand burns make it difficult to function. The rehabilitation process has benefited from the usage of virtual reality (VR). LMC is a form of virtual reality game that uses new technology to provide biofeedback and training for fine motor function and functional skills. According to this review paper, leap motion training will enable hand burns patient to enhance finger mobility, reduce scar thickness, and improve hand function along with enhancing quality of life of patient. Although improvement in condition depends on severity of burn but this gadget proved to be a boon for hand rehab. on burn population. . Finally in conclusion of review it can be said that leap motion training may aid patients with hand burns in increasing finger range of motion, reducing scar thickness, and improving hand function.

Gamifying rehabilitation: MILORD platform as an upper limb motion rehabilitation service

Motor learning is based on the correct repetition of specific movements for their permanent storage in the central nervous system (CNS). Rehabilitation relies heavily on the repetition of specific movements, and game scenarios are ideal environments to build routines of repetitive exercises that have entertaining characteristics. In this respect, the gamification of the rehabilitation program, through the introduction of game-specific techniques and design concepts, has gained attention as a complementary or alternative to routine rehabilitation programs. A gamified rehabilitation program promises to gain the patient's attention, to reduce the monotony of the process and preserve motivation to attend, and to create virtual incentives through the game, toward maintaining compliance to the “prescribed” program. This is often achieved through goal-oriented tasks and real-time feedback in the form of points and other in-game rewards. This paper describes MILORD rehabilitation platform, an affordable technological solution, which aims to support health professionals and enable remote rehabilitation, while maintaining health service characteristics and monitoring. MILORD is an end-to-end platform that consists of an interactive computer game, utilizing a leap motion sensor, a centralized user management system, an analysis platform that processes the data generated by the game, and an analysis dashboard presenting a set of meaningful features that describe upper limb movement. Our solution facilitates the monitoring of the patients' progress and provides an alternative way to analyze hand movement. The system was tested with normal subjects and patients and experts to record user's experience, receive feedback, identify any problems, and understand the system's value in monitoring and support motion defect and progress. This small-scale study indicated the capacity of the analysis to quantify the movement in a meaningful way and express the differences between normal and pathological movement, and the user experience was positive with both patients and normal subjects.

High-Dimensional Analysis of Finger Motion and Screening of Cervical Myelopathy With a Noncontact Sensor: Diagnostic Case-Control Study.

Cervical myelopathy (CM) causes several symptoms such as clumsiness of the hands and often requires surgery. Screening and early diagnosis of CM are important because some patients are unaware of their early symptoms and consult a surgeon only after their condition has become severe. The 10-second hand grip and release test is commonly used to check for the presence of CM. The test is simple but would be more useful for screening if it could objectively evaluate the changes in movement specific to CM. A previous study analyzed finger movements in the 10-second hand grip and release test using the Leap Motion, a noncontact sensor, and a system was developed that can diagnose CM with high sensitivity and specificity using machine learning. However, the previous study had limitations in that the system recorded few parameters and did not differentiate CM from other hand disorders. This study aims to develop a system that can diagnose CM with higher sensitivity and specificity, and distinguish CM from carpal tunnel syndrome (CTS), a common hand disorder. We then validated the system with a modified Leap Motion that can record the joints of each finger. In total, 31, 27, and 29 participants were recruited into the CM, CTS, and control groups, respectively. We developed a system using Leap Motion that recorded 229 parameters of finger movements while participants gripped and released their fingers as rapidly as possible. A support vector machine was used for machine learning to develop the binary classification model and calculated the sensitivity, specificity, and area under the curve (AUC). We developed two models, one to diagnose CM among the CM and control groups (CM/control model), and the other to diagnose CM among the CM and non-CM groups (CM/non-CM model). The CM/control model indexes were as follows: sensitivity 74.2%, specificity 89.7%, and AUC 0.82. The CM/non-CM model indexes were as follows: sensitivity 71%, specificity 72.87%, and AUC 0.74. We developed a screening system capable of diagnosing CM with higher sensitivity and specificity. This system can differentiate patients with CM from patients with CTS as well as healthy patients and has the potential to screen for CM in a variety of patients.

4D dynamic system for visual-motor integration analysis

It is very important to evaluate visual-motor integration (VMI), as it is used as an index to evaluate cognitive abilities. However, it is difficult to use the existing paper-based tests to evaluate the dynamic process, including spatial and depth perception abilities. Therefore, this study aims to extract kinematic and dynamic features for dynamic assessment for VMI. We propose a 4D dynamic analysis system that implements a VMI test in a virtual space using Leap motion controller and Unity3D and acquires the time-series data of hand joints and traces. We selected three categories of features: postural control ability, spatial and depth perception ability, and 4D analysis. The degree and patterns of postural maintenance differed between subjects in the VMI and MC tests. In addition, the personal patterns were identified with dynamic features, including their fluency and hesitation in relation to the task figures of the VMI test tool. As such, this system enables dynamic feature extraction and analysis which were previously impossible and presents performance results for healthy adults.

Interaction Design of Rural Brand IP Image Based on Processing Algorithmic Art

Processing-based programming interactive art has gradually become a research hotspot. To improve users' immersion and experience, this paper proposes a rural brand IP interactive design method based on a Processing algorithm. While solving the contradiction between the graphical perceptual thinking of rural brand IP and the logic of computer language architecture, it also interconnects with Arduino, Kinect, Leap Motion, and other devices. It further provides ideas for multi-sensing interactive rural brand IP design.

Interaction Design of Rural Brand IP Image Based on Processing Algorithmic Art

Processing-based programming interactive art has gradually become a research hotspot. To improve users' immersion and experience, this paper proposes a rural brand IP interactive design method based on a Processing algorithm. While solving the contradiction between the graphical perceptual thinking of rural brand IP and the logic of computer language architecture, it also interconnects with Arduino, Kinect, Leap Motion, and other devices. It further provides ideas for multi-sensing interactive rural brand IP design.

Efficacy of Leap Motion tracking device in Rehabilitation

One of the most common fractures seen in emergency rooms is distal radial fractures (DRF), which commonly occur when an upper extremity is broken. A FOOSH injury is one that occurs when the mechanism of injury involves a dorsal bending force being applied across the distal radial (fall onto an outstretched hand). The DRF has a bimodal distribution and is the most common upper limb fracture across all age groups, peaking in young men and postmenopausal women with an incidence ratio of 1:4. A randomised controlled experiment was conducted to evaluate and analyse the improvement in functional status and pain in patients who followed a home exercise programme without supervision and a physiotherapy plan under observation. A comprehensive analysis of Following some upper limb fractures, exercise helps patients function better and minimises disability. A randomised controlled trial comparing the effectiveness of physical therapy alone to physical therapy and occupational therapy in patients with distal radial fractures receiving conservative treatment A new experimental investigation that focuses on the efficacy of leap motion tracking equipment must be carried out with larger samples, more standardisation in the type of intervention, and treatment methodology.

Interactive 3D touch and gesture capable holographic light field display with automatic registration between user and content

AbstractThis paper presents an interactive three‐dimensional (3D) touch and gesture user interface that utilizes a light‐field display and a 3D sensor. In contrast to conventional gesture interfaces, in which usually the image space and gesture space are separated, a user can directly interact with the 3D image in this interface system. To achieve this trait, the 3D registration between the sensor and display coordinate systems is necessary. In this study, we propose a method to automatically match the position of the reconstructed 3D image with the position of the hand of the user based on scattered light detection. The proposed calibration process is automatic and enables self‐calibration during practical operation without laborious measurement work. The method allows us to use the gesture detection capabilities of the leap motion controller to develop an interface where content and gesture positions match. We show some examples of 3D touch and gesture interactions in which the gesture position and the generated content match on a light field display.