Leap Motion Controller Research Articles

Comparison of Conventional and Virtual Reality Box and Blocks Tests in Upper Limb Amputees: A Case-Control Study

Previous studies have demonstrated the potential of virtual reality as an effective teaching tool for motor training. Despite the growing interest in this field, only a few studies have determined the validity of the tasks made in virtual environments to real physical environments. This case-control study compares the score of pick and place activity in the real and virtual environment by using the box and blocks test setup on 4 able-bodied and 4 transradial amputees (2 myoelectric prosthetic users and 2 non-prosthetic users). This study integrates the traditional Box and Blocks Test mechanics into gameplay by using the Leap Motion controller and Oculus Rift headset. The participants were instructed to complete the test in both environments randomly for 10 sessions, with 30 minutes of training in each session. Pearson’s correlation interpretation was conducted to investigate the relation between the test’s score with the training duration, also the score obtained in the real and virtual environment. Independent samples t-test was also carried out to compare the score from the different test environments. All participants showed a greater percentage change of test score in the virtual version and better performance was achieved with increasing training duration. Both environments were positively correlated. However, there was a significant difference in the test score obtained in the real and virtual environment for able-bodied t(9) = 18.19, p < 0.05 and myoelectric prosthetic user t(9) = 4.51, p < 0.05, but not for the non-prosthetic user. This study has demonstrated that two different environments showed significantly comparable results in pick and place tasks by individuals with different abilities.

The Effect of Semi-Transparent and Interpenetrable Hands on Object Manipulation in Virtual Reality

Manipulating objects with the hand is a common task in Virtual Reality (VR). However, some issues can occur during these manipulations. Occlusion is one issue that happens when the virtual hand covers an object, and therefore the perception of that object is hindered. Semi-transparent hands could solve this problem. Another issue is the difficulty of delicate hand object manipulation. By using interpenetrable hands, turning off the physics of the hands, this difficulty might decrease. Still, there is a lack of research into how significant the impact is while using these methods. In this paper, with the designs of semi-transparent hands and interpenetrable hands, we present the results of our conducted user study focusing on the effect of semi-transparent and interpenetrable hands on hand object manipulation tasks in virtual reality. The user study includes a VR environment where participants are asked to perform tasks. These tasks are recorded in objective results of accuracy and speed. Afterwards, they fill out a questionnaire about their opinion on the difficulty while using the different methods in which the subjective findings are recorded. Additionally, we improved the semi-transparent hands by adding a feature that smoothly transitions the hand from opaque to semi-transparent. As an input device, we used the Leap Motion Controller (LMC) for this user study. However, any hand tracking sensor that tracks hands from the VR headset's point of view could be used. Semi-transparent and interpenetrable hands have shown significant improvement for precise manipulation, which was verified by user feedback from the questionnaire and the data from the tasks.

Development of flight simulation system based on leap motion controller

Abstract Aiming at the problems of low precision and discontinuous interaction in traditional human-computer interaction, A natural interactive system that can be used for flight simulation is proposed, which is based on Leap Motion combined with machine learning. Firstly, a gesture Motion suitable for flight simulation is introduced, and the Leap Motion is used to collect custom interactive gesture data. Then, linear regression and local weighted regression algorithm are respectively used to fit the collected original data, so as to obtain the interactive control curve that most conforms to the experimenter. The experimental results show that the control curve obtained by linear regression and local weighted regression algorithm have increased accuracy by 21.40% and 65.27% respectively compared with the data curve obtained by the original data, which is very helpful to improve the human-computer interaction experience in flight simulation.

RehabHand: Oriented-tasks serious games for upper limb rehabilitation by using Leap Motion Controller and target population in spinal cord injury

There is a growing interest in the use of technology in the field of neurorehabilitation in order to quantify and generate knowledge about sensorimotor disorders after neurological diseases, understanding that the technology has a high potential for its use as therapeutic tools. Taking into account that the rehabilitative process of motor disorders should extend beyond the inpatient condition, it's necessary to involve low-cost technology, in order to have technological solutions that can approach the outpatient period at home. to present the virtual applications-based RehabHand prototype for the rehabilitation of manipulative skills of the upper limbs in patients with neurological conditions and to determine the target population with respect to spinal cord injured patients. Seven virtual reality applications have been designed and developed with a therapeutic sense, manipulated by means of Leap Motion Controller. The target population was determined from a sample of 40 people, healthy and patients, analyzing hand movements and gestures. The hand movements and gestures were estimated with a fitting rate between the range 0.607-0.953, determining the target population by cervical levels and upper extremity motor score. Leap Motion is suitable for a determined sample of cervical patients with a rehabilitation purpose.

Retracted] Using Depth Cameras for Recognition and Segmentation of Hand Gestures

In recent years, in combination with technological advances, new paradigms of interaction with the user have emerged. This has motivated the industry to create increasingly powerful and accessible natural user interface devices. In particular, depth cameras have achieved high levels of user adoption. These devices include the Microsoft Kinect, the Intel RealSense, and the Leap Motion Controller. This type of device facilitates the acquisition of data in human activity recognition. Hand gestures can be static or dynamic, depending on whether they present movement in the image sequences. Hand gesture recognition enables human‐computer interaction (HCI) system developers to create more immersive, natural, and intuitive experiences and interactions. However, this task is not easy. That is why, in the academy, this problem has been addressed using machine learning techniques. The experiments carried out have shown very encouraging results indicating that the choice of this type of architecture allows obtaining an excellent efficiency of parameters and prediction times. It should be noted that the tests are carried out on a set of relevant data from the area. Based on this, the performance of this proposal is analysed about different scenarios such as lighting variation or camera movement, different types of gestures, and sensitivity or bias by people, among others. In this article, we will look at how infrared camera images can be used to segment, classify, and recognise one‐handed gestures in a variety of lighting conditions. A standard webcam was modified, and an infrared filter was added to the lens to create the infrared camera. The scene was illuminated by additional infrared LED structures, allowing it to be used in various lighting conditions.

A Study Protocol for Checking Efficacy of Leap Motion Device on Gross Hand Dexterity in Sub-Acute Stroke Patients

Introduction: Stroke survivors have limited everyday living tasks, often because of dexterous issues.Videogame-based training (VBT) with the appearance of virtual reality helps to improve the role of theupper limbs. The leap Motion controller can track both hands and fingers’ fine movements. The studiesthus demonstrates the efficacy of the leap motion system on gross hand dexterity in patients with sub-acutestroke.Method: The research has been designed as an experimental study. The total of 40 participants will be takenfrom AVBRH, sawangi Meghe for study as per inclusion and exclusion criteria. With intervention the spanof the study will be 6 months. Leap Motion-based, augmented reality training will be provided to patientsfor half hour, every single day, Five days of the week a month. Pre and post, box and block test and Systemusability scale will be taken. Those two will be the patient’s major outcome.Discussion:-This study protocol aims to assess the efficacy of the leap motion controller on the rehabilitationof gross hand dexterity in patients with stroke. The study’s expected outcome will concentrate on theevaluation of the usability of VBT using the Leap Motion Controller (LMC) to train gross hand dexterity instroke patients’ early recovery process.

American sign language recognition and training method with recurrent neural network

Though American sign language (ASL) has gained recognition from the American society, few ASL applications have been developed with educational purposes. Those designed with real-time sign recognition systems are also lacking. Leap motion controller facilitates the real-time and accurate recognition of ASL signs. It allows an opportunity for designing a learning application with a real-time sign recognition system that seeks to improve the effectiveness of ASL learning. The project proposes an ASL learning application prototype. The application would be a whack-a-mole game with a real-time sign recognition system embedded. Since both static and dynamic signs (J, Z) exist in ASL alphabets, Long-Short Term Memory Recurrent Neural Network with k-Nearest-Neighbour method is adopted as the classification method is based on handling of sequences of input. Characteristics such as sphere radius, angles between fingers and distance between finger positions are extracted as input for the classification model. The model is trained with 2600 samples, 100 samples taken for each alphabet. The experimental results revealed that the recognition rate for 26 ASL alphabets yields an average of 99.44% accuracy rate and 91.82% in 5-fold cross-validation with the use of leap motion controller.

A Haptic feedback system based on leap motion controller for prosthetic hand application

Leap Motion Controller (LMC) is a gesture sensor consists of three infrared light emitters and two infrared stereo cameras as tracking sensors. LMC translates hand movements into graphical data that are used in a variety of applications such as virtual/augmented reality and object movements control. In this work, we intend to control the movements of a prosthetic hand via (LMC) in which fingers are flexed or extended in response to hand movements. This will be carried out by passing in the data from the Leap Motion to a processing unit that processes the raw data by an open-source package (Processing i3) in order to control five servo motors using a micro-controller board. In addition, haptic setup is proposed using force sensors (FSR) and vibro-motors in which the speed of these motors is proportional to the amount of the grasp force exerted by the prosthetic hand. Investigation for optimal placement of the FSRs on a prosthetic hand to obtain convenient haptic feedback has been carried out. The results show the effect of object shape and weight on the obtained response of the FSR and how they influence the locations of the sensors.

Effect of Virtual Reality Training Using Leap Motion Controller on Impairments and Disability in Patients with Wrist and Hand Stiffness

Background and Purpose: Virtual reality (VR) training targets not only the musculoskeletal, but also theneurological system and has been successfully used in neurological rehabilitation. Our objective was to testwhether there are any benefits of adding VR training with conventional physiotherapy on impairments anddisability of wrist and hand stiffness, in patients with distal forearm fractures and early rheumatoid handaffection.Methodology: In this experimental study, 50 patients suffering from wrist and hand stiffness were alternatelyallocated into either Conventional physiotherapy group and VR with conventional physiotherapy group.Both the groups completed 8-12 training sessions over a period of 4 weeks comprising of similar protocol ofconventional physiotherapy with only the VR group receiving additional VR training. Patient was assessedon outcome measures pre and post treatment.Results: All the outcome measures showed significant improvement in both groups. On between groupcomparisons, the VR group showed significantly greater improvements on the outcome measures of gripstrength, dexterity (left hand and assembly), wrist flexion, ulnar deviation, forearm range of motion andwork component of Michigan Hand Questionnaire when compared to the conventional group. There wereno differences between groups on outcome measure of pain, wrist extension, wrist ulnar deviation and totaldisability score of Michigan Hand Questionnaire.Conclusion: We conclude that, adding virtual reality training to conventional physiotherapy has benefits onoutcome of grip strength, hand dexterity, work component of disability and direction specific improvementin the range of motion of wrist and forearm as compared to conventional physiotherapy alone.

Fingertip Gestures Recognition Using Leap Motion and Camera for Interaction with Virtual Environment

The emergence in computing and the latest hardware technologies realized the use of natural interaction with computers. Gesture-based interaction is one of the prominent fields of natural interactions. The recognition and application of hand gestures in virtual environments (VEs) need extensive calculations due to the complexities involved, which directly affect the performance and realism of interaction. In this paper, we propose a new interaction technique that uses single fingertip-based gestures for interaction with VEs. The objective of the study is to minimize the computational cost, increase performance, and improve usability. The interaction involves navigation, selection, translation, and release of objects. For this purpose, we propose a low-cost camera-based system that uses a colored fingertip for the fastest and accurate recognition of gestures. We also implemented the proposed interaction technique using the Leap Motion controller. We present a comparative analysis of the proposed system with the Leap Motion controller for gesture recognition and operation. A VE was developed for experimental purposes. Moreover, we conducted a comprehensive analysis of two different recognition setups including video camera and the Leap Motion sensor. The key parameters for analysis were task accuracy, interaction volume, update rate, and spatial distortion of accuracy. We used the Standard Usability Scale (SUS) for system usability analysis. The experiments revealed that camera implementation was found with good performance, less spatial distortion of accuracy, and large interaction volume as compared to the Leap Motion sensor. We also found the proposed interaction technique highly usable in terms of user satisfaction, user-friendliness, learning, and consistency.

Natural Human–Machine Interface With Gesture Tracking and Cartesian Platform for Contactless Electromagnetic Force Feedback

In this article, a novel human–machine interface, in which two Leap Motion (LM) controllers and a coil are attached to a Cartesian platform to provide contactless electromagnetic force feedback for enhancing the accuracy and efficiency of human–robot manipulation tasks is presented. To implement such an interface, an interval Kalman filter, an improved particle filter, and a mean filter are integrated to estimate accurately the position and orientation of the hand gesture tracked by the two LM controllers, and to smoothen the movement of the Cartesian platform. The back propagation neural network is employed to regulate the electric currents of the coil attached to the Cartesian platform for accurate force feedback. A series of comparative experiments are performed, and the results show that the presented interface greatly improved the efficiency and accuracy of human–robot manipulation tasks in comparison with existing methods, indicating its great potentials for many industry scenarios.

Arabic sign language recognition using Ada-Boosting based on a leap motion controller

According to the World Health Organization (WHO), 466 million people are suffering from hearing loss, i.e., 5% of the world population, of which 432 million (93%) are adults and 34 million (17%) children. The main problem is how deaf and hearing-impaired communicate with people and each other, how they get education or do their daily activities. Sign language is the main communication method for them. Building automatic hand gestures recognition system has many challenges specially in Arabic. Solving recognition problem and practically develop real-time recognition system is another challenge. Several types of research have been conducted on sign language recognition systems but for Arabic Sign Language are very limited. In this paper, an Arabic Sign Language (ArSL) recognition system that uses a Leap Motion Controller and Latte Panda is introduced. The recognition phase depends on two machine learning algorithms: (a) KNN (k-Nearest Neighbor) and (b) SVM (Support Vector Machine). Afterwards, an Ada-Boosting technique is applied to enhance the accuracy of both algorithms. A direct matching technique, DTW (Dynamic Time Wrapping), is applied and compared with AdaBoost. The proposed system is applied on 30 hand gestures which are composed of 20 single-hand gestures and 10 double-hand gestures. The experimental results show that the DTW achieved an accuracy of 88% for single-hand gestures and 86% for double-hand gestures. Overall, the proposed model’s recognition rate reached 92.3% for single-hand gestures and 93% for double-hand gestures after applying the Ada-Boosting. Finally, a prototype of our model was implemented in a single board (Latte Panda) to increase the system’s reliability and mobility.

Deep Learning based Dynamic Hand Gesture Recognition with Leap Motion Controller

Deep Learning based Dynamic Hand Gesture Recognition with Leap Motion Controller

Evaluation of leap motion controller effectiveness on 2D game environments using usability heuristics

The use of sensors for tasks such as (1) interface manipulation, (2) digital games and (3) simulations is quickly becoming commonplace. Novel digital interfaces are being established and, as a result, users are getting better and better at both learning and using them. Medicine, engineering, and entertainment are but a few of the fields making use of sensors for the purpose of scientific progress or to improve the user’s experience. However, the use of emerging sensor devices is significantly more prominent in 3D games and simulations. This article evaluates the effectiveness of the Leap Motion Controller in 2D gaming environments. In the study, other devices such as a mouse, keyboard, gamepad, and touch screen are used to further analyze the performance data gathered with the Leap Motion Controller. Data collection employs user experience heuristics and pre-determined key performance indicators. The indicators were used to measure the effectiveness of the controllers in four main tasks: interface handling, character movement, item collection and combat gameplay. Users showed more effective control in item collection and combat gameplay, according to the data gathered from our subject’s gameplay sessions. Item collection was performed better with the Leap Motion Controller and touch screen, based on the data collected. Observed results also showed that conventional interfaces outperformed the Leap Motion Controller in character movement.

Application of AR and VR in hand rehabilitation: A systematic review.

The human hand is the part of the body most frequently injured in work related accidents, accounting for a third of all accidents at work and often involving surgery and long periods of rehabilitation. Several applications of Augmented Reality (AR) and Virtual Reality (VR) have been used to improve the rehabilitation process. However, there is no sound evidence about the effectiveness of such applications nor the main drivers of therapeutic success. The objective of this study was to review the efficacy of AR and VR interventions for hand rehabilitation. A systematic search of publications was conducted in October 2019 in IEEE Xplore, Web of Science, Cochrane library, and PubMed databases. Search terms were: (1) video game or videogame, (2) hand, (3) rehabilitation or therapy and (4) VR or AR. Articles were included if (1) were written in English, (2) were about VR or AR applications, (3) were for hand rehabilitation, (4) the intervention had tests on at least ten patients with injuries or diseases which affected hand function and (5) the intervention had baseline or intergroup comparisons (AR or VR intervention group versus conventional physical therapy group). PRISMA protocol guidelines were followed to filter and assess the articles. From the eight selected works, six showed improvements in the intervention group, and two no statistical differences between groups. We were able to identify motivators of patients' adherence, namely real-time feedback to the patients, challenge, and increased individualized difficulty. Automated tracking, easy integration in the home setting and the recording of accurate metrics may increase the scalability and facilitate healthcare professionals' assessments. This systematic review provided advantages and drivers for the success of AR/VR application for hand rehabilitation. The available evidence suggests that patients can benefit from the use of AR or VR interventions for hand rehabilitation.

Restoration and Development of Urban Heritage Sites (Rehabilitation of Middle cities and Heritage towns)

There are various restoration methods and techniques that would help specialists whether planners, designers and archeologists to study historical sites and their valued heritage buildings in various cities at Egypt as Alexandria governorate which is considered as one of heritage towns. One of these techniques is Virtual Reality (VR) that could be used to improve the field of Virtual Heritage (VH) which is directed towards accurate representation of historic structures in urban heritage sites, objects or artefacts. Beside, using Geographical Information System (GIS) and Google earth model to define these urban heritage sites accurately with all database related to the specific area under study. Digital models of historical buildings and its urban spaces only give a sense of precision for their historical values. Yet, human attitude and cultural traditions remain a gap in current research and advanced technology in heritage visualization. So, Virtual Heritage Environments (VHE) suffers from the lack of ‘thematic interactivity’ due to the limited cultural heritage content and engaging modules largely used in photorealistic systems. So, this paper focuses in using various digital restoration techniques and methods to investigate and incorporate a cultural-feed into digital platforms of VH especially at Alexandria in Egypt as Qaitbay castle plaza. This site discusses a conceptual and practical workflow for the development of virtual heritage platforms as a research area, educational purpose. The study includes engagement tool of historic conservation in Alexandria, brings historic spaces and buildings to users in detailed studies whether researchers or students. The results of the research would be used for the renewal of the current plaza of Qaitbay castle by making it more attractive. Also, this plaza is considered as a tourist and historical area as well as the organization for paths of movement and transforming the work places for sale from street vendors to organized markets. The scenario was realized by integrating low-cost Virtual Reality (VR) techniques, as the Kolor Panotour Pro software, some tools as Oculus Rift DK2 and the Leap Motion controller. Further, the 3D virtual model was implemented by integrated steps through Virtual Tours and Informational Modeling (VT/IM) to improve real urban data of heritage sites. The paper describes the process for optimizing the 3D virtual model, the implementation of the interactive scenario and the results of a proposed 3D VT/IM of redesigning model for the Qaitbay Castle Plaza. Finally, the paper predicts future research directions for virtual reality, with a particular focus on interaction interfaces and explores the implications for the cultural heritage domain.

Perfectionistic Concerns are Detrimental to Skill Learning for Minimally Invasive Surgery

The pursuit of high standards and continuous self-improvement are important aspects of a professional attitude in medicine. However, when identity and self-esteem are dependent on flawless performance, healthy striving can become perfectionism. The present study examines the influence of perfectionism on the learning process of basic instrument handling for minimally invasive surgery (MIS). Thirty medical students volunteered to participate in the study (19 females, 11 males). The mean age was 19.8 years (SD = 1.8). The Perfectionism Inventory was used to determine the degree of perfectionist characteristics in two main factors: Self-evaluative perfectionism and Conscientious perfectionism. Participants practiced with the loops and wire task on a surgical simulator and were tested for skill retention within 48 hours. During practice instrument movement was captured in three-dimensional space using a Leap Motion controller. Performance was assessed by time and total path length travelled by the instruments. Self-evaluative perfectionism was negatively related to skill retention with regard to movement efficiency, but did not predict change in average time on task. The Conscientious perfectionism factor was not a predictor of skill retention with regard to path length or completion time. Self-evaluative perfectionism was a significant predictor of change in average pathlength between sessions but did not predict change in average time. The Conscientious perfectionism factor was not a predictor of changes in path length or completion time. Overly negative self-evaluation during MIS skill practice undermined the learning process. Trainees with self-critical perfectionistic tendencies may not optimally benefit from their efforts during practice.

In-air signature verification system using Leap Motion

Signature verification is a widely explored field due to its high acceptance and its compromise between security and comfort. Recently, different techniques have appeared to improve the capture, processing, and classification of signatures. In this work, authors present a novel and robust in-air signature verification system, which applies the use of Leap Motion controller to characterize in-air strokes, due to its stability and good performance for this task, as it will be demonstrated. Therefore, a database has been built for developing the experiments, which is composed of 100 users, with 10 genuine and 10 forgery samples per user. The implemented system is tested against two tests of impostor samples, zero effort attacks and active impostors. The second type of attacks are developed by different users, who showed very good abilities with the sensor. The classification is done by a Least Square Support Vector Machine. The equal error rate was 0.25% and 1.20%, respectively. The proposed system achieves very good results in comparison with the state-of-the-art one, which suggests that in-air signature processing gives an opportunity to increase systems’ security.

Interactive serious game for shoulder rehabilitation based on real-time hand tracking.

Virtual reality is becoming popular in the rehabilitation field thanks to the several advantages it can offer to patients and physicians. Indeed, serious games can: motivate and engage the patient; offer different levels of challenge and difficulty based on the patient baseline, and integrate objective measures of the patient's performance during each rehabilitation session. We designed and implemented a serious game for shoulder rehabilitation based on real-time hand tracking. The aim was to maintain the medical benefits of traditional rehabilitation, while reducing human resources and costs and facilitating active patient participation. Our software application provides the user with a shoulder horizontal adduction exercise. This exercise takes place in a 2D interactive game environment, controlled by hand movements on a desk pad. The hardware includes a standard desktop computer and screen, and the Leap Motion Controller: a hand tracking system. Changing the desk pad material allows the physiotherapist to vary the friction between the user hand and the supporting surface. Fourteen healthy volunteers and six rehabilitation experts tested our serious game. The results showed that the application is attractive, ergonomic and clinically useful. Despite promising results, clinical validation is necessary to demonstrate the efficacy of the serious game.

Effects of virtual reality associated with serious games for upper limb rehabilitation in patients with multiple sclerosis: randomized controlled trial

BackgroundDexterity and activities of daily living limitations on the upper limb (UL) represent one of the most common problems in patients with multiple sclerosis (MS). The aim of this study was to evaluate the effectiveness of the specially developed Serious Games that make use of the Leap Motion Controller (LMC) as main user interface for improving UL grip muscle strength, dexterity, fatigue, quality of life, satisfaction and compliance.MethodsA single-blinded randomized controlled trial was conducted. The sample was randomized into two groups: an experimental group who received treatment based on serious games designed by the research team using the developed LMC based Serious Games for the UL plus conventional rehabilitation, and a control group who received the same conventional rehabilitation for the UL. Both groups received two 60 min sessions per week over a ten-week period. Grip muscle strength, coordination, speed of movements, fine and gross UL dexterity, fatigue, quality of life, satisfaction and compliance were assessed in both groups pre-treatment, post-treatment and in a follow-up period of 1 month without receiving any treatment.ResultsIn the experimental group compared to the control group, significant improvements were observed in the post-treatment assessment for coordination, speed of movements, fine and gross UL dexterity. Also, significant results were found in the follow-up in coordination, speed of movements, fine and gross for the more affected side.ConclusionsAn experimental protocol using an LMC based Serious Games designed for UL rehabilitation showed improvements for unilateral gross manual dexterity, fine manual dexterity, and coordination in MS patients with high satisfaction and excellent compliance.Trial registrationThis randomized controlled trial has been registered at ClinicalTrials.gov Identifier: NCT04171908, Nov 2019.