Hand Tracking Research Articles

融合红外深度信息的视觉交互手部跟踪算法

融合红外深度信息的视觉交互手部跟踪算法

The Development of Motion Capture System Based on Kinect Sensor and Bluetooth-Gloves

The article describes a motion capture system based on the Kinect sensor and an original hand motion reading device – Bluetooth Gloves. It is used for operator's hand tracking including finger bending. A gyroscope assisted by an accelerometer measures the slope angles in relation to the axis of the sensor system while rotary potentiometers define the finger bending angles. The resulting trajectory data is translated into AR-600E robot movements. Data transmission is performed by means of Bluetooth interface. Software that reads telemetry from the Kinect Sensor and Bluetooth Gloves is developed using C++ and LabVIEW programming languages. Graphic User Interface and System architecture are given below.

Sphere-meshes for real-time hand modeling and tracking

Modern systems for real-time hand tracking rely on a combination of discriminative and generative approaches to robustly recover hand poses. Generative approaches require the specification of a geometric model. In this paper, we propose a the use of sphere-meshes as a novel geometric representation for real-time generative hand tracking. How tightly this model fits a specific user heavily affects tracking precision. We derive an optimization to non-rigidly deform a template model to fit the user data in a number of poses. This optimization jointly captures the user's static and dynamic hand geometry, thus facilitating high-precision registration. At the same time, the limited number of primitives in the tracking template allows us to retain excellent computational performance. We confirm this by embedding our models in an open source real-time registration algorithm to obtain a tracker steadily running at 60Hz. We demonstrate the effectiveness of our solution by qualitatively and quantitatively evaluating tracking precision on a variety of complex motions. We show that the improved tracking accuracy at high frame-rate enables stable tracking of extended and complex motion sequences without the need for per-frame re-initialization. To enable further research in the area of high-precision hand tracking, we publicly release source code and evaluation datasets.

Dynamic Balanced Reach: A Temporal and Spectral Analysis Across Increasing Performance Demands.

Standing balanced reach is a fundamental task involved in many activities of daily living that has not been well analyzed quantitatively to assess and characterize the multisegmental nature of the body's movements. We developed a dynamic balanced reach test (BRT) to analyze performance in this activity; in which a standing subject is required to maintain balance while reaching and pointing to a target disk moving across a large projection screen according to a sum-of-sines function. This tracking and balance task is made progressively more difficult by increasing the disk's overall excursion amplitude. Using kinematic and ground reaction force data from 32 young healthy subjects, we investigated how the motions of the tracking finger and whole-body center of mass (CoM) varied in response to the motion of the disk across five overall disk excursion amplitudes. Group representative performance statistics for the cohort revealed a monotonically increasing root mean squared (RMS) tracking error (RMSE) and RMS deviation (RMSD) between whole-body CoM (projected onto the ground plane) and the center of the base of support (BoS) with increasing amplitude (p < 0.03). Tracking and CoM response delays remained constant, however, at 0.5 s and 1.0 s, respectively. We also performed detailed spectral analyses of group-representative response data for each of the five overall excursion amplitudes. We derived empirical and analytical transfer functions between the motion of the disk and that of the tracking finger and CoM, computed tracking and CoM responses to a step input, and RMSE and RMSD as functions of disk frequency. We found that for frequencies less than 1.0 Hz, RMSE generally decreased, while RMSE normalized to disk motion amplitude generally increased. RMSD, on the other hand, decreased monotonically. These findings quantitatively characterize the amplitude- and frequency-dependent nature of young healthy tracking and balance in this task. The BRT is not subject to floor or ceiling effects, overcoming an important deficiency associated with most research and clinical instruments used to assess balance. This makes a comprehensive quantification of young healthy balance performance possible. The results of such analyses could be used in work space design and in fall-prevention instructional materials, for both the home and work place. Young healthy performance represents "exemplar" performance and can also be used as a reference against which to compare the performance of aging and other clinical populations at risk for falling.

Hand tracking with physiological constraints

Articulated hand tracking systems have been commonly used in virtual reality applications, including systems with human–computer interaction or interaction with game consoles; hand pose estimation has various other applications comprising sign language recognition and animation synthesis. The advanced technological achievements in motion capture over the last decade allow data acquisition with high accuracy and low cost. However, due to the high complexity of the human hand, it is still challenging to animate a hand model able to deal in details with the anatomical and physiological constraints of the hand. In this paper, we present a simple and efficient methodology for tracking and reconstructing 3D hand poses. Using an optical motion capture system, where markers are positioned at strategic points, we manage to acquire the movement of the hand and establish its orientation using a minimum number of markers. An Inverse Kinematics solver was then employed to control the postures of the hand, subject to physiological constraints that restrict the allowed movements to a feasible and natural set. The proposed methodology produces smooth and biomechanically correct movements, while the required processing time remains low, enabling an effective real-time hand motion tracking and reconstruction system.

Project anywhere: An interface for virtual architecture

Virtual and augmented realities open a new world of great potential for spatial research and experimentation by allowing new forms of unbuilt sensible architectural space. This article starts with a sketch of the current context in virtual reality and continues by outlining the development and structure of the research ‘project Anywhere’. The project is an easily deployable, wireless, multi-user, augmented reality app system that offers full body immersion through body, head and hands tracking. It can host multiple concurrent users, able to move freely in the virtual space, by moving in the real and also perform actions through a gesture interface to affect their shared environment. In conclusion, we describe the inherent properties of such a space, which we propose as a novel spatio-temporal medium for architecture that suggests an enriched notion of space for exploration and experimentation, through an example of a potential application.

Enhanced Sign Language Transcription System via Hand Tracking and Pose Estimation

In this study, we propose a new system for constructing parallel corpora for sign languages, which are generally under-resourced in comparison to spoken languages. In order to achieve scalability and accessibility regarding data collection and corpus construction, our system utilizes deep learning-based techniques and predicts depth information to perform pose estimation on hand information obtainable from video recordings by a single RGB camera. These estimated poses are then transcribed into expressions in SignWriting. We evaluate the accuracy of hand tracking and hand pose estimation modules of our system quantitatively, using the American Sign Language Image Dataset and the American Sign Language Lexicon Video Dataset. The evaluation results show that our transcription system has a high potential to be successfully employed in constructing a sizable sign language corpus using various types of video resources.

A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment

This study presents a 3D virtual reality (VR) keyboard system with realistic haptic feedback. The system uses two five-fingered data gloves to track finger positions and postures, uses micro-speakers to create simulated vibrations, and uses a head-mounted display (HMD) for 3D display. When users press a virtual key in the VR environment, the system can provide realistic simulated key click haptic feedback to users. The results of this study show that the advantages of the haptic VR keyboard are that users can use it when wearing HMDs (users do not need to remove HMDs to use the VR keyboard), the haptic VR keyboard can pop-up display at any location in the VR environments (users do not need to go to a specific location to use an actual physical keyboard), and the haptic VR keyboard can be used to provide realistic key click haptic feedback (which other studies have shown enhances user performance). The results also show that the haptic VR keyboard system can be used to create complex vibrations that simulate measured vibrations from a real keyboard and enhance keyboard interaction in a fully immersive VR environment.

Frequency response experiments of eye-vergence visual servoing in lateral motion with 3D evolutionary pose tracking

Visual servoing towards moving target with hand-eye cameras fixed at hand is inevitably affected by hand dynamical oscillations. Therefore, it is difficult to make target position keep always at the center of camera's view, as nonlinear dynamical effects of whole manipulator stand against tracking ability. To overcome this defect of the hand-eye fixed camera system, an eye-vergence system has been put forward, where the cameras could rotate to observe the target object. The visual servoing controllers of hand and eye-vergence are installed independently, so that it can observe the target object at the center of camera images through eye-vergence function. The dynamical superiorities of eye-vergence system are verified through frequency response experiments, comparing with hand tracking performances and the proposed eye-vergence tracking performances. This paper analyzes the performance of 3D-object position and orientation tracking, in which orientation representation method is based on quaternion, and the orientation tracking results are shown with more comprehensive analysis of system performance.

Skin color model adaptation under varying lighting conditions

Skin region detection is crucial for face recognition, hand tracking, and motion detection. In the detection process, a skin color model is usually required to confine the distribution of skin colors. However, skin color models are sensitive to lighting conditions. Skin segmentation under varying lighting conditions produces poor results. This article presents a skin detection procedure for human–computer interaction sessions under varying lighting conditions. The proposed method requests a skin sample from the user to estimate the color temperature of the light source. Then, the color temperature is used to correct the skin sample. At the subsequent step, the mean of the corrected skin sample is utilized to adapt the skin color model. Finally, the adapted skin color model is employed to segment skin regions in the video stream. Tests using the proposed method and some adaptive skin detection algorithms have been conducted. Statistical data show that the proposed method is superior to color constancy methods and the Gaussian mixture model in skin region segmentation. The proposed method improves the true positive rate by more than 13% in segmenting skin regions of a database. Its true positive rate is 20% better if real-life images are used as test data.

Fingers and Gesture Recognition with Kinect v2 Sensor

Abstract The paper presents enhancements and innovative solutions of the proposed in [3] algorithms for fingers tracking and hand gesture recognition based on new defined features describing hand gestures and exploiting new-tracked tip and thumb joints from Kinect v2 sensor. Dynamic Time Warping (DTW) algorithm is used for gestures recognition. We increased its accuracy, scale and rotational invariance by defining new 3D featuring angles describing gestures and used for training a gesture database. 3D positions for fingertips are extracted from depth sensor data and used for calculation of featuring angles between vectors. The provided by Kinect v2 3D positions for thumb, tip and hand joints also participates during the phases of recognition. A comparison with the latest published approach for finger tracking has been performed. The feasibility of the algorithms have been proven by real experiments.

Hand Tracking Based Projection Mapping System and Applications

Hand Tracking Based Projection Mapping System and Applications

3D character recognition using binocular camera for medical assist

Unmarked tracking of hands and fingers is a promising enabler for human–computer interaction. It can also be applied to medical monitoring and handicap aids. This will give much help to the deaf-mutes who have trouble in communicating with others or interact with the computers. In this paper, we present a 3D hand tracking and character recognition algorithm using binocular stereo camera, which can get the crucial depth information of the target. To precisely understand the characters that the user writes in the air, we define a valid gesture indicating that the user is inputting useful information and a null gesture indicating that corresponding movements are not necessary in the characters. Our approach can adapt to angle variation and make the writing freer. We show the benefits of our method by evaluating on several videos.

Serious Gaming in Augmented Reality using HMDs for Assessment of Upper Extremity Motor Dysfunctions

Abstract For a better understanding of how different disorders affect motor function, a uniform, standardized and objective evaluation is a desirable goal for the clinical community. We explore the potential of Augmented Reality (AR) combined with serious gaming and free hand tracking to facilitate objective, cost-effective and patient-friendly methods for evaluation of upper extremity motor dysfunction in different patient groups. In this paper, we describe the design process of the game and the system architecture of the AR framework to meet these requirements. Furthermore, we report our findings from two pilot studies we conducted with healthy people aged over 50. First, we present a usability study (n = 5) on three different modalities of visual feedback for natural hand interaction with AR objects (i. e., no augmented hand, partial augmented hand and a full augmented hand model). The results show that a virtual representation of the fingertips or hand improves the usability of natural hand interaction. Secondly, a study about game engagement is presented. The results of this experiment (n = 8) show that there might be potential for engagement, but usability needs to be improved before it can emerge.

Continuous Prediction of Finger Movements Using Force Myography

The tracking and prediction of hand and finger movements along with gesture recognition is an active subject of research due to its vast applications in many fields, such as prosthetic control and robotic telemanipulation of rehabilitation and assistive devices. The common challenge in all of these fields is developing a control interface for operating an automated assistive device in a more precise and human-like manner. In this paper, we aim to study the possibility of using a force-sensing resistors (FSRs) strap for the continuous prediction of finger movements, specifically those of the thumb and middle and index fingers. An array of 8 FSRs is used to construct a wearable band that wraps around the forearm. Ten healthy volunteers participated in this study. The subjects were asked to perform 3 representative grasping motions. The relative displacements of the tips of the thumb and middle and index fingers with respect to a reference marker placed on the hand (slightly above the wrist) was captured using a camera and colored marker system. An epsilon-based function of support vector regression with a radial basis function kernel is employed to train a regression model using the force myography (FMG) signal acquired from the forearm with the displacement of each fingertip. The results demonstrate the feasibility of using an FSR band for continuously predicting the displacement of the fingertips with an average squared correlation coefficient of 0.96 for the thumb and middle and index fingers, spanning three types of grasp.

An adaptive local binary pattern for 3D hand tracking

Ever since the availability of real-time three-dimensional (3D) data acquisition sensors such as time-of-flight and Kinect depth sensor, the performance of gesture recognition can be largely enhanced. However, since conventional two-dimensional (2D) image based feature extraction methods such as local binary pattern (LBP) generally use texture information, they cannot be applied to depth or range image which does not contain texture information. In this paper, we propose an adaptive local binary pattern (ALBP) for effective depth images based applications. Contrasting to the conventional LBP which is only rotation invariant, the proposed ALBP is invariant to both rotation and the depth distance in range images. Using ALBP, we can extract object features without using texture or color information. We further apply the proposed ALBP for hand tracking using depth images to show its effectiveness and its usefulness. Our experimental results validate the proposal.

Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences

Fully articulated hand tracking promises to enable fundamentally new interactions with virtual and augmented worlds, but the limited accuracy and efficiency of current systems has prevented widespread adoption. Today's dominant paradigm uses machine learning for initialization and recovery followed by iterative model-fitting optimization to achieve a detailed pose fit. We follow this paradigm, but make several changes to the model-fitting, namely using: (1) a more discriminative objective function; (2) a smooth-surface model that provides gradients for non-linear optimization; and (3) joint optimization over both the model pose and the correspondences between observed data points and the model surface. While each of these changes may actually increase the cost per fitting iteration, we find a compensating decrease in the number of iterations. Further, the wide basin of convergence means that fewer starting points are needed for successful model fitting. Our system runs in real-time on CPU only, which frees up the commonly over-burdened GPU for experience designers. The hand tracker is efficient enough to run on low-power devices such as tablets. We can track up to several meters from the camera to provide a large working volume for interaction, even using the noisy data from current-generation depth cameras. Quantitative assessments on standard datasets show that the new approach exceeds the state of the art in accuracy. Qualitative results take the form of live recordings of a range of interactive experiences enabled by this new approach.

Movement Epenthesis Detection for Continuous Sign Language Recognition

AbstractAutomatic sign language recognition (SLR) is a current area of research as this is meant to serve as a substitute for sign language interpreters. In this paper, we present the design of a continuous SLR system that can extract out the meaningful signs and consequently recognize them. Here, we have used height of the hand trajectory as a salient feature for separating out the meaningful signs from the movement epenthesis patterns. Further, we have incorporated a unique set of spatial and temporal features for efficient recognition of the signs encapsulated within the continuous sequence. The implementation of an efficient hand segmentation and hand tracking technique makes our system robust to complex background as well as background with multiple signers. Experiments have established that our proposed system can identify signs from a continuous sign stream with a 92.8% spotting rate.

3D CAPTURING PERFORMANCES OF LOW-COST RANGE SENSORS FOR MASS-MARKET APPLICATIONS

Abstract. Since the advent of the first Kinect as motion controller device for the Microsoft XBOX platform (November 2010), several similar active and low-cost range sensing devices have been introduced on the mass-market for several purposes, including gesture based interfaces, 3D multimedia interaction, robot navigation, finger tracking, 3D body scanning for garment design and proximity sensors for automotive. However, given their capability to generate a real time stream of range images, these has been used in some projects also as general purpose range devices, with performances that for some applications might be satisfying. This paper shows the working principle of the various devices, analyzing them in terms of systematic errors and random errors for exploring the applicability of them in standard 3D capturing problems. Five actual devices have been tested featuring three different technologies: i) Kinect V1 by Microsoft, Structure Sensor by Occipital, and Xtion PRO by ASUS, all based on different implementations of the Primesense sensor; ii) F200 by Intel/Creative, implementing the Realsense pattern projection technology; Kinect V2 by Microsoft, equipped with the Canesta TOF Camera. A critical analysis of the results tries first of all to compare them, and secondarily to focus the range of applications for which such devices could actually work as a viable solution.

3D CAPTURING PERFORMANCES OF LOW-COST RANGE SENSORS FOR MASS-MARKET APPLICATIONS

Since the advent of the first Kinect as motion controller device for the Microsoft XBOX platform (November 2010), several similar active and low-cost range sensing devices have been introduced on the mass-market for several purposes, including gesture based interfaces, 3D multimedia interaction, robot navigation, finger tracking, 3D body scanning for garment design and proximity sensors for automotive. However, given their capability to generate a real time stream of range images, these has been used in some projects also as general purpose range devices, with performances that for some applications might be satisfying. This paper shows the working principle of the various devices, analyzing them in terms of systematic errors and random errors for exploring the applicability of them in standard 3D capturing problems. Five actual devices have been tested featuring three different technologies: i) Kinect V1 by Microsoft, Structure Sensor by Occipital, and Xtion PRO by ASUS, all based on different implementations of the Primesense sensor; ii) F200 by Intel/Creative, implementing the Realsense pattern projection technology; Kinect V2 by Microsoft, equipped with the Canesta TOF Camera. A critical analysis of the results tries first of all to compare them, and secondarily to focus the range of applications for which such devices could actually work as a viable solution.