Hand Trajectory Research Articles

Virtual manipulations for draping

Purpose – Draping is one method used in clothing design. It is important to virtualize draping in real time, and virtual cloth handling is a key technology for this purpose. A mouse is often used for real-time cloth handling in many studies. However, gesture manipulation is more realistic than movements using the mouse. The purpose of this paper is to demonstrate virtual cloth manipulation using hand gestures in the real world. Design/methodology/approach – In this study, the authors demonstrate three types of manipulation: moving, cutting, and attaching. The user’s hand coordinates are obtained with a Kinect, and the cloth model is manipulated by them. The cloth model is moved based on the position of the hand coordinates. The cloth model is cut along a cut line calculated from the hand coordinates. In attaching the cloth model, it is mapped to a dummy model and then part of the cloth model is fixed and another part is released. Findings – This method can move the cloth model according to the motion of the hands. The authors have succeeded in cutting the cloth model based on the hand trajectory. The cloth model can be attached to the dummy model and its form is changed along the dummy model shape. Originality/value – Cloth handling in many studies is based on indirect manipulation using a mouse. In this study, the cloth model is manipulated according to hand motion in the real world in real time.

Toward an intuitive sign language animation authoring system for the deaf

This paper presents an ongoing effort toward an online collaborative framework allowing Deaf individuals to author intelligible signs using a dedicated 3D animation authoring interface. The results that are presented mainly focus on the design of a dedicated user interface assisted by novel input devices. This design cannot only benefit the Deaf but also the linguists studying sign language by providing them with a novel kind of study material. This material would consist of a symbolic representation of intelligible sign language animations together with a fine-grained log of the user's edit actions. Two user studies demonstrate how Leap Motion and Kinect-like devices can be used together for recording and authoring hand trajectories as well as facial animation.

Control model for dampening hand vibrations using information of internal and external coordinates.

In the present study, we investigate a control mechanism that dampens hand vibrations. Here, we propose a control method with two components to suppress hand vibrations. The first is a passive suppression method that lowers the joint stiffness to passively dampen the hand vibrations. The second is an active suppression method that adjusts an equilibrium point based on skyhook control to actively dampen the hand vibrations. In a simulation experiment, we applied these two methods to dampen hand vibrations during the shoulder’s horizontal oscillation. We also conducted a measurement experiment wherein a subject’s shoulder was sinusoidally oscillated by a platform that generated horizontal oscillations. The results of the measurement experiments showed that the jerk of each part of the arm in a task using a cup filled with water was smaller than the shoulder jerk and that in a task with a cup filled with stones was larger than the shoulder jerk. Moreover, the amplitude of the hand trajectory in both horizontal and vertical directions was smaller in a task using a cup filled with water than in a task using a cup filled with stones. The results of the measurement experiments were accurately reproduced by the active suppression method based on skyhook control. These results suggest that humans dampen hand vibrations by controlling the equilibrium point through the information of the external workspace and the internal body state rather than by lowering joint stiffness only by using internal information.

Automated Video Analysis of Handwashing Behavior as a Potential Marker of Cognitive Health in Older Adults.

The identification of different stages of cognitive impairment can allow older adults to receive timely care and plan for the level of caregiving. People with existing diagnosis of cognitive impairment go through episodic phases of dementia requiring different levels of care at different times. Monitoring the cognitive status of existing patients is, thus, critical to deciding the level of care required by older adults. In this paper, we present a system to assess the cognitive status of older adults by monitoring a common activity of daily living, namely handwashing. Specifically, we extract features from handwashing trials of participants diagnosed with different levels of dementia ranging from cognitively intact to severe cognitive impairment, as assessed by the mini-mental state exam (MMSE). Based on videos of handwashing trials, we extract two classes of features: one characterizing the occupancy of different sink regions by the participant, and the other capturing the path tortuosity of the motion trajectory of participant's hands. We perform correlation analysis to assess univariate capacity of individual features to predict MMSE scores. To assess multivariate performance, we use machine learning methods to train models that predict the cognitive status (aware, mild, moderate, severe), as well as the MMSE scores. We present results demonstrating that features derived from hand washing behavior can be potential surrogate markers of a person's dementia, which can be instrumental in developing automated tools for continuously monitoring the cognitive status of older adults.

1P1-H04 手先軌道を利用したインタフェースの開発

This paper describes an interface to operate the computer using hand trajectory. The skeleton information in humans of the Kinect sensor was used for detection of arm posture and hand positions. From the obtained information, the motion of the operator was recognized to reflect a mouse operation. In the mouse operations, the user could decide the position of the cursor with the right hand, and click with the left hand. In addition, we built the system which could choose two displays by changes in the depth position of the hand and the operability of the developed system was examined by the trial experiments.

RGBD Video Based Human Hand Trajectory Tracking and Gesture Recognition System

The task of human hand trajectory tracking and gesture trajectory recognition based on synchronized color and depth video is considered. Toward this end, in the facet of hand tracking, a joint observation model with the hand cues of skin saliency, motion and depth is integrated into particle filter in order to move particles to local peak in the likelihood. The proposed hand tracking method, namely, salient skin, motion, and depth based particle filter (SSMD-PF), is capable of improving the tracking accuracy considerably, in the context of the signer performing the gesture toward the camera device and in front of moving, cluttered backgrounds. In the facet of gesture recognition, a shape-order context descriptor on the basis of shape context is introduced, which can describe the gesture in spatiotemporal domain. The efficient shape-order context descriptor can reveal the shape relationship and embed gesture sequence order information into descriptor. Moreover, the shape-order context leads to a robust score for gesture invariant. Our approach is complemented with experimental results on the settings of the challenging hand-signed digits datasets and American sign language dataset, which corroborate the performance of the novel techniques.

Prediction in the Vestibular Control of Arm Movements.

The contribution of vestibular signals to motor control has been evidenced in postural, locomotor, and oculomotor studies. Here, we review studies showing that vestibular information also contributes to the control of arm movements during whole-body motion. The data reviewed suggest that vestibular information is used by the arm motor system to maintain the initial hand position or the planned hand trajectory unaltered during body motion. This requires integration of vestibular and cervical inputs to determine the trunk motion dynamics. These studies further suggest that the vestibular control of arm movement relies on rapid and efficient vestibulomotor transformations that cannot be considered automatic. We also reviewed evidence suggesting that the vestibular afferents can be used by the brain to predict and counteract body-rotation-induced torques (e.g., Coriolis) acting on the arm when reaching for a target while turning the trunk.

Brain Machine Interface for forces and motion based on musculo-skeletal model

Event Abstract Back to Event Brain Machine Interface for forces and motion based on musculo-skeletal model Yasuharu Koike1*, Hiroyuki Kambara1 and Natsue Yoshimura1 1 Tokyo Institute of Technology, Japan Non-invasive measurement method, such as EEG, fMRI or NIRS, has been used for Brain Computer Interface. EEG has nice temporal resolution, and it is used for BCI, such as amplitudes of different frequency bands; imagining movement of different parts of the body; slow cortical potentials and gamma band rhythms. Recently, Electrocorticography (ECoG) is an alternative approach to less invasive BMIs. Since ECoG records directly from neuronal activities on the cortical surface, ECoG has higher spatio- temporal resolution with better signal-to-noise ratio than scalp EEG. ECoG has also shown potential as a stable long-term recording method. Several studies using ECoG have already succeeded in the classification of movement direction, grasp type, and prediction of hand trajectory. Despite these successes, however, there still remains considerable work for the realization of ECoG- based prosthesis. The human neuromuscular system naturally modulates mechanical stiffness and vis- cosity to achieve proper interaction with the environment. Current rehabilitation robots can perform sophisticated operations including stiffness control. Our previous works suggested that the angle, torque, and stiffness of joints can be predicted from muscle activity. Therefore, decoding muscle activity is an important component for realizing BMI systems capable of controlling interaction force or stiffness. In this talk, we introduce a reconstruction of muscle activity time series using a simple mathematical musculoskeletal model (1,2). Then we introduce BMI using this musculoskeletal model from EEG, ECoG recordings (3-6). Reference (1) Hiroyuki Kambara, Duk Shin, and Yasuharu Koike. A computational model for optimal muscle activity considering muscle viscoelasticity in wrist movements. Journal of neurophysiology, 109:2145–60, 2013. (2) Duk Shin, Jaehyo Kim, and Yasuharu Koike. A myokinetic arm model for estimating joint torque and stiffness from EMG signals during maintained posture. Journal of neurophysiology, 101:387–401, 2009. (3) N Yoshimura, C S Dasalla, T Hanakawa, M A Sato, and Y Koike. Reconstruction of flexor and extensor muscle activities from electroencephalography cortical currents. Neuroimage, 59(2):1324–1337, 2012. (4) Duk Shin, Hidenori Watanabe, Hiroyuki Kambara, Atsushi Nambu, Tadashi Isa, Yukio Nishimura, and Yasuharu Koike. Prediction of Muscle Activities from Electrocorticograms in Primary Motor Cortex of Primates. PLoS ONE, 7, 2012. (5) Yasuhiko Nakanishi, Takufumi Yanagisawa, Duk Shin, Ryohei Fukuma, Chao Chen, Hiroyuki Kambara, Nat- sue Yoshimura, Masayuki Hirata, Toshiki Yoshimine, and Yasuharu Koike. Prediction of Three-Dimensional Arm Trajectories Based on ECoG Signals Recorded from Human Sensorimotor Cortex. PLoS ONE, 8, 2013. (6) Chao Chen, Duk Shin, Hidenori Watanabe, Yasuhiko Nakanishi, Hiroyuki Kambara, Natsue Yoshimura, Atsushi Nambu, Tadashi Isa, Yukio Nishimura, and Yasuharu Koike. Decoding grasp force profile from electrocorticography signals in non-human primate sensorimotor cortex. Neuroscience research, 83:1–7, 2014. Keywords: Brain machine interface (BMI), ECoG signals, fMRI, NIRS, Musculo-skeletal model Conference: 2015 International Workshop on Clinical Brain-Machine Interfaces (CBMI2015), Tokyo, Japan, 13 Mar - 15 Mar, 2015. Presentation Type: Oral presentation / lecture Topic: Clinical Brain-Machine Interfaces Citation: Koike Y, Kambara H and Yoshimura N (2015). Brain Machine Interface for forces and motion based on musculo-skeletal model. Conference Abstract: 2015 International Workshop on Clinical Brain-Machine Interfaces (CBMI2015). doi: 10.3389/conf.fnhum.2015.218.00013 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 23 Apr 2015; Published Online: 29 Apr 2015. * Correspondence: Dr. Yasuharu Koike, Tokyo Institute of Technology, Meguro, Tokyo, Japan, koike@pi.titech.ac.jp Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Yasuharu Koike Hiroyuki Kambara Natsue Yoshimura Google Yasuharu Koike Hiroyuki Kambara Natsue Yoshimura Google Scholar Yasuharu Koike Hiroyuki Kambara Natsue Yoshimura PubMed Yasuharu Koike Hiroyuki Kambara Natsue Yoshimura Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Extending the reach of mousetracking in numerical cognition: a comment on Fischer and Hartmann (2014).

Extending the reach of mousetracking in numerical cognition: a comment on Fischer and Hartmann (2014).

Stable and artifact-resistant decoding of 3D hand trajectories from ECoG signals using the generalized additive model

Objective. The key criterion for reliability of brain–computer interface (BCI) devices is their stability and robustness in natural environments in the presence of spurious signals and artifacts. Approach. To improve stability and robustness, a generalized additive model (GAM) is proposed for BCI decoder identification. Together with partial least squares (PLS), GAM can be applied to treat high-dimensional data and it is compatible with real-time applications. For evaluation of prediction quality, along with standard criteria such as Pearson correlation, root mean square error (RMSE), mean absolute error (MAE), additional criteria, mean absolute differential error (MADE) and dynamic time warping (DTW) distance, are chosen. These criteria reflect the smoothness and dissimilarity of the predicted and observed signals in the presence of phase desynchronization. Main results. The efficiency of the GAM–PLS model is tested on the publicly available database of simultaneous recordings of the continuous three-dimensional hand trajectories and epidural electrocorticogram signals of the Japanese macaque. GAM–PLS outperforms the generic PLS and improves the evaluation criteria: 22% (Pearson correlation), 8% (RMSE), 13% (MAE), 31% (MADE), 20% (DTW). Significance. Motor-related BCIs are systems to improve the quality of life of individuals with severe motor disabilities. The improvement of the reliability of the BCI decoder is an important step toward real-life applications of BCI technologies.

Eye-hand coordination during visuomotor adaptation with different rotation angles.

This study examined adaptive changes of eye-hand coordination during a visuomotor rotation task. Young adults made aiming movements to targets on a horizontal plane, while looking at the rotated feedback (cursor) of hand movements on a monitor. To vary the task difficulty, three rotation angles (30°, 75°, and 150°) were tested in three groups. All groups shortened hand movement time and trajectory length with practice. However, control strategies used were different among groups. The 30° group used proportionately more implicit adjustments of hand movements than other groups. The 75° group used more on-line feedback control, whereas the 150° group used explicit strategic adjustments. Regarding eye-hand coordination, timing of gaze shift to the target was gradually changed with practice from the late to early phase of hand movements in all groups, indicating an emerging gaze-anchoring behavior. Gaze locations prior to the gaze anchoring were also modified with practice from the cursor vicinity to an area between the starting position and the target. Reflecting various task difficulties, these changes occurred fastest in the 30° group, followed by the 75° group. The 150° group persisted in gazing at the cursor vicinity. These results suggest that the function of gaze control during visuomotor adaptation changes from a reactive control for exploring the relation between cursor and hand movements to a predictive control for guiding the hand to the task goal. That gaze-anchoring behavior emerged in all groups despite various control strategies indicates a generality of this adaptive pattern for eye-hand coordination in goal-directed actions.

Spatiotemporal representation of 3D hand trajectory based on beta-elliptic models

The aim of this paper was to model the hand trajectory during grasping by an extension in 3D of the 2D written language beta-elliptic model. The interest of this model is that it takes into account both geometric and velocity information. The method relies on the decomposition of the task space trajectories in elementary bricks. The latter is characterized by a velocity profile modelled with beta functions and a geometry modelled with elliptic shapes. A data base of grasping movements has been constructed and the errors of reconstruction were assessed (distance and curvature) considering two variations of the beta-elliptic model (‘quarter ellipse’ and ‘two tangents points’ method). The results showed that the method based on two tangent points outperforms the quarter ellipse method with average and maximum relative errors of 2.73% and 8.62%, respectively, and a maximum curvature error of 9.26% for the former. This modelling approach can find interesting application to characterize the improvement due to a rehabilitation or teaching process by a quantitative measurement of hand trajectory parameters.

Haptic proprioception in a virtual locomotor task.

Normal gait needs both proprioceptive and visual feedback to the nervous system to effectively control the rhythmicity of motor movement. Current preprogrammed exoskeletons provide only visual feedback with no user control over the foot trajectory. We propose an intuitive controller where hand trajectories are mapped to control contralateral foot movement. Our study shows that proprioceptive feedback provided to the users hand in addition to visual feedback result in better control during virtual ambulation than visual feedback alone. Hand trajectories resembled normal foot trajectories when both proprioceptive and visual feedback was present. Our study concludes that haptic feedback is essential for both temporal and spatial aspects of motor control in rhythmic movements.

Efficient Optimization for Autonomous Robotic Manipulation of Natural Objects

Manipulating natural objects of irregular shapes, such as rocks, is an essential capability of robots operating in outdoor environments. Physics-based simulators are commonly used to plan stable grasps for man-made objects. However, planning is an expensive process that is based on simulating hand and object trajectories in different configurations, and evaluating the outcome of each trajectory. This problem is particularly concerning when the objects are irregular or cluttered, because the space of feasible grasps is significantly smaller, and more configurations need to be evaluated before finding a good one. In this paper, we first present a learning technique for fast detection of an initial set of potentially stable grasps in a cluttered scene. The best detected grasps are further optimized by fine-tuning the configuration of the hand in simulation. To reduce the computational burden of this last operation, we model the outcomes of the grasps as a Gaussian Process, and use an entropy-search method in order to focus the optimization on regions where the best grasp is most likely to be. This approach is tested on the task of clearing piles of real, unknown, rock debris with an autonomous robot. Empirical results show a clear advantage of the proposed approach when the time window for decision is short.

Mapping the feel of the arm with the sight of the object: on the embodied origins of infant reaching.

For decades, the emergence and progression of infant reaching was assumed to be largely under the control of vision. More recently, however, the guiding role of vision in the emergence of reaching has been downplayed. Studies found that young infants can reach in the dark without seeing their hand and that corrections in infants' initial hand trajectories are not the result of visual guidance of the hand, but rather the product of poor movement speed calibration to the goal. As a result, it has been proposed that learning to reach is an embodied process requiring infants to explore proprioceptively different movement solutions, before they can accurately map their actions onto the intended goal. Such an account, however, could still assume a preponderant (or prospective) role of vision, where the movement is being monitored with the scope of approximating a future goal-location defined visually. At reach onset, it is unknown if infants map their action onto their vision, vision onto their action, or both. To examine how infants learn to map the feel of their hand with the sight of the object, we tracked the object-directed looking behavior (via eye-tracking) of three infants followed weekly over an 11-week period throughout the transition to reaching. We also examined where they contacted the object. We find that with some objects, infants do not learn to align their reach to where they look, but rather learn to align their look to where they reach. We propose that the emergence of reaching is the product of a deeply embodied process, in which infants first learn how to direct their movement in space using proprioceptive and haptic feedback from self-produced movement contingencies with the environment. As they do so, they learn to map visual attention onto these bodily centered experiences, not the reverse. We suggest that this early visuo-motor mapping is critical for the formation of visually-elicited, prospective movement control.

Decoding fingertip trajectory from electrocorticographic signals in humans

Seeking to apply brain–machine interface technology in neuroprosthetics, a number of methods for predicting trajectory of the elbow and wrist have been proposed and have shown remarkable results. Recently, the prediction of hand trajectory and classification of hand gestures or grasping types have attracted considerable attention. However, trajectory prediction for precise finger motion has remained a challenge. We proposed a method for the prediction of fingertip motions from electrocorticographic signals in human cortex. A patient performed extension/flexion tasks with three fingers. Average Pearson's correlation coefficients and normalized root-mean-square errors between decoded and actual trajectories were 0.83–0.90 and 0.24–0.48, respectively. To confirm generalizability to other users, we applied our method to the BCI Competition IV open data sets. Our method showed that the prediction accuracy of fingertip trajectory could be equivalent to that of other results in the competition.

A wavelet-based method for extracting intermittent discontinuities observed in human motor behavior

Human motor behavior often shows intermittent discontinuities even when people try to follow a continuously moving target. Although most previous studies revealed common characteristics of this “motor intermittency” using frequency analysis, this technique is not always appropriate because the nature of the intermittency is not stationary, i.e., the temporal intervals between the discontinuities may vary irregularly. In the present paper, we propose a novel method for extracting intermittent discontinuities using a continuous wavelet transform (CWT). This method is equivalent to the detection of peak of the jerk profile in principle, but it successfully and stably detects discontinuities using the amplitude and phase information of the complex wavelet transform. More specifically, the singularity point on the time-scale plane plays a key role in detecting the discontinuities. Another important feature is that the proposed method does not require parameter tuning because it is based on the nature of hand movement. In addition, this method does not contain any optimization process, which avoids explosive increase in computational cost for long time-series data. The performance of the proposed method was examined using an artificial trajectory composed of several primitive movements, and an actual hand trajectory in a continuous target-tracking task. The functional rationale of the proposed method is discussed.

A Machine Vision-Based Gestural Interface for People With Upper Extremity Physical Impairments

A machine vision-based gestural interface was developed to provide individuals with upper extremity physical impairments an alternative way to perform laboratory tasks that require physical manipulation of components. A color and depth based 3-D particle filter framework was constructed with unique descriptive features for face and hands representation. This framework was integrated into an interaction model utilizing spatial and motion information to deal efficiently with occlusions and its negative effects. More specifically, the suggested method proposed solves the false merging and false labeling problems characteristic in tracking through occlusion. The same feature encoding technique was subsequently used to detect, track and recognize users' hands. Experimental results demonstrated that the proposed approach was superior to other state-of-the-art tracking algorithms when interaction was present (97.52% accuracy). For gesture encoding, dynamic motion models were created employing the dynamic time warping method. The gestures were classified using a conditional density propagation-based trajectory recognition method. The hand trajectories were classified into different classes (commands) with a recognition accuracy of 95.9%. In addition, the new approach was validated with the “one shot learning” paradigm with comparable results to those reported in 2012. In a validation experiment, the gestures were used to control a mobile service robot and a robotic arm in a laboratory chemistry experiment. Effective control policies were selected to achieve optimal performance for the presented gestural control system through comparison of task completion time between different control modes.

Scene Particles: Unregularized Particle-Based Scene Flow Estimation

In this paper, an algorithm is presented for estimating scene flow, which is a richer, 3D analog of optical flow. The approach operates orders of magnitude faster than alternative techniques and is well suited to further performance gains through parallelized implementation. The algorithm employs multiple hypotheses to deal with motion ambiguities, rather than the traditional smoothness constraints, removing oversmoothing errors and providing significant performance improvements on benchmark data, over the previous state of the art. The approach is flexible and capable of operating with any combination of appearance and/or depth sensors, in any setup, simultaneously estimating the structure and motion if necessary. Additionally, the algorithm propagates information over time to resolve ambiguities, rather than performing an isolated estimation at each frame, as in contemporary approaches. Approaches to smoothing the motion field without sacrificing the benefits of multiple hypotheses are explored, and a probabilistic approach to occlusion estimation is demonstrated, leading to 10 and 15 percent improved performance, respectively. Finally, a data-driven tracking approach is described, and used to estimate the 3D trajectories of hands during sign language, without the need to model complex appearance variations at each viewpoint.

Two Types of Motor Strategy for Accurate Dart Throwing

This study investigated whether expert dart players utilize hand trajectory patterns that can compensate for the inherent variability in their release timing. In this study, we compared the timing error and hand trajectory patterns of expert players with those of novices. Eight experts and eight novices each made 60 dart throws, aiming at the bull’s-eye. The movements of the dart and index finger were captured using seven 480-Hz cameras. The data were interpolated using a cubic spline function and analyzed by the millisecond. The estimated vertical errors on the dartboard were calculated as a time-series by using the state variables of the index finger (position, velocity, and direction of motion). This time-series error represents the hand trajectory pattern. Two variables assessing the performance outcome in the vertical plane and two variables related to the timing control were quantified on the basis of the time-series error. The results revealed two typical types of motor strategies in the expert group. The timing error of some experts was similar to that of novices; however, these experts had a longer window of time in which to release an accurately thrown dart. These subjects selected hand trajectory patterns that could compensate for the timing error. Other experts did not select the complementary hand trajectories, but greatly reduced their error in release timing.