Hand-held Object Research Articles

Adaptation to the absence of tactile and proprioceptive feedback in object handling

Dextrous manipulation necessitates efficient sensorimotor feedback mechanisms to orientate handheld objects, adjust for movement dynamics and respond to perturbations. In particular, cutaneous receptors and proprioception are paramount in the regulation of hand-object interactions (Johansson, 1991; Nowak et al., 2004). This study further examined the object handling skills of a person with massive peripheral deafferentation (consequent to sensory neuropathy) in order to improve knowledge of how people adapt/compensate for the loss of tactile/proprioceptive feedback. Object handling abilities of a deafferented woman (GL) were compared to eight age-matched control subjects using three experimental tasks: – discrete vertical movements; – functional grasp and place and; – static holding with perturbations (taps) applied to the top of the object—firstly by the person themselves and secondly by the experimenter. Tasks 1 and 3 were performed under full vision and blinded conditions. Grip force and object acceleration/orientation were recorded using a portable, instrumented object. When compared to control subjects, GL demonstrated a global increase in grip force with diminished temporal coupling to changes in object acceleration and limited ability to maintain orientation of the handheld object in the absence of visual feedback. All phases of the grasp and place task were of greater duration for GL and a qualitatively distinct acceleration profile was observed. In the hold perturbation tasks, GL exhibited greater time delays between perturbation and grip force response but proved efficient at gauging grip force response with respect to the magnitude of the perturbation. Whilst proprioceptive deficits limit temporal precision of grip force adjustments in object handling, visuomotor strategies and auditory feedback may assist in regulating object orientation during functional tasks and grip force scaling during perturbations. Future studies might investigate the feasibility of augmented visual/auditory feedback devices to optimise object handling for people with somatosensory deficits.

Intermittent coupling between grip force and load force during oscillations of a hand-held object.

Tightly coordinated grip force adaptations in response to changing load forces have been reported as continuous, stable, and proportional to the load force changes. Considering the existence of inherent sensorimotor feedback delays, current accounts of grip force-load force coupling invoke explicit predictive mechanisms in the form of internal models for feedforward control to account for anticipatory grip force modulations. However, recent findings suggest that the stability and regularity of grip force-load force coupling is less persistent than previously thought. Thus, the objective of the current study was to comprehensively quantify the time-varying characteristics of grip force-load force coupling. Investigations into the coupling's dynamics during continuous 30s bouts of load force oscillation revealed intermittent phases of coordination, as well as phases that varied in stability, rather than a persistent and continuously stable pattern of coordination. These findings have important implications for accounts of grip force-load force coupling and of anticipation in motor control, more broadly.

Leg Perception of Object Heaviness

ABSTRACTPerceived heaviness is a function of both the muscle activity used to wield an object and the resulting movement. Wielding reveals invariant properties of the effector-object system, such as rotational inertia. Recent research has proposed a psychophysiological mechanism for perceiving the heaviness of a handheld object through dynamic touch that captures how arm muscle activity and angular movement combined reveal this invariance (Waddell, Fine, Likens, Amazeen & Amazeen, 2016). The current study extends this hypothesis by investigating the dynamics of heaviness perception with the leg. Participants lifted objects of varying mass with knee extension lifts while reporting perceived heaviness. During each lift, the electromyogram (EMG) was recorded from the quadriceps, and peak angular acceleration was recorded about the knee. The resulting psychophysiological function revealed the hypothesized ratio of muscle activity to movement, similar to that found in Waddell et al. (2016). This suggests that the dynamics for heaviness perception in the leg is similar to that shown in the arm in previous work.

Effects of transcranial direct current stimulation on grip force control in patients with cerebellar degeneration

BackgroundThe control of grip forces when moving a hand held object is impaired in patients with cerebellar degeneration. We asked the question whether after-effects of anodal transcranial direct current stimulation (tDCS) applied to the lateral cerebellum or M1 improved grip force control in cerebellar patients.MethodsGrip force control while holding an object during cyclic arm movements was assessed in patients with pure cerebellar degeneration (n = 14, mean age 50.2 years ± SD 8.8 years) and age- and sex-matched control participants (n = 14, mean age 50.7 years ± SD 9.8 years). All subjects were tested before and after application of tDCS (2 mA, 22 min) in a within-subject design. Each subject received anodal tDCS applied to the cerebellum, anodal tDCS applied to M1 or sham-stimulation with a break of 1 week between the three experimental sessions.ResultsThere were no clear after-effects of tDCS on grip force control neither in control participants nor in cerebellar patients. Cerebellar patients showed typical impairments with higher grip forces, a higher variability of movements.ConclusionIn the present study, deficits in grip force control were neither improved by tDCS applied over the cerebellum nor M1 in cerebellar degeneration.

Grip force control and hand dexterity are impaired in individuals with diabetic peripheral neuropathy.

Diabetic peripheral neuropathy (DPN) affects the sensory function of the hands and, consequently, may negatively impact hand dexterity, maximum grip strength (GSMax), and hand grip force (GF) control during object manipulation. The aims of this study were to examine and compare the GF control during a simple holding task as well as GSMax and hand dexterity of individuals with DPN and healthy controls. Ten type 2 diabetic individuals diagnosed with DPN and ten age- and gender-matched healthy controls performed two traditional timed hand dexterity tests (i.e., nine-hole peg test and Jebsen-Taylor hand function test), a GSMax test, and a GF control test (i.e., hold a instrumented handle). The results indicated that individuals with DPN and controls produced similar GSMax. However, individuals with DPN took longer to perform the hand dexterity tests and set lower safety margin (exerted lower GF) than controls when holding the handle. The findings showed that mild to moderate DPN did not significantly affect maximum hand force generation, but does impair hand dexterity and hand GF control, which could impair the performance of daily living manipulation tasks and put them in risk of easily dropping handheld objects.

Smart-Pockets: Body-deictic gestures for fast access to personal data during ambient interactions

This work introduces Smart-Pockets, a new set of whole-body gesture recognition techniques that enables users to access their personal digital content efficiently for visualization on ambient displays. Smart-Pockets works by recognizing users' body-deictic gestures entailing access to their pockets, for which associations between specific pockets and personal digital content anchored to those pockets has been managed a priori. The pocket metaphor that we explore in this work enables links to digital content using physical personal containers (i.e., pockets) placed at convenient locations on the user's body, containers that have been specifically devised over decades of fashion design to store and carry people's personal belongings comfortably and conveniently. Consequently, Smart-Pockets gestures are fast, require absolutely no precision to perform effectively, and are robustly recognized in user-independent scenarios with absolutely no training required from the user of the ambient display. Also, the Smart-Pockets technique is flexible and easily extensible to other physical containers, such as bags and hand-held objects, which we demonstrate in the form of Smart-Containers. We evaluate the accuracy of several techniques for recognizing Smart-Pockets access gestures, for which we report +99% accuracy for user-independent classification and explicit segmentation. We discuss users' kinematic performance with Smart-Pockets and Smart-Containers and show that the average pocket access time of 2.2s is comparable to the average production time of touch gestures on smart mobile devices and is much smaller than the time required to produce other whole-body gestures. Beyond their practical implications for advancing knowledge in gesture-based interface design for ambient interactions, we believe that the contributions introduced by the Smart-Pockets concept will also foster new developments by pointing the community attention toward (i) more examination of the potential of a new class of whole-body gestures, i.e., body-deictics, (ii) more attention toward how users access their personal digital content on public displays, an important preliminary step before actual interaction, and (iii) inspiring work in the community to examine new and creative associations between users' physical personal objects and their digital content visualized on ambient displays. HighlightsSmart-Pockets is a new technique to access personal digital content on ambient displays.The pocket metaphor creates the link between physical objects and personal digital content.Smart-Pockets employ a new type of hybrid deictic and body-deictic gestures.Smart-Pockets actions are fast to perform (2.2s) and robustly recognized (+99%).The Smart-Pockets concept is flexible and extendable to Smart-Containers.

非対称振動を伴う物体の挙錘により生成される重量錯覚(「ハプティックインタラクション」特集)

非対称振動を伴う物体の挙錘により生成される重量錯覚(「ハプティックインタラクション」特集)

Precision Grip Control while Walking Down a Stair Step.

The aim of this study was to determine whether the internal model regulating grip force (GF)/load force (LF) coordination during a brisk load increase is preserved when the lower extremities produce a perturbation during a single step-down task. We observed the coordination of the vertical ground reaction force (vGRF), GF and LF while holding a handheld object during a single step-down task. The 3 forces (vGRF, GF and LF) decreased during the start of the task. While the subject was descending, LF and GF became dissociated from vGRF and increased in value, probably to anticipate the first foot contact. Coordination of LF and GF was maintained until the maximal vGRF (knee extension). LF peaked in the same time window as vGRF, whereas GF peaked about 70 ms later. This desynchronization, which was previously observed in direct load increase on a handheld object, was interpreted to be a predictive action to ensure the smooth management of the brisk increase in load induced by the lower extremities. Incidentally, in this group, kinematic and dynamic differences were observed between men and women, which may highlight a gender-specific strategy to perform the step-down task. In conclusion, these results suggest that the internal model of precision grip is able to integrate a brisk load change, whatever its origin, and regulate the forces to provide an ideal GF to dampen a brisk load increase and secure the object.

Electro-Mechanical Based Automated Whiteboard Wiper

Whiteboards although not new, form the basis for an effective learning in organisations and the educational sector of technology.The most common technique of erasing a whiteboard is by the use of a whiteboard wiper which is a hand held object manually controlled by the student or teacher. Although the type of material used in wiping the board has evolved, the manual labour is still a constant factor. Technologically, improvements have been achieved in this area where we have the Interactive Whiteboards, However, the cost of acquisition and installation is on the high side for most institutions. The need to save time, funds and effort in schools, offices and various other areas where boards are applied is exceedingly growing and through the use of an automated whiteboard wiper, this can be wholly achieved. In this paper, we design and construct a white board wiper that will completely wipe clean the board after being used with the aid of a push button. The Whiteboard Wiper is a purely electro-mechanical automated whiteboard erasing system, which incorporates a logic gate based control circuit and a mechanical system that will allow for adaptability to various sizes and is reasonably affordable.

Separating Timing, Movement Conditions and Individual Differences in the Analysis of Human Movement.

A central task in the analysis of human movement behavior is to determine systematic patterns and differences across experimental conditions, participants and repetitions. This is possible because human movement is highly regular, being constrained by invariance principles. Movement timing and movement path, in particular, are linked through scaling laws. Separating variations of movement timing from the spatial variations of movements is a well-known challenge that is addressed in current approaches only through forms of preprocessing that bias analysis. Here we propose a novel nonlinear mixed-effects model for analyzing temporally continuous signals that contain systematic effects in both timing and path. Identifiability issues of path relative to timing are overcome by using maximum likelihood estimation in which the most likely separation of space and time is chosen given the variation found in data. The model is applied to analyze experimental data of human arm movements in which participants move a hand-held object to a target location while avoiding an obstacle. The model is used to classify movement data according to participant. Comparison to alternative approaches establishes nonlinear mixed-effects models as viable alternatives to conventional analysis frameworks. The model is then combined with a novel factor-analysis model that estimates the low-dimensional subspace within which movements vary when the task demands vary. Our framework enables us to visualize different dimensions of movement variation and to test hypotheses about the effect of obstacle placement and height on the movement path. We demonstrate that the approach can be used to uncover new properties of human movement.

Control of grip force and vertical posture while holding an object and being perturbed.

We investigated motor control perspectives of coordinating maintenance of posture and application of grip force when holding an object and being perturbed. Ten subjects stood on the force platform holding an instrumented object in their dominant hand and were exposed to an external perturbation applied to their shoulders. Task demands were manipulated by positioning a slippery cap on top of the instrumented object. Grip force applied to the object, the object acceleration and the center of pressure (COP) were recorded and analyzed during the time intervals typical for the anticipatory (APA) and compensatory (CPA) components of postural control. Onsets of grip force were seen before the onsets of the COP displacement and initiation of movements of the handheld object during the APA phase of postural control, while the onsets of maximum grip force preceded the maximum COP displacement during the CPA phase. When the task demands increased byholding a handheld object with the slippery cap, subjects tended to generate grip force earlier and of a smaller magnitude; also, the COP displacement in the APA phase was smaller as compared to holding a handheld object only. The outcome provides a foundation for future studies of maintenance of vertical posture in people with impairments of balance and grip force control when holding an object and being perturbed.

Adaptive human-inspired compliant contact primitives to perform surface–surface contact under uncertainty

This paper focuses on devising a control policy that is inspired by human strategy to enable robots to perform surface–surface contact between a hand-held object (e.g. a box) and the environment (e.g. table). We assume the object’s shape is partially known and consider uncertainties in both the environment and the object grasp. Our analysis on ten untrained subjects indicates that during this task: (1) the subjects start decreasing the angular velocity before the complete alignment (in contrast to existing approaches), and (2) they do not control the contact force to remain at a fixed value. Our study is also consistent with the hypothesis that the subjects determine an over-estimate of the relative angle between the object in hand and the environment. Based on these observations we propose a novel control policy, called Surface-Surface Contact Primitive (SSCP), which can perform the surface–surface alignment task with only partial information about the hand-held object and the environment. Furthermore, SSCP only requires a rough estimate of the surface normal vector and does not rely on either the estimation of contact type (i.e. point or edge contact) or locations of contact points. We evaluate the performance of the proposed controller on a set of robot experiments using two seven-degree-of-freedom robots, one for imposing uncertainty to the environment, and the other to perform the experiment. We show the applicability of the controller on four objects with different geometries, its generalization to four different surface materials, and its robustness to uncertainty in environment and grasp as well as external perturbations.

PORTABLE CAMERA-BASED ASSISTIVE TEXT AND PRODUCT LABEL READING FROM HAND-HELD OBJECTS FOR BLIND PERSONS

We propose a camera-based assistive text reading framework to help blind persons read text labels and product packaging from hand-held objects in their daily lives. To isolate the object from cluttered backgrounds or other surrounding objects in the camera view, we first propose an efficient and effective motionbased method to define a region of interest (ROI) in the video by asking the user to shake the object. This method extracts moving object region by a mixture of- Gaussians based background subtraction method. In the extracted ROI, text localization and recognition are conducted to acquire text information. To automatically localize the text regions from the object ROI, we propose a novel text localization algorithm by learning gradient features of stroke orientations and distributions of edge pixels. Text characters in the localized text regions are recognized by off-the-shelf optical character recognition (OCR) software. The recognized text codes are output to blind users in speech. The proof-of-concept prototype is also evaluated on a dataset collected using 10 blind persons, to evaluate the effectiveness of the system's hardware. We explore user interface issues, and assess robustness of the algorithm in extracting and reading text nfrom different objects with complex backgrounds.

Modality-specific and hierarchical feature learning for RGB-D hand-held object recognition

Hand-held object recognition is an important research topic in image understanding and plays an essential role in human-machine interaction. With the easily available RGB-D devices, the depth information greatly promotes the performance of object segmentation and provides additional channel information. While how to extract a representative and discriminating feature from object region and efficiently take advantage of the depth information plays an important role in improving hand-held object recognition accuracy and eventual human-machine interaction experience. In this paper, we focus on a special but important area called RGB-D hand-held object recognition and propose a hierarchical feature learning framework for this task. First, our framework learns modality-specific features from RGB and depth images using CNN architectures with different network depth and learning strategies. Secondly a high-level feature learning network is implemented for a comprehensive feature representation. Different with previous works on feature learning and representation, the hierarchical learning method can sufficiently dig out the characteristics of different modal information and efficiently fuse them in a unified framework. The experimental results on HOD dataset illustrate the effectiveness of our proposed method.

Extending “out of the body” tactile phantom sensations to 2D and applying it to mobile interaction

Funneling and saltation are the two major perceptual illusion techniques for vibro-tactile feedback. They can be used to minimize the number of vibrators on the interaction device in contact with the user body and thereby build a less cumbersome and less expensive feedback device. Recently, these techniques have been found to elicit an "out of the body" experience, i.e., phantom tactile sensations indirectly felt from the handheld object (external to the body). This paper explores the practical applicability of this theoretical result to mobile tactile interaction. Two psychophysical experiments were first conducted to validate the effects of "out of the body" funneling and saltation on an actual handheld smart phone along (1) 1D and (2) when extended, for the first time, to 2D. A third experiment was run to evaluate user experience, applying phantom sensation based on vibro-tactile feedback, using funneling, to a real-world application. Experimental results have confirmed the same "out of the body" 1D illusory effects on an actual mobile device. In addition, a 2D modulated phantom sensation with a resolution of 5 × 3 on a 3.5-inch display space was achieved through saltation and funneling. Finally, we showed that the phantom sensation does significantly improve the user experience for a real-world application with a small additional cost of a few more vibrators, compared to a conventional single vibrator-based device.

Rapid Visuomotor Corrective Responses during Transport of Hand-Held Objects Incorporate Novel Object Dynamics.

Numerous studies have shown that people are adept at learning novel object dynamics, linking applied force and motion, when performing reaching movements with hand-held objects. Here we investigated whether the control of rapid corrective arm responses, elicited in response to visual perturbations, has access to such newly acquired knowledge of object dynamics. Participants first learned to make reaching movements while grasping an object subjected to complex load forces that depended on the distance and angle of the hand from the start position. During a subsequent test phase, we examined grip and load force coordination during corrective arm movements elicited (within ∼150 ms) in response to viewed sudden lateral shifts (1.5 cm) in target or object position. We hypothesized that, if knowledge of object dynamics is incorporated in the control of the corrective responses, grip force changes would anticipate the unusual load force changes associated with the corrective arm movements so as to support grasp stability. Indeed, we found that the participants generated grip force adjustments tightly coupled, both spatially and temporally, to the load force changes associated with the arm movement corrections. We submit that recently learned novel object dynamics are effectively integrated into sensorimotor control policies that support rapid visually driven arm corrective actions during transport of hand held objects. Significance statement: Previous studies have demonstrated that the motor system can learn, and make use of, internal models of object dynamics to generate feedforward motor commands. However, it is not known whether such internal models are incorporated into rapid, automatic arm movement corrections that compensate for errors that arise during movement. Here we demonstrate, for the first time, that internal models of novel object dynamics are integrated into rapid corrective arm movements made in response to visuomotor perturbations that, importantly, do not directly perturb the object.

Brain activity and the perception of self-agency while viewing a video of hand grasping: a functional near-infrared spectroscopy study.

Self-agency is the recognition of one's own movement and plays a vital role in purposeful, voluntary movement. A sense of self-agency can be elicited in individuals who view their own simple finger movements as they are projected onto a screen and aligned with their actual hand position. Here, we examined whether individuals perceived self-agency when they viewed a video of a hand grasping a wooden cylinder and whether the perception of self-agency correlated with simultaneous changes in oxy-hemoglobin in the parietal or the prefrontal cortical areas. All participants reported the perception of self-agency, which was correlated with oxy-hemoglobin increases in the right prefrontal area. We conclude that self-agency also relates to hand-held objects displayed on video.

Mapping kinematic functional abilities of the hand to three dimensional shapes for inclusive design

Loss of hand function can have adverse effects on an individual's ability to maintain independence. The ability to perform daily activities, such as food preparation and medication delivery, is dependent on the hand's ability to grasp and manipulate objects. Therefore, the goal of this research was to demonstrate that three dimensional (3D) modeling of hand function can be used to improve the accessibility of handheld objects for individuals with reduced functionality through informed design. Individual models of hand functionality were created for 43 participants and group models were developed for groups of individuals without (Healthy) and with reduced functionality due to arthritis (RFA) of the hand. Cylindrical models representative of auto-injectors of varying diameters were analyzed in 3D space relative to hand function. The individual model mappings showed the cylinder diameter with the highest mapped functional values varied depending on the type of functional weighting chosen: kinematic redundancy of fingertip pad positional placement, fingertip pad orientation, or finger force directionality. The group mappings showed that for a cylinder to be grasped in a power grasp by at least 75% of the Healthy or RFA groups, a diameter of 40mm was required. This research utilizes a new hand model to objectively compare design parameters across three different kinematic factors of hand function and across groups with different functional abilities. The ability to conduct these comparisons enables the creation of designs that are universal to all – including accommodation of individuals with limits in their functional abilities.

Moving a hand-held object: Reconstruction of referent coordinate and apparent stiffness trajectories

This study used the framework of the referent configuration hypothesis and slow changes in the external conditions during vertical oscillation of a hand-held object to infer the characteristics of hypothetical control variables. The study had two main objectives: (1) to show that hypothetical control variables, namely, referent coordinates and apparent stiffness of vertical hand position and grip force can be measured in an experiment; and (2) to establish relation(s) between these control variables that yield the classic grip-force–load-force coupling. Healthy subjects gripped a handle and performed vertical oscillations between visual targets at one of five metronome-prescribed frequencies. A HapticMaster robot was used to induce slow changes in the vertical force applied to the handle, while the size of the handle was changed slowly leading to changes in the grip aperture. The subjects were instructed not to react to possible changes in the external forces. A linear, second-order model was used to reconstruct the referent coordinate and apparent stiffness values for each phase of the vertical oscillation cycle using across-cycle regressions. The reconstructed time profiles of the referent coordinates and apparent stiffness showed consistent trends across subjects and movement frequencies. To validate the method, these values were used to predict the vertical force and the grip force applied to the handle for movement cycles that were not utilized in the reconstruction process. Analysis of the coupling between the four variables, two referent coordinates and two apparent stiffness values, revealed a single strong constraint reflecting the coupling between the grip force and vertical force. We view these data as providing experimental support for the idea of controlling natural, multi-muscle actions with shifts in a low-dimensional set of referent coordinates.

RGB-D Hand-Held Object Recognition Based on Heterogeneous Feature Fusion

Object recognition has many applications in human-machine interaction and multimedia retrieval. However, due to large intra-class variability and inter-class similarity, accurate recognition relying only on RGB data is still a big challenge. Recently, with the emergence of inexpensive RGB-D devices, this challenge can be better addressed by leveraging additional depth information. A very special yet important case of object recognition is hand-held object recognition, as manipulating objects with hands is common and intuitive in human-human and human-machine interactions. In this paper, we study this problem and introduce an effective framework to address it. This framework first detects and segments the hand-held object by exploiting skeleton information combined with depth information. In the object recognition stage, this work exploits heterogeneous features extracted from different modalities and fuses them to improve the recognition accuracy. In particular, we incorporate handcrafted and deep learned features and study several multi-step fusion variants. Experimental evaluations validate the effectiveness of the proposed method.