Dynamic Manipulation Tasks Research Articles

Design and Evaluation of a Rapid Monolithic Manufacturing Technique for a Novel Vision-Based Tactile Sensor: C-Sight.

Tactile sensing has become indispensable for contact-rich dynamic robotic manipulation tasks. It provides robots with a better understanding of the physical environment, which is a vital supplement to robotic vision perception. Compared with other existing tactile sensors, vision-based tactile sensors (VBTSs) stand out for augmenting the tactile perception capabilities of robotic systems, owing to superior spatial resolution and cost-effectiveness. Despite their advantages, VBTS production faces challenges due to the lack of standardised manufacturing techniques and heavy reliance on manual labour. This limitation impedes scalability and widespread adoption. This paper introduces a rapid monolithic manufacturing technique and evaluates its performance quantitatively. We further develop and assess C-Sight, a novel VBTS sensor manufactured using this technique, focusing on its tactile reconstruction capabilities. Experimental results demonstrate that the monolithic manufacturing technique enhances VBTS production efficiency significantly. Also, the fabricated C-Sight sensor exhibits its reliable tactile perception and reconstruction capabilities, proofing the validity and feasibility of the monolithic manufacturing method.

Zero-Shot Sim-to-Real Transfer of Tactile Control Policies for Aggressive Swing-Up Manipulation

This letter aims to show that robots equipped with a vision-based tactile sensor can perform dynamic manipulation tasks without prior knowledge of all the physical attributes of the objects to be manipulated. For this purpose, a robotic system is presented that is able to swing up poles of different masses, radii and lengths, to an angle of 180 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> , while relying solely on the feedback provided by the tactile sensor. This is achieved by developing a novel simulator that accurately models the interaction of a pole with the soft sensor. A feedback policy that is conditioned on a sensory observation history, and which has no prior knowledge of the physical features of the pole, is then learned in the aforementioned simulation. When evaluated on the physical system, the policy is able to swing up a wide range of poles that differ significantly in their physical attributes without further adaptation. To the authors' knowledge, this is the first work where a feedback policy from high-dimensional tactile observations is used to control the swing-up manipulation of poles in closed-loop.

Grip and load force control and coordination in individuals with diabetes in different manipulation tasks.

The study aimed to investigate the control and coordination of grip force (normal component) and load force (tangential component) in three different manipulation tasks in individuals with diabetes with and with no diagnosis of diabetic peripheral neuropathy (DPN) and healthy controls. Twenty-four individuals with type 2 diabetes mellitus, 12 with no (nDPN) and 12 with DPN (wDPN), and 12 healthy controls performed three manipulation tasks (static holding, lifting and holding, and oscillation) with the dominant hand, using an instrumented handle. Relative safety margin (% of GF exerted above the minimum GF needed to hold the object) was measured in all tasks. Individuals with diabetes from the nDPN and wDPN groups set lower relative safety margin than controls only in the static holding task. No other group effect was revealed, except a lower coefficient of friction between skin and object surface in individuals with DPN. The coordination between grip and load force and grip force control was not affected by the diabetes during dynamic manipulation tasks (lifting and holding and oscillation). However, when individuals with diabetes without and with DPN performed a manipulation task in which the inflow of cutaneous information was small and stable (static holding), grip force control was affected by the disease. This finding indicates that individuals with type 2 diabetes mellitus not diagnosed with DPN, already show mild impairments in the nervous system that could affect grip force control and that could be one of the first signs of neuropathy caused by the diabetes.

Learning compositional models of robot skills for task and motion planning

The objective of this work is to augment the basic abilities of a robot by learning to use sensorimotor primitives to solve complex long-horizon manipulation problems. This requires flexible generative planning that can combine primitive abilities in novel combinations and, thus, generalize across a wide variety of problems. In order to plan with primitive actions, we must have models of the actions: under what circumstances will executing this primitive successfully achieve some particular effect in the world? We use, and develop novel improvements to, state-of-the-art methods for active learning and sampling. We use Gaussian process methods for learning the constraints on skill effectiveness from small numbers of expensive-to-collect training examples. In addition, we develop efficient adaptive sampling methods for generating a comprehensive and diverse sequence of continuous candidate control parameter values (such as pouring waypoints for a cup) during planning. These values become end-effector goals for traditional motion planners that then solve for a full robot motion that performs the skill. By using learning and planning methods in conjunction, we take advantage of the strengths of each and plan for a wide variety of complex dynamic manipulation tasks. We demonstrate our approach in an integrated system, combining traditional robotics primitives with our newly learned models using an efficient robot task and motion planner. We evaluate our approach both in simulation and in the real world through measuring the quality of the selected primitive actions. Finally, we apply our integrated system to a variety of long-horizon simulated and real-world manipulation problems.

A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulation

In this letter, we propose a whole-body planning framework that unifies dynamic locomotion and manipulation tasks by formulating a single multi-contact optimal control problem. We model the hybrid nature of a generic multi-limbed mobile manipulator as a switched system, and introduce a set of constraints that can encode any pre-defined gait sequence or manipulation schedule in the formulation. Since the system is designed to actively manipulate its environment, the equations of motion are composed by augmenting the robot's centroidal dynamics with the manipulated-object dynamics. This allows us to describe any high-level task in the same cost/constraint function. The resulting planning framework could be solved on the robot's onboard computer in real-time within a model predictive control scheme. This is demonstrated in a set of real hardware experiments done in free-motion, such as base or end-effector pose tracking, and while pushing/pulling a heavy resistive door. Robustness against model mismatches and external disturbances is also verified during these test cases.

Discrimination Improvement Through Undesirable Feedback in Coupling Object Manipulation Tasks.

Subjective effort can significantly affect the ability of humans to act optimally in dynamic manipulation tasks. In a previous study, we designed a complex object coupling manipulation task that required tight performance and induced high cognitive workload. We hypothesize that strong-effort-related physiological reactivity during the dynamic manipulation task improves the user performance in an undesired task feedback situation. To test this hypothesis, using the motor intentions' discrimination from electroencephalogram (EEG) measurements, we evaluate the effort expended by 20 participants in a controlling task with constraints involving complex coupling objects. Specifically, the finer motor decisions are obtained from the controlling information in EEG by using two fingers from the same hand rather than two hands. The motor intention is decoded from a task-dependent EEG through a regularized discriminant analysis, and the area under the curve is [Formula: see text]. Furthermore, we compare the undesired and desired task feedback conditions along with the individual's effort dynamic adjustment, and investigate whether the undesired task feedback improved the discrimination of the motor activities. A stronger effort to attain the desired feedback state corresponds to improved motor activity discrimination from the EEG in the undesired task feedback scenario. The differences in the brain activities under the undesired and desired task feedback conditions are analyzed using brain-network-based topographical scalp maps. Our experiment provides preliminary evidence that inducing strong effort can improve discrimination performance during highly demanding tasks. This finding can advance our understanding of human attention, potentially improve the accuracy of intention recognition, and may inspire better EEG acquisition contexts.

Iterative residual tuning for system identification and sim-to-real robot learning

Robots are increasingly learning complex skills in simulation, increasing the need for realistic simulation environments. Existing techniques for approximating real-world physics with a simulation require extensive observation data and/or thousands of simulation samples. This paper presents iterative residual tuning (IRT), a deep learning system identification technique that modifies a simulator’s parameters to better match reality using minimal real-world observations. IRT learns to estimate the parameter difference between two parameterized models, allowing repeated iterations to converge on the true parameters similarly to gradient descent. In this paper, we develop and analyze IRT in depth, including its similarities and differences with gradient descent. Our IRT implementation, TuneNet, is pre-trained via supervised learning over an auto-generated simulated dataset. We show that TuneNet can perform rapid, efficient system identification even when the true parameter values lie well outside those in the network’s training data, and can also learn real-world parameter values from visual data. We apply TuneNet to a sim-to-real task transfer experiment, allowing a robot to perform a dynamic manipulation task with a new object after a single observation.

A Feasibility Study of Autism Behavioral Markers in Spontaneous Facial, Visual, and Hand Movement Response Data.

Autism spectrum disorder (ASD) is a neurodevelopmental disability with atypical traits in behavioral and physiological responses. These atypical traits in individuals with ASD may be too subtle and subjective to measure visually using tedious methods of scoring. Alternatively, the use of intrusive sensors in the measurement of psychophysical responses in individuals with ASD may likely cause inhibition and bias. This paper proposes a novel experimental protocol for non-intrusive sensing and analysis of facial expression, visual scanning, and eye-hand coordination to investigate behavioral markers for ASD. An institutional review board approved pilot study is conducted to collect the response data from two groups of subjects (ASD and control) while they engage in the tasks of visualization, recognition, and manipulation. For the first time in the ASD literature, the facial action coding system is used to classify spontaneous facial responses. Statistical analyses reveal significantly (p <0.01) higher prevalence of smile expression for the group with ASD with the eye-gaze significantly averted (p<0.05) from viewing the face in the visual stimuli. This uncontrolled manifestation of smile without proper visual engagement suggests impairment in reciprocal social communication, e.g., social smile. The group with ASD also reveals poor correlation in eye-gaze and hand movement data suggesting deficits in motor coordination while performing a dynamic manipulation task. The simultaneous sensing and analysis of multimodal response data may provide useful quantitative insights into ASD to facilitate early detection of symptoms for effective intervention planning.

Modulation of grip force in unimanual manipulation tasks after theta burst stimulation over the dorsal premotor cortex

Aiming to examine the role of dorsal premotor cortex (PMd) during fine coordination of fingertip forces we applied theta burst repetitive magnetic stimulation (TBS) to disrupt neural processing in that specific area, after which subjects were performed various static and dynamic manipulation tasks (ramp-and-hold, oscillation force producing and simple lifting tasks) that recorded the grip (G) and load force (L) performed unimanual. In the two experiments nine healthy subjects (27.2 ± 4.1 yrs) used an instrumented device that recorded the grip and load force before and after we applied continous TBS (cTBS) either sham stimulation over contralateral PMd. There were no changes for static manipulation tasks in both stimulation protocols. After cTBS applied over dominant hemisphere grip force was disrupted only for simple lifting task. Our subjects have shown consistent differences of G/L ratio pre (1.10 ± 0.21) and post cTBS procedure (0.95 ± 0.23; p = 0.019). Our results suggest that inhibitory TBS protocol applied over PMd modulates the function of skillful and precise finger movements during object lifting. These findings support the role of PMd in human motor control and forces generation required to hold small objects stable in our hands.

A Dynamic Manipulation Strategy for an Intervention: Autonomous Underwater Vehicle

This paper presents the modelling and the control architecture of an Autonomous Underwater Vehicle for Intervention (I-AUV). Autonomous underwater manipulation with free-floating base is still an open topic of research, far from reaching an industrial product. Dynamic manipulation tasks, where relevant vehicle velocities are required during manipulation, over an additional challenge. In this paper, the accurate modelling of an I-AUV is described, not neglecting the interaction with the fluid. A grasp planning strategy is proposed and integrated in the control of the whole system. The performances of the I-AUV have been analysed by means of simulations of a dynamic manipulation task.

Quantification of dexterity as the dynamical regulation of instabilities: comparisons across gender, age, and disease.

Dexterous manipulation depends on using the fingertips to stabilize unstable objects. The Strength–Dexterity paradigm consists of asking subjects to compress a slender and compliant spring prone to buckling. The maximal level of compression [requiring low fingertip forces <300 grams force (gf)] quantifies the neural control capability to dynamically regulate fingertip force vectors and motions for a dynamic manipulation task. We found that finger dexterity is significantly affected by age (p = 0.017) and gender (p = 0.021) in 147 healthy individuals (66F, 81M, 20–88 years). We then measured finger dexterity in 42 hands of patients following treatment for osteoarthritis of the base of the thumb (CMC OA, 33F, 65.8 ± 9.7 years), and 31 hands from patients being treated for Parkinson’s disease (PD, 6F, 10M, 67.68 ± 8.5 years). Importantly, we found no differences in finger compression force among patients or controls. However, we did find stronger age-related declines in performance in the patients with PD (slope −2.7 gf/year, p = 0.002) than in those with CMC OA (slope −1.4 gf/year, p = 0.015), than in controls (slope −0.86 gf/year). In addition, the temporal variability of forces during spring compression shows clearly different dynamics in the clinical populations compared to the controls (p < 0.001). Lastly, we compared dexterity across extremities. We found stronger age (p = 0.005) and gender (p = 0.002) effects of leg compression force in 188 healthy subjects who compressed a larger spring with the foot of an isolated leg (73F, 115M, 14–92 years). In 81 subjects who performed the tests with all four limbs separately, we found finger and leg compression force to be significantly correlated (females ρ = 0.529, p = 0.004; males ρ = 0.403, p = 0.003; 28F, 53M, 20–85 years), but surprisingly found no differences between dominant and non-dominant limbs. These results have important clinical implications, and suggest the existence – and compel the investigation – of systemic versus limb-specific mechanisms for dexterity.

Grasping force based manipulation for multifingered hand-arm Robot using neural networks

Multifingered hand-arm robots play an important role in dynamic manipulation tasks. They can grasp and move various shaped objects. It is important to plan the motion of the arm and appropriately control the grasping forces for the multifingered hand-arm robots. In this paper, we perform the grasping force based manipulation of the multifingered hand-arm robot by using neural networks. The motion parameters are analyzed and planned with the constraint of the multi-arms kinematics. The optimal grasping force problem is recast as the second-order cone program. The semismooth Newton method with the Fischer-Burmeister function is then used to efficiently solve the second-order cone program. The neural network manipulation system is obtained via the fitting of the data that are generated from the optimal manipulation simulations. The simulations of optimal grasping manipulation are performed to demonstrate the effectiveness of the proposed approach.

Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi-modal human-in-the-loop approach

We propose an approach to efficiently teach robots how to perform dynamic manipulation tasks in cooperation with a human partner. The approach utilises human sensorimotor learning ability where the human tutor controls the robot through a multi-modal interface to make it perform the desired task. During the tutoring, the robot simultaneously learns the action policy of the tutor and through time gains full autonomy. We demonstrate our approach by an experiment where we taught a robot how to perform a wood sawing task with a human partner using a two-person cross-cut saw. The challenge of this experiment is that it requires precise coordination of the robot's motion and compliance according to the partner's actions. To transfer the sawing skill from the tutor to the robot we used Locally Weighted Regression for trajectory generalisation, and adaptive oscillators for adaptation of the robot to the partner's motion.

Dynamic contact force/torque observer: Sensor fusion for improved interaction control

The potential areas of application for robots have gradually extended beyond the classical industrial settings in large-scale enterprizes: nowadays, the integration of robots into daily life has become a central development in robotics. Here, one major challenge is the physical interaction with unknown and/or changing environments. Such an interaction requires knowledge of the exchanged contact forces and torques. To this end, robotic systems are typically equipped with force/torque sensors at the wrist. By using force control schemes that rely on measurements from these sensors, conventional manipulation tasks are successfully executed. In particular, for dynamic manipulation tasks, however, the problem arises that the inertial forces/torques of the end effector have a non-negligible effect on the measurements of the wrist sensor. This degrades the performance of the interaction control and constitutes a safety risk since the actual interaction forces/torques deviate from the desired values. As a solution to this problem, the paper discusses four contact force/torque observer designs: two approaches are based on the extended Kalman filter (EKF) and two approaches are based on the unscented Kalman filter (UKF). For both types, EKF and UKF, two different measurement vectors are considered: the first one only uses pose and force/torque measurements, whereas the second one also uses acceleration measurements to determine the contact forces and torques. The four observer designs are evaluated in simulation and experiment for six-degree-of-freedom (DOF) tasks.

Grip force adaptation in manipulation activities performed under different coating and grasping conditions

The aim of the study was to evaluate grip force (GF; normal component of hand–object interaction) adaptation across different manipulation conditions. We hypothesized (1) that the absolute safety margin (the difference between the exerted GF and the minimum GF that prevents slippage; absolute SM), rather than the relative SM (the same difference relative to the minimum GF required), could be an invariant feature of manipulation, as well as (2) that the SM would be higher in static than in dynamic tasks. Fourteen participants performed the free holding and the static holding tasks that required a same pulling force. Each task was performed using a variety of grasps and two different object coatings that both provided different frictions acting between the hand and the hand-held object. Both tasks revealed an increase in the relative SM associated with an increase in friction, while the absolute SM either remained unchanged (free holding) or suggested a moderate negative relationship (static holding task). Both relative and absolute SM were also higher in the free holding than in the static holding. The later result could be a consequence of the task mechanical conditions (i.e., dynamic vs. static), rather than of the difference in neural control mechanisms (feedback vs. feed-forward, respectively). The obtained findings suggest that the absolute SM (rather than the relative one) should be used in future studies of hand force coordination in healthy and clinical populations, while GF adaptation obtained from static and dynamic manipulation tasks should be separately assessed.

Hand function in multiple sclerosis: Force coordination in manipulation tasks

Objective To evaluate the methodology for exploring the specific aspects of functional impairment in multiple sclerosis (MS) through the pattern of forces exerted in various manipulation tasks. Methods Twelve mildly involved MS patients (EDSS 2.5–5.5) and 12 healthy controls performed various static and dynamic manipulation tasks with an instrumented device that recorded the grip ( G; normal to the digit device contact area) and load force ( L; tangential force that causes lifting). Results MS patients consistently displayed lower indices of task performance (as assessed by the ability to produce the required L profiles) and force coordination (as assessed by G/ L ratio, coupling of G and L, and G modulation) than the healthy controls across all tested tasks. Conclusions The applied methodology could be sensitive enough to detect the hand dysfunction in mildly involved individuals with MS. Particularly recommended for future evaluations of the impairment of hand function could be a simple lifting task and the static task of tracing a gradually changing L, as well as the variables depicting both the task performance and G/ L ratio. Significance The applied methodology could be developed into a standard clinical test for the assessment of hand function in MS and, possibly, in other neurological diseases.

Impaired Object Manipulation in Mildly Involved Individuals with Multiple Sclerosis

We investigated hand function in mildly involved multiple sclerosis (MS) patients (N = 16; Expanded Disability Status Scale 1-5, 9-hole peg test 14-32 s) during static and dynamic manipulation tasks using an instrumented device. When compared with healthy controls (N = 16), the patients revealed impaired task performance regarding their ability to exert prescribed patterns of load force (L; force acting tangentially at the digits-object surface). Regarding the coordination of grip force (G; normal component) and L, the data only revealed an elevated G/L ratio, although both the G and L coupling (maximum correlation coefficients and the time lags between them) and the G modulation (gain and offset of G with respect to L) remained comparable in the two groups. Finally, most of the data suggested no MS-specific effects of switching from uni- to bimanual tasks, from available visual feedback to deprived feedback conditions. We conclude that the deterioration in the ability for precise control of external forces and overgripping could precede the decoupling of G and L and decreased G modulation in early phases of the disease. The results also suggest that the applied methodology could be sensitive enough to detect mild levels of impairment of hand function in MS and, possibly, other neurological diseases.

Real-Time Motion Planning for a Volleyball Robot Task Based on a Multi-Agent Technique

This paper takes the volleyball robot task as an example of a dynamic manipulation task and presents a multi-agent-based motion-planning framework for it. Each subtask in the motion planning is defined as an agent. The motion planning is accomplished by the solution of each agent activated by a blackboard. It is convenient to extend the module and enable real-time control compared to the hierarchy and subsumption architecture. In the solution of each subtask, one difference from the motion planning of other ball-playing robots is that simple fuzzy rules are adopted to determine the hitting position, which leads to smaller hitting velocities beneficial for task control. The other difference is that the task-based directional manipulability measure (TDMM) is used to optimize the robot configuration on the basis of optimal hitting path, so as to improve the manipulating ability on task direction. The simulation of a planar three-link manipulator ball-hitting task is implemented by the MATLAB/Simulink tool. The virtual target of the hitting motion is validated on the ABB manipulator by the ball-catching experiment.

Manipulating the edge of instability

We investigate the integration of visual and tactile sensory input for dynamic manipulation. Our experimental data and computational modeling reveal that time-delays are as critical to task-optimal multisensory integration as sensorimotor noise. Our focus is a dynamic manipulation task “at the edge of instability.” Mathematical bifurcation theory predicts that this system will exhibit well-classified low-dimensional dynamics in this regime. The task was using the thumbpad to compress a slender spring prone to buckling as far as possible, just shy of slipping. As expected from bifurcation theory, principal components analysis gives a projection of the data onto a low dimensional subspace that captures 91–97% of its variance. In this subspace, we formulate a low-order model for the brain+hand+spring dynamics based on known mechanical and neurophysiological properties of the system. By systematically occluding vision and anesthetically blocking thumbpad sensation in 12 consenting subjects, we found that vision contributed to dynamic manipulation only when thumbpad sensation was absent. The reduced ability of the model system to compress the spring with absent sensory channels closely resembled the experimental results. Moreover, we found that the model reproduced the contextual usefulness of vision only if we took account of time-delays. Our results shed light on critical features of dynamic manipulation distinct from those of static pinch, as well as the mechanism likely responsible for loss of manual dexterity and increased reliance on vision when age or neuromuscular disease increase noisiness and/or time-delays during sensorimotor integration.

Static Single-Arm Force Generation With Kinematic Constraints

Smooth, frictionless, kinematic constraints on the motion of a grasped object reduce the motion freedoms at the hand, but add force freedoms, that is, force directions that do not affect the motion of the object. We are studying how subjects make use of these force freedoms in static and dynamic manipulation tasks. In this study, subjects were asked to use their right hand to hold stationary a manipulandum being pulled with constant force along a low-friction linear rail. To accomplish this task, subjects had to apply an equal and opposite force along the rail, but subjects were free to apply a force against the constraint, orthogonal to the pulling force. Although constraint forces increase the magnitude of the total force vector at the hand and have no effect on the task, we found that subjects applied significant constraint forces in a consistent manner dependent on the arm and constraint configurations. We show that these results can be interpreted in terms of an objective function describing how subjects choose a particular hand force from an infinite set of hand forces that accomplish the task. Without assuming any particular form for the objective function, the data show that its level sets are convex and scale invariant (i.e., the level set shapes are independent of the hand-force magnitude). We derive the level sets, or "isocost" contours, of subjects' objective functions directly from the experimental data.