Complex Object Manipulation Research Articles

A framework for robotic excavation and dry stone construction using on-site materials.

Automated building processes that enable efficient in situ resource utilization can facilitate construction in remote locations while simultaneously offering a carbon-reducing alternative to commonplace building practices. Toward these ends, we present a robotic construction pipeline that is capable of planning and building freeform stone walls and landscapes from highly heterogeneous local materials using a robotic excavator equipped with a shovel and gripper. Our system learns from real and simulated data to facilitate the online detection and segmentation of stone instances in spatial maps, enabling robotic grasping and textured 3D scanning of individual stones and rubble elements. Given a limited inventory of these digitized stones, our geometric planning algorithm uses a combination of constrained registration and signed-distance-field classification to determine how these should be positioned toward the formation of stable and explicitly shaped structures. We present a holistic approach for the robotic manipulation of complex objects toward dry stone construction and use the same hardware and mapping to facilitate autonomous terrain-shaping on a single construction site. Our process is demonstrated with the construction of a freestanding stone wall (10 meters by 1.7 meters by 4 meters) and a permanent retaining wall (65.5 meters by 1.8 meters by 6 meters) that is integrated with robotically contoured terraces (665 square meters). The work illustrates the potential of autonomous heavy construction vehicles to build adaptively with highly irregular, abundant, and sustainable materials that require little to no transportation and preprocessing.

Investigating the effects of the aging brain on real tool use performance-an fMRI study.

Healthy aging affects several domains of cognitive and motor performance and is further associated with multiple structural and functional neural reorganization patterns. However, gap of knowledge exists, referring to the impact of these age-related alterations on the neural basis of tool use-an important, complex action involved in everyday life throughout the entire lifespan. The current fMRI study aims to investigate age-related changes of neural correlates involved in planning and executing a complex object manipulation task, further providing a better understanding of impaired tool use performance in apraxia patients. A balanced number of sixteen older and younger healthy adults repeatedly manipulated everyday tools in an event-related Go-No-Go fMRI paradigm. Our data indicates that the left-lateralized network, including widely distributed frontal, temporal, parietal and occipital regions, involved in tool use performance is not subjected to age-related functional reorganization processes. However, age-related changes regarding the applied strategical procedure can be detected, indicating stronger investment into the planning, preparatory phase of such an action in older participants.

Motor control beyond reach-how humans hit a target with a whip.

Humans are strikingly adept at manipulating complex objects, from tying shoelaces to cracking a bullwhip. These motor skills have highly nonlinear interactive dynamics that defy reduction into parts. Yet, despite advances in data recording and processing, experiments in motor neuroscience still prioritize experimental reduction over realistic complexity. This study embraced the fully unconstrained behaviour of hitting a target with a 1.6-m bullwhip, both in rhythmic and discrete fashion. Adopting an object-centered approach to test the hypothesis that skilled movement simplifies the whip dynamics, the whip's evolution was characterized in relation to performance error and hand speed. Despite widely differing individual strategies, both discrete and rhythmic styles featured a cascade-like unfolding of the whip. Whip extension and orientation at peak hand speed predicted performance error, at least in the rhythmic style, suggesting that humans accomplished the task by setting initial conditions. These insights may inform further studies on human and robot control of complex objects.

Incipient Slip Detection Using Injected Vibration in Biomimetic Tactile Sensor

In robot manipulation, the prediction of gross slip and grasp stabilization control enable complex object manipulation. To realize this, researchers have been studying the use of incipient slip, which is a local deformation phenomenon of the contact surface. However, there are many issues in terms of practicality, because the cost of distributed tactile sensors for detecting incipient slip is high and there are restrictions on the motion state of the robot. In this study, we proposed a method for detecting incipient slip by focusing on vibrotactile sensing and static stick region. In the proposed method, a vibrotactile sensor with a flexible structure and a vibration motor are used to detect the change in the stick region as a propagation characteristic of the injected vibration. In the verification experiment, the proposed method showed an incipient slip detection performance close to that of the distributed tactile sensor, and was able to detect incipient slip even in static conditions that the conventional method could not handle. The method is expected to realize robust grasping control with a low-cost tactile sensor system.

Improvement decoding performance based on GQDA during a high engagement demanding paradigm

This research concentrates on the study of dexterous brain-computer interaction, combining a new paradigm and method for the decoding of electroencephalography (EEG) motor intention. An EEG evoking paradigm by implementing margin energy constraint in complex coupling objects is used for inducing strong intention and providing high recognition accuracy. Motor intention decoding from task-dependent EEG was given by a generalized quadratic discriminant analysis, which is proposed based on discriminant analysis. The generalization method is flexible and adaptive to any given data. Furthermore, we conduct a systematic comparison relative to extracted features in different dimensions and to find the optimal feature dimension. Cross-validation is used to make the generalized quadratic discriminant analysis method adaptive individually for each volunteer. The discriminant performance of state recognition given by area under the receiver operating characteristics curve is ∼ 0.96 from ten volunteers, this performance has been found to compare very favorably with some well-known powerful classifiers. It has been demonstrated that the proposed method generalized quadratic discriminant analysis has excellent decoding performance in manipulating complex objects. Together, this work can provide a guarantee for improving the accuracy of intention recognition on data level with the high engagement demanding task, and may inspire better EEG acquisition contexts.

The State of Lifelong Learning in Service Robots:

Service robots are appearing more and more in our daily life. The development of service robots combines multiple fields of research, from object perception to object manipulation. The state-of-the-art continues to improve to make a proper coupling between object perception and manipulation. This coupling is necessary for service robots not only to perform various tasks in a reasonable amount of time but also to continually adapt to new environments and safely interact with non-expert human users. Nowadays, robots are able to recognize various objects, and quickly plan a collision-free trajectory to grasp a target object in predefined settings. Besides, in most of the cases, there is a reliance on large amounts of training data. Therefore, the knowledge of such robots is fixed after the training phase, and any changes in the environment require complicated, time-consuming, and expensive robot re-programming by human experts. Therefore, these approaches are still too rigid for real-life applications in unstructured environments, where a significant portion of the environment is unknown and cannot be directly sensed or controlled. In such environments, no matter how extensive the training data used for batch learning, a robot will always face new objects. Therefore, apart from batch learning, the robot should be able to continually learn about new object categories and grasp affordances from very few training examples on-site. Moreover, apart from robot self-learning, non-expert users could interactively guide the process of experience acquisition by teaching new concepts, or by correcting insufficient or erroneous concepts. In this way, the robot will constantly learn how to help humans in everyday tasks by gaining more and more experiences without the need for re-programming. In this paper, we review a set of previously published works and discuss advances in service robots from object perception to complex object manipulation and shed light on the current challenges and bottlenecks.

Deep Residual Networks for User Authentication via Hand-Object Manipulations.

With the ubiquity of wearable devices, various behavioural biometrics have been exploited for continuous user authentication during daily activities. However, biometric authentication using complex hand behaviours have not been sufficiently investigated. This paper presents an implicit and continuous user authentication model based on hand-object manipulation behaviour, using a finger-and hand-mounted inertial measurement unit (IMU)-based system and state-of-the-art deep learning models. We employed three convolutional neural network (CNN)-based deep residual networks (ResNets) with multiple depths (i.e., 50, 101, and 152 layers) and two recurrent neural network (RNN)-based long short-term memory (LSTMs): simple and bidirectional. To increase ecological validity, data collection of hand-object manipulation behaviours was based on three different age groups and simple and complex daily object manipulation scenarios. As a result, both the ResNets and LSTMs models acceptably identified users’ hand behaviour patterns, with the best average accuracy of 96.31% and F1-score of 88.08%. Specifically, in the simple hand behaviour authentication scenarios, more layers in residual networks tended to show better performance without showing conventional degradation problems (the ResNet-152 > ResNet-101 > ResNet-50). In a complex hand behaviour scenario, the ResNet models outperformed user authentication compared to the LSTMs. The 152-layered ResNet and bidirectional LSTM showed an average false rejection rate of 8.34% and 16.67% and an equal error rate of 1.62% and 9.95%, respectively.

EEG intentions recognition in dynamic complex object control task by functional brain networks and regularized discriminant analysis

Most of the tasks involved in neurological rehabilitation are generally constrained. However the complexity of experimental paradigms with constraints for acquiring electroencephalography (EEG) are relatively low. In order to improve the level of arousal in the brain, we propose an energy-constrained dynamic and complex experimental paradigm. In this novel experimental paradigm, EEG signals collected under the novel paradigm have different motor intentions. In this study, we combine the phase synchronization method and common spatial pattern (CSP) to build functional brain networks of subjects. Based on this method, we investigate the correlation of the global and local features of the functional brain networks and use regularized discriminant analysis (RDA) to recognize features. The accuracy obtained by the proposed method for EEG intention recognition can reach 93.47% (p < 0.01). The results show that the proposed method can decode EEG intention with high recognition performance during the manipulation of complex objects, which lays a foundation for the study of neurological rehabilitation.

A systematic review of the current state of collaborative mixed reality technologies: 2013–2018

Over the last few decades, Mixed Reality (MR) interfaces have received great attention from academia and industry. Although a considerable amount of research has already been done to support collaboration between users in MR, there is still no systematic review to determine the current state of collaborative MR applications. In this paper, collaborative MR studies published from 2013 to 2018 were reviewed. A total of 259 papers have been categorised based on their application areas, types of display devices used, collaboration setups, and user interaction and experience aspects. The primary contribution of this paper is to present a high-level overview of collaborative MR influence across several research disciplines. The achievements from each application area are summarised. In addition, remarkable papers in their respective areas are highlighted. Among other things, our study finds that there are three complementary factors to support and enhance collaboration in MR environments: (i) annotation techniques, which provide non-verbal communication cues to users, (ii) cooperative object manipulation techniques, which divide complex 3D object manipulation process into simpler tasks between different users, and (iii) user perception and cognition studies, which aim to lessen cognitive workload for task understanding and completion, and to increase users’ perceptual awareness and presence. Finally, this paper identifies research gaps and future directions that can be useful for researchers who want to explore ways on how to foster collaboration between users and to develop collaborative applications in MR.

From Visual Understanding to Complex Object Manipulation

Planning and executing object manipulation requires integrating multiple sensory and motor channels while acting under uncertainty and complying with task constraints. As the modern environment is tuned for human hands, designing robotic systems with similar manipulative capabilities is crucial. Research on robotic object manipulation is divided into smaller communities interested in, e.g., motion planning, grasp planning, sensorimotor learning, and tool use. However, few attempts have been made to combine these areas into holistic systems. In this review, we aim to unify the underlying mechanics of grasping and in-hand manipulation by focusing on the temporal aspects of manipulation, including visual perception, grasp planning and execution, and goal-directed manipulation. Inspired by human manipulation, we envision that an emphasis on the temporal integration of these processes opens the way for human-like object use by robots.

Sharing hand gesture and sketch cues in remote collaboration

Many systems have been developed to support remote guidance, where a local worker manipulates objects under guidance of a remote expert helper. These systems typically use speech and visual cues between the local worker and the remote helper, where the visual cues could be pointers, hand gestures, or sketches. However, the effects of combining visual cues together in remote collaboration has not been fully explored. We conducted a user study comparing remote collaboration with an interface that combined hand gestures and sketching (the HandsInTouch interface) to one that only used hand gestures, when solving two tasks; Lego assembly and repairing a laptop. In the user study, we found that (1) adding sketch cues improved the task completion time, only with the repairing task as this had complex object manipulation but (2) using gesture and sketching together created a higher task load for the user.

Slip Detection With a Biomimetic Tactile Sensor

Slip detection helps to prevent robotic hands from dropping grasped objects and would thus enable complex object manipulation. Here we present a method of detecting slip with a biomimetic optical tactile sensor-the TacTip-that operates by measuring the positions of internal pins embedded in its compliant skin. We investigate whether local pin movement is a strong signal of slip. Accurate and robust discrimination between static and slipping objects is obtained with a support vector machine (accuracy 99.88%). We then demonstrate performance on a task in which a slipping object must be caught. For fast reaction times, a modified TacTip is made for high-speed data collection. Performance of the slip detection method is then validated under several test conditions, including varying the speed at which slip onset occurs and using novel shaped objects. The proposed methods should apply to tactile sensors that can detect the local velocities of surface movement. The sensor and slip detection methods are also well-suited for integration onto robotic hands for deploying slip control under manipulation.

Resilience to the contralateral visual field bias as a window into object representations

Viewing images of manipulable objects elicits differential blood oxygen level-dependent (BOLD) contrast across parietal and dorsal occipital areas of the human brain that support object-directed reaching, grasping, and complex object manipulation. However, it is unknown which object-selective regions of parietal cortex receive their principal inputs from the ventral object-processing pathway and which receive their inputs from the dorsal object-processing pathway. Parietal areas that receive their inputs from the ventral visual pathway, rather than from the dorsal stream, will have inputs that are already filtered through object categorization and identification processes. This predicts that parietal regions that receive inputs from the ventral visual pathway should exhibit object-selective responses that are resilient to contralateral visual field biases. To test this hypothesis, adult participants viewed images of tools and animals that were presented to the left or right visual fields during functional magnetic resonance imaging (fMRI). We found that the left inferior parietal lobule showed robust tool preferences independently of the visual field in which tool stimuli were presented. In contrast, a region in posterior parietal/dorsal occipital cortex in the right hemisphere exhibited an interaction between visual field and category: tool-preferences were strongest contralateral to the stimulus. These findings suggest that action knowledge accessed in the left inferior parietal lobule operates over inputs that are abstracted from the visual input and is contingent on analysis by the ventral visual pathway, consistent with its putative role in supporting object manipulation knowledge.

Predictability and Robustness in the Manipulation of Dynamically Complex Objects.

Manipulation of complex objects and tools is a hallmark of many activities of daily living, but how the human neuromotor control system interacts with such objects is not well understood. Even the seemingly simple task of transporting a cup of coffee without spilling creates complex interaction forces that humans need to compensate for. Predicting the behavior of an underactuated object with nonlinear fluid dynamics based on an internal model appears daunting. Hence, this research tests the hypothesis that humans learn strategies that make interactions predictable and robust to inaccuracies in neural representations of object dynamics. The task of moving a cup of coffee is modeled with a cart-and-pendulum system that is rendered in a virtual environment, where subjects interact with a virtual cup with a rolling ball inside using a robotic manipulandum. To gain insight into human control strategies, we operationalize predictability and robustness to permit quantitative theory-based assessment. Predictability is quantified by the mutual information between the applied force and the object dynamics; robustness is quantified by the energy margin away from failure. Three studies are reviewed that show how with practice subjects develop movement strategies that are predictable and robust. Alternative criteria, common for free movement, such as maximization of smoothness and minimization of force, do not account for the observed data. As manual dexterity is compromised in many individuals with neurological disorders, the experimental paradigm and its analyses are a promising platform to gain insights into neurological diseases, such as dystonia and multiple sclerosis, as well as healthy aging.

Representations for robot knowledge in the KnowRob framework

In order to robustly perform tasks based on abstract instructions, robots need sophisticated knowledge processing methods. These methods have to supply the difference between the (often shallow and symbolic) information in the instructions and the (detailed, grounded and often real-valued) information needed for execution. For filling these information gaps, a robot first has to identify them in the instructions, reason about suitable information sources, and combine pieces of information from different sources and of different structure into a coherent knowledge base. To this end we propose the KnowRob knowledge processing system for robots. In this article, we discuss why the requirements of a robot knowledge processing system differ from what is commonly investigated in AI research, and propose to re-consider a KR system as a semantically annotated view on information and algorithms that are often already available as part of the robot's control system. We then introduce representational structures and a common vocabulary for representing knowledge about robot actions, events, objects, environments, and the robot's hardware as well as inference procedures that operate on this common representation. The KnowRob system has been released as open-source software and is being used on several robots performing complex object manipulation tasks. We evaluate it through prototypical queries that demonstrate the expressive power and its impact on the robot's performance.

Acquisition of novel and complex motor skills: stable solutions where intrinsic noise matters less.

Most experimental paradigms in motor neuroscience have used relatively focal and experimentally constrained tasks to allow precise measurement and experimental control. Therefore, practice-induced improvements and learning have been confined to relatively simple changes or adaptations to external perturbations. Here, we propose an approach to study more complex skills that are novel and require more extensive practice, leading to quantitative and qualitative changes in overt performance. Central to these skills is that they have extrinsic redundancy that allows exploration and exploitation of dynamic properties of the task. We hypothesize that in such skills, humans seek stable solutions that are robust to perturbations that make their intrinsic noise matter less. Three experimental paradigms exemplify our model-based and hypothesis-driven approach to skill acquisition: discrete throwing, rhythmic ball bouncing, and complex object manipulation. In skittles, a throwing skill, results show that actors are sensitive to the error tolerance afforded by the task. In ball bouncing, we show that subjects exploit the dynamic stability of the task, where small errors and noise self-stabilize without explicit corrections. In manipulating a “cup of coffee,” subjects learn to optimize the safety or energy margins and scale it to their intrinsic variability. This research presents new experimental paradigms that characterize the behavioral correlates of neuroplasticity in more complex skill acquisition. This fundamental work is a platform for future work to develop behavioral interventions for clinical applications.

A multitasking architecture for humanoid robot programming

We propose a hierarchical, three-tiered motion programming architecture for humanoid robots that allows for the prioritized coordination of multiple tasks while taking into account the dynamics and other physics-based constraints that underlie typical humanoid robot tasks. We first introduce a data structure for generic humanoid robots based on a general description of what constitutes a humanoid that is workable and practical from a programming perspective, without overly restricting the diversity of humanoid designs. For the low-level language we develop an extension of Brockett's motion description language (MDL) that allows for the prioritized coordination of multiple tasks taking into account the dynamics-based requirements of typical humanoid manipulation tasks. The extended multitasking motion description language (MDLm) inherits the advantages of the original MDL, while making use of change of coordinates and the null space control formalism of Sentis and Khatib (Int J Humanoid Robot 2(4):505---518, 2005). We also develop a high-level language consisting of pre-defined motion primitives that constitute a vocabulary for generating more complex free-space motions and object manipulation tasks. These occupational tasks and free-space motions are derived based on methods and principles for measuring human task performance Drumwright (The Task Matrix: a robot-independent framework for programming humanoids. Ph.D dissertation, University of Southern California, 2007), Karger and Bayha (Engineered work measurement. Industrial Press, New York, 1965). The structure of the high level language is also inspired in part by well-established notation for dance choreography Huang and Hudak (Dance: a declarative language for the control of humanoid robots, 2003) Hutchinson (Labanotation or Kinetography Laban: system of analyzing and recording movement. Oxford University Press, New York, 1972). In the resulting architecture, high-level motion primitives and MDLm are integrated in a balanced and consistent manner that allows for flexible and intuitive programming in an efficient manner. A case study is offered to illustrate our architecture.

Exploratory behaviour and adaptation to novelty in preschool children with autism – a preliminary report

AbstractThe purpose of this study was to compare exploratory behaviours in children with autism and typically developing preschool children and the course of their adaptation to novelty. A series of five repeated trials was conducted, during which children were allowed to freely explore the experimental room. The results revealed differences between study groups in the overall rate of exploratory activity, which was lower in children with autism. Patterns of time characteristics of exploratory activity showed both similarities and differences between the groups. In both groups, the rate of simple exploratory behaviours (i.e. looking at an object, touching the object, manipulating one object) decreased with time, while the levels of diversive exploration (i.e. touching the wall or floor) increased. Children with autism engaged in less complex object manipulation than their peers. Similarly, their adaptation and habituation to a novel environment proceeded in a different way in the low stimulation zone than in the high stimulation zone. In the low and medium stimulation zones, the rate of exploration decreased with time, while in the high stimulation zone it remained relatively constant. In typically developing children, habituation occurred in all stimulation zones. These results suggest the presence of some differences between the patterns of adaptation to novelty in the two groups, which emerge in a stimulation-rich environment. Due to the limitations of the study, in particular the small number of subjects, the present paper should be treated as a preliminary report.

Design of a Lightweight, Cost Effective Thimble-Like Sensor for Haptic Applications Based on Contact Force Sensors

This paper describes the design and calibration of a thimble that measures the forces applied by a user during manipulation of virtual and real objects. Haptic devices benefit from force measurement capabilities at their end-point. However, the heavy weight and cost of force sensors prevent their widespread incorporation in these applications. The design of a lightweight, user-adaptable, and cost-effective thimble with four contact force sensors is described in this paper. The sensors are calibrated before being placed in the thimble to provide normal and tangential forces. Normal forces are exerted directly by the fingertip and thus can be properly measured. Tangential forces are estimated by sensors strategically placed in the thimble sides. Two applications are provided in order to facilitate an evaluation of sensorized thimble performance. These applications focus on: (i) force signal edge detection, which determines task segmentation of virtual object manipulation, and (ii) the development of complex object manipulation models, wherein the mechanical features of a real object are obtained and these features are then reproduced for training by means of virtual object manipulation.

Multiple objects tracking in the presence of long-term occlusions

We present a robust object tracking algorithm that handles spatially extended and temporally long object occlusions. The proposed approach is based on the concept of “object permanence” which suggests that a totally occluded object will re-emerge near its occluder. The proposed method does not require prior training to account for differences in the shape, size, color or motion of the objects to be tracked. Instead, the method automatically and dynamically builds appropriate object representations that enable robust and effective tracking and occlusion reasoning. The proposed approach has been evaluated on several image sequences showing either complex object manipulation tasks or human activity in the context of surveillance applications. Experimental results demonstrate that the developed tracker is capable of handling several challenging situations, where the labels of objects are correctly identified and maintained over time, despite the complex interactions among the tracked objects that lead to several layers of occlusions.