Object Manipulation Research Articles

Survey of learning-based approaches for robotic in-hand manipulation

Human dexterity is an invaluable capability for precise manipulation of objects in complex tasks. The capability of robots to similarly grasp and perform in-hand manipulation of objects is critical for their use in the ever changing human environment, and for their ability to replace manpower. In recent decades, significant effort has been put in order to enable in-hand manipulation capabilities to robotic systems. Initial robotic manipulators followed carefully programmed paths, while later attempts provided a solution based on analytical modeling of motion and contact. However, these have failed to provide practical solutions due to inability to cope with complex environments and uncertainties. Therefore, the effort has shifted to learning-based approaches where data is collected from the real world or through a simulation, during repeated attempts to complete various tasks. The vast majority of learning approaches focused on learning data-based models that describe the system to some extent or Reinforcement Learning (RL). RL, in particular, has seen growing interest due to the remarkable ability to generate solutions to problems with minimal human guidance. In this survey paper, we track the developments of learning approaches for in-hand manipulations and, explore the challenges and opportunities. This survey is designed both as an introduction for novices in the field with a glossary of terms as well as a guide of novel advances for advanced practitioners.

Exploring and Modeling Directional Effects on Steering Behavior in Virtual Reality.

Steering is a fundamental task in interactive Virtual Reality (VR) systems. Prior work has demonstrated that movement direction can significantly influence user behavior in the steering task, and different interactive environments (VEs) can lead to various behavioral patterns, such as tablets and PCs. However, its impact on VR environments remains unexplored. Given the widespread use of steering tasks in VEs, including menu adjustment and object manipulation, this work seeks to understand and model the directional effect with a focus on barehand interaction, which is typical in VEs. This paper presents the results of two studies. The first study was conducted to collect behavioral data with four categories: movement time, average movement speed, success rate, and reenter times. According to the results, we examined the effect of movement direction and built the SθModel. We then empirically evaluated the model through the data collected from the first study. The results proved that our proposed model achieved the best performance across all the metrics (r2 > 0.95), with more than 15% improvement over the original Steering Law in terms of prediction accuracy. Next, we further validated the SθModel by another study with the change of device and steering direction. Consistent with previous assessments, the model continues to exhibit optimal performance in both predicting movement time and speed. Finally, based on the results, we formulated design recommendations for steering tasks in VEs to enhance user experience and interaction efficiency.

Grasp context-dependent uncertainty alters the relative contribution of anticipatory and feedback-based mechanisms in object manipulation

Predictive control within dexterous object manipulation while allowing for the choice of contact points has been shown to employ a predominantly feedback-based force modulation. The anticipation is thought to be facilitated through the internal representation of the object dynamics being integrated and updated on a trial-to-trial basis with the feedback of contact locations on the object. This is as opposed to the classically studied memory representation-based fingertip force control for grasping with pre-selected contact locations. We designed a study to examine this grasp context-dependent asymmetry in sensorimotor integration by introducing binary uncertainty about the grasp type before movement initiation within the framework of motor planning. An inverted T-shaped instrumented object was presented to 24 participants as the manipulandum, and they were asked to reach, grasp, and lift it while minimising the peak roll. We dissociated the planning and the execution phases by pseudo-randomly manipulating the availability of visual contact cues on the object after movement onset. We analysed both derived as well as direct kinetic and kinematic measures of the grasp during the loading phase to understand the anticipatory coordination. Our findings suggest that uncertainty about the grasp context during movement preparation resulted in a shift towards feedback-based mechanisms for grasp force modulation despite the persistence of visual cues.

Exploring the Effect of Viewing Attributes of Mobile AR Interfaces on Remote Collaborative and Competitive Tasks.

Mobile devices have the potential to facilitate remote tasks through Augmented Reality (AR) solutions by integrating digital information into the real world. Although prior studies have explored Mobile Augmented Reality (MAR) for co-located collaboration, none have investigated the impact of various viewing attributes that can influence remote task performance, such as target object viewing angles, synchronization styles, or having a secondary small screen showing other users current view in the MAR environment. In this paper, we explore five techniques considering these attributes, specifically designed for two modes of remote tasks: collaborative and competitive. We conducted a user study employing various combinations of those attributes for both tasks. In both instances, results indicate users' optimal performance and preference for the technique that allows asynchronous viewing of object manipulations on the small screen. Overall, this paper contributes novel techniques for remote tasks in MAR, addressing aspects such as viewing angle and synchronization in object manipulation alongside secondary small-screen interfaces. Additionally, it presents the results of a user study evaluating the effectiveness, usability, and user preference of these techniques in remote settings and offers a set of recommendations for designing and implementing MAR solutions to enhance remote activities.

Investigating Object Translation in Room-scale, Handheld Virtual Reality.

Handheld devices have become an inclusive alternative to head-mounted displays in virtual reality (VR) environments, enhancing accessibility and allowing cross-device collaboration. Object manipulation techniques in 3D space with handheld devices, such as those in handheld augmented reality (AR), have been typically evaluated in table-top scale and we currently lack an understanding of how these techniques perform in larger scale environments. We conducted two studies, each with 30 participants, to investigate how different techniques impact usability and performance for room-scale handheld VR object translations. We compared three translation techniques that are similar to commonly studied techniques in handheld AR: 3DSlide, VirtualGrasp, and Joystick. We also examined the effects of target size, target distance, and user mobility conditions (stationary vs. moving). Results indicated that the Joystick technique, which allowed translation in relation to the user's perspective, was the fastest and most preferred, without difference in precision. Our findings provide insights for designing room-scale handheld VR systems, with potential implications for mixed reality systems involving handheld devices.

Rearranging Deformable Linear Objects for Implicit Goals with Self‐Supervised Planning and Control

The robotic manipulation of deformable linear objects is a frontier problem with many potential applications in diverse industries. However, most existing research in this area focuses on shape control for a provided explicit goal and does not consider physical constraints, which limits its applicability in many real‐world scenarios. In this study, a self‐supervised planning and control approach are proposed to address the challenge of rearranging deformable linear objects for implicit goals. Specifically, the context of making both ends of the object reachable (inside the robotic access range) and graspable (outside potential collision regions) by dual‐arm robots is considered. Firstly, the object is described with sequential keypoints and the correspondence‐based action is parameterized. Secondly, a generator capable of producing multiple explicit targets is developed, which adhere to implicit conditions. Thirdly, value models are learnt to assign the most promising explicit target as guidance and determine the goal‐conditioned action. All models within the policy are trained in a self‐supervised manner based on data collected from simulations. Importantly, the learned policy can be directly applied to real‐world settings since we do not rely on accurate dynamic models. The performance of the new method is validated with simulations and real‐world experiments.

Enhanced interlocking in granular jamming grippers through hard and soft particle mixtures

We investigate the influence of particle stiffness on the grasping performance of granular grippers, a class of soft robotic effectors that utilize granular jamming for object manipulation. Through experimental analyses and X-ray imaging, we show that grippers with soft particles exhibit improved wrapping of the object after jamming, in contrast to grippers with rigid particles. This results in significantly increased holding force through the interlocking. The addition of a small proportion of rigid particles into a predominantly soft particle mixture maintains the improved wrapping but also significantly increases the maximum holding force. These results suggest a tunable approach to optimizing the design of granular grippers for improved performance in soft robotics applications.Graphic abstract

A generalized framework for identification and geometrical feature extraction of circular and polygonal shaped prisms

Abstract This paper introduces a versatile framework crucial for robotic applications such as object manipulation, mobile robot navigation, and pole climbing. It addresses the identification of geometric shapes and dimensions of diverse objects found in varied environments. The proposed method utilizes LiDAR scanning to capture objects from different angles, generating point clouds merged through transformations and superimpositions. After filtering and slicing, intersections are isolated and projected onto a chosen datum plane. The framework employs Non-Linear Least Square fitting via Gauss Newton iterative approach, utilizing pseudo-inverse Jacobian of a hypotrochoid to approximate polygons. The algorithm consecutively fits polygon prisms, determining the best fit with the least norm of error. Results indicate an average least square error of less than 9% for radius fitting and a high f-score for shape identification.

Using an electrically charged mist to visualize the potential wells in “sonomaglev,” a combined diamagnetic-acoustic levitator

We use a water mist to experimentally visualize the wells in the potential energy of material levitated by a combined diamagnetic-acoustic levitator. The levitator consists of an 18.5 T superconducting magnet, which can levitate diamagnetic material, such as water, plastics, and organic materials, by applying a magnetic body force counteracting the force of gravity. Low-power ultrasound transducers operated at 37.5 kHz generate an acoustic field that spatially modulates the net force acting on the diamagnetically levitated material, making “sonomaglev” capable of levitating multiple objects in stable equilibrium. In these experiments, we levitate a mist of water droplets that are electrically charged so that they repel each other, preventing them from coalescing as a single drop in each of the local potential minima. The shapes of the potential wells are revealed by the shapes of clusters of droplets, which conform to the isosurfaces of the sum of the magnetic, gravitational, and acoustic potentials. The spacing of the droplets in a cluster is shown to depend on their charge, volume, and the force constant of the well in a simple model. Compared to acoustic levitation alone, the combination of diamagnetic and acoustic levitation allows more scope for the manipulation of levitated objects, since the acoustic field is not constrained by the requirement to balance the force of gravity. The method demonstrated here allows the influence on the potential energy of switching on the acoustic field to be observed directly.

Switchable Optical Trapping of Mie‐Resonant Phase‐Change Nanoparticles

AbstractOptical tweezers revolutionized the manipulation of nanoscale objects. Typically, tunable manipulations of optical tweezers rely on adjusting either the trapping laser beams or the optical environment surrounding the nanoparticles. Here, tunable and switchable trapping using nanoparticles made of a phase‐change material (vanadium dioxide or VO2) are achieved. By varying the intensity of the trapping beam, transitions of the VO2 between monoclinic and rutile phases are induced. Depending on the nanoparticles' sizes, they exhibit one of three behaviors: small nanoparticles (in the settings, radius wavelength ) remain always attracted by the laser beam in both material phases, large nanoparticles () remain always repelled. However, within the size range of , the phase transition of the VO2 switches optical forces between attractive and repulsive, thereby pulling/pushing them toward/away from the beam center. The effect is reversible, allowing the same particle to be attracted and repelled repeatedly. The phenomenon is governed by optical Mie modes of the nanoparticles and their alterations during the phase transition of the VO2. This work provides an alternative solution for dynamic optical tweezers and paves a way to new possibilities, including optical sorting, light‐driven optomechanics and single‐molecule biophysics.

Fourth-Order Accurate Strain-Parameterized Shape Representation of Beam Elements for Modeling Continuum Robots and for Robotic Manipulation of Slender Objects

Fourth-Order Accurate Strain-Parameterized Shape Representation of Beam Elements for Modeling Continuum Robots and for Robotic Manipulation of Slender Objects

Octopus-Inspired Adhesives with Switchable Attachment to Challenging Underwater Surfaces.

Adhesives that excel in wet or underwater environments are critical for applications ranging from healthcare and underwater robotics to infrastructure repair. However, achieving strong attachment and controlled release on difficult substrates, such as those that are curved, rough, or located in diverse fluid environments, remains a major challenge. Here, an octopus-inspired adhesive with strong attachment and rapid release in challenging underwater environments is presented. Inspired by the octopus's infundibulum structure, a compliant, curved stalk, and an active deformable membrane for multi-surface adhesion are utilized. The stalk's curved shape enhances conformal contact on large-scale curvatures and increases contact stress for adaptability to small-scale roughness. These synergistic mechanisms improve contact across multiple length scales, resulting in switching ratios of over 1000 within ≈30 ms with consistent attachment strength of over 60 kPa on diverse surfaces and conditions. These adhesives are demonstrated through the robust attachment and precise manipulation of rough underwater objects.

Topological magnetic defects in a strong permanent magnet Nd2Fe14B

Skyrmions, highly mobile and manipulable magnetic objects, have garnered significant interest for their potential applications in modern magnetic device technology. However, their formation has been limited to a small number of materials due to the delicate balance of magnetic energies involved. In this study, we report the irreversible formation of skyrmions in Nd2Fe14B, a rare-earth permanent magnet characterized by large magnetic anisotropy. Remarkably, these magnetic topological defects exist over a wide range of temperatures and magnetic fields (295–565 K and 0–0.1 T) due to the high Curie temperature and significant magnetic anisotropy of Nd2Fe14B. Our findings not only unveil a new material where skyrmions can form but also open up possibilities for exploring topological defects in various magnetic systems, including strong hard magnets.

Tailoring stiffness distribution of tendon-driven continuum finger from manipulation force vector

Continuum soft-robotic fingers/arms have emerged as a promising solution to realize abilities to delicately manipulate objects, conform to various shapes, and maintain cost-efficiency (i.e. fewer actuators and less computational resource). This paper introduces a novel approach to designing continuum flexible fingers by optimizing the fingers' thickness distribution to achieve specific target force vectors for specific manipulations, enabling actions like pull-in handing (retraction), push-out manipulation, secure grip, and gentle object hold. The proposed method employs a genetic algorithm to optimize the thickness distribution of the continuum finger driven by a single motor, resulting in a highly versatile and cost-effective solution. The proposed method includes physical simulations to validate the approach's effectiveness in applying desired force vectors during manipulation interactions. This work represents a significant advancement in continuum robotic systems, potentially impacting a wide range of applications in human-robot collaboration.

Does Embodiment in Virtual Reality Boost Learning Transfer? Testing an Immersion-Interactivity Framework

This study investigates the role of embodiment when learning a technical procedure in immersive virtual reality (VR) by introducing a framework based on immersion and interactivity. The goal is to determine how increasing the levels of immersion and interactivity affect learning experiences and outcomes. In a 2 × 2 factorial design, 177 high school students were assigned to one of four experimental conditions, varying levels of immersion (learning in immersive virtual reality wearing a head-mounted display (VR) vs. learning via a computer screen (PC)) and interactivity (directly manipulating objects using controllers/mouse and keyboard (congruent) vs. indirectly manipulating objects with a laser pointer to select a course of action (incidental)). The main outcome measure was a transfer task in which students were required to perform the task they had learned in the virtual environment using concrete objects in real life. Results demonstrated that students in the VR conditions experienced significantly higher levels of presence, agency, location, body ownership, and embodied learning compared to participants in the PC conditions. Additionally, students’ performance during the virtual lesson predicted their real-life transfer test. However, there were no significant effects of immersion or interactivity on any of the transfer measures. The results suggest that high immersion in VR can increase self-reported measures of presence, agency, location, body ownership, and embodied learning among students. However, increased embodiment—manipulated by adding immersion and congruent manipulation of objects did not improve transfer.

Precise Manipulation of Micro Objects by Light Radiation Pressure

Precise Manipulation of Micro Objects by Light Radiation Pressure

Implementing a Vision-Based ROS Package for Reliable Part Localization and Displacement from Conveyor Belts

The use of computer vision in the industry has become fundamental, playing an essential role in areas such as quality control and inspection, object recognition/tracking, and automation. Despite this constant growth, robotic cell systems employing computer vision encounter significant challenges, such as a lack of flexibility to adapt to different tasks or types of objects, necessitating extensive adjustments each time a change is required. This highlights the importance of developing a system that can be easily reused and reconfigured to address these challenges. This paper introduces a versatile and adaptable framework that exploits Computer Vision and the Robot Operating System (ROS) to facilitate pick-and-place operations within robotic cells, offering a comprehensive solution for handling and sorting random-flow objects on conveyor belts. Designed to be easily configured and reconfigured, it accommodates ROS-compatible robotic arms and 3D vision systems, ensuring adaptability to different technological requirements and reducing deployment costs. Experimental results demonstrate the framework’s high precision and accuracy in manipulating and sorting tested objects. Thus, this framework enhances the efficiency and flexibility of industrial robotic systems, making object manipulation more adaptable for unpredictable manufacturing environments.

Sensorless Human–Robot Interaction: Real‐Time Estimation of Co‐Grasped Object Mass and Human Wrench for Compliant Interaction

Human–robot physical interaction in shared object manipulation can fully leverage the strengths of both humans and collaborative robots, achieving better interaction outcomes. However, in typical physical interactions, the inertia parameters of the object are either assumed to be known or ignored, simplifying the interaction to be directly between the human and the collaborative robot. This simplification can lead to inaccuracies in the robot's understanding of human intent due to deviations in object dynamics. This article presents a method that eliminates the need for a force sensor on the human side during human–robot collaboration. Using an extended Kalman filter, the method estimates object parameters online and applies a disturbance observer to determine human force. This approach reduces human effort and improves tracking performance compared to traditional methods. It also avoids the hassle of installing and removing sensors in complex scenarios. By estimating object mass and human force in real time, the robot can adjust its behavior to ensure safer and smoother interactions. Furthermore, the system can handle various objects without requiring specific programming for each case.

Obstacle avoidance shape control of deformable linear objects with online parameters adaptation based on differentiable simulation

The manipulation of deformable linear objects (DLOs) such as ropes, cables, and hoses by robots has promising applications in various fields such as product assembly and surgical suturing. However, DLOs are more difficult to manipulate than rigid objects because their shape changes during manipulation. Furthermore, preventing a DLO from colliding with the environment is important to prevent it from becoming entangled and causing shape control to fail. In this paper, we proposed an obstacle avoidance and shape control scheme for DLOs based on differentiable simulation that does not require prior data or a specialized controller. First, we established a dynamic model of the DLO that allows for both forward dynamics transfer and error backpropagation to obtain gradients. Then, we employed model predictive control to optimize the embedded neural network for predicting the actions that would manipulate the DLO. Finally, the control scheme was made applicable to DLOs with different material properties by allowing online adaptation of the model parameters essential to deformation during manipulation. Simulations and real-world experiments demonstrate that the proposed control scheme could manipulate the DLO stably and accurately to avoid obstacles and achieve the goal state. In addition, the online adaptation of parameters helped mitigate the sim-to-real gap.

RoboMan: An Adult-Sized Humanoid Robot with Enhanced Performance, Inherent Stability, and Two-Stage Balance Control to Facilitate Research on Humanoids

Creating an adult-sized humanoid robot with stable walking capabilities is a major challenge in robotics. While many renowned research groups focus on robots for perilous work environments and precision tasks, our approach simplifies balance control, making it accessible to robotics research groups and educational institutes. This facilitates the development of complex functionalities such as vision and object manipulation for adult-sized humanoids. This research article introduces RoboMan II, an advanced version of RoboMan I, which won the most prestigious award in all humanoid robot leagues at RoboCup 2016 due to its exceptional performance in walking and playing soccer. RoboMan II features significant improvements in performance, inherent stability, recovery after falls, and balance control. To facilitate its development, RoboMan II is lighter and incorporates a modified foot and parallel structure for its leg to boost its inherent stability, along with a two-stage balance control system for Immediate Response and Gradual Adaptation, enhancing its adaptability in various environments. Our simulation results demonstrate that RoboMan II’s walking stability on flat surfaces improved significantly in the face of minor perturbations, with the number of steps within the stable region increasing from 24%, with only the immediate controller to 58% when both controllers were used. Similar improvements were observed on inclined surfaces. Additionally, the 3D CAD files for all of the robot parts are released as open source in conjunction with this paper to facilitate reproduction and further innovation. The forthcoming RoboMan III will incorporate custom servo motors for increased speed, torque, and enhanced fall recovery, preventing disengagement of the gear box after a fall. It promises to be an invaluable asset for research and practical applications in humanoid robotics.