3D Manipulation Research Articles

Understanding user experience, task performance, and task interdependence in symmetric and asymmetric VR collaborations

Asymmetric collaboration is an important topic for the research of multiuser collaborative systems. Previous works have shown that by providing different abilities, devices or content to different users, users can take advantage of the unique features of each side and collaborate effectively with each other. However, there is limited work comparing the differences between asymmetric and symmetric Virtual Reality (VR) collaboration systems. How task complexity may affect symmetric and asymmetric VR collaboration is also unclear. In this paper, we present a comparative study that investigated how user experiences and task performance vary in symmetric and asymmetric VR collaboration. In addition, we also explored how task interdependence correlates with user experience and task performance. Participants were asked to collaboratively perform 3D object selection and manipulation tasks in pairs. A within-subjects study was conducted, where participants used PC and PC, VR and VR, and PC and VR, respectively in three conditions. Our results revealed that the asymmetric collaboration using both PC and VR showed the best results in closeness of relationship, social presence and task performance; the PC symmetric collaborative system showed the worst user experiences and task performance. Both user experience and task performance showed a positive correlation with task interdependence. We discussed the effects of the collaborativ mode and device on the user experience and task performance, and the implications for future symmetric and asymmetric VR collaboration systems.Graphical abstract

Observer-Based Gate Recurrent Learning for Model-Free Control of 6-DoF Manipulators: Considerations for Input Saturation and Output Constraints

Observer-Based Gate Recurrent Learning for Model-Free Control of 6-DoF Manipulators: Considerations for Input Saturation and Output Constraints

A Neuro-Fuzzy Solution for Inverse Kinematics of Redundant Manipulators

Abstract: This paper presents a novel approach for solving the inverse kinematics (IK) problem of redundant robotic manipulators using an Adaptive Neuro-Fuzzy Inference System (ANFIS). Redundant manipulators, characterized by having more degrees of freedom (DOF) than necessary for a task, introduce complexity in their IK solutions due to their non-linear, time-varying, and transcendental nature. Traditional methods such as algebraic or geometric solutions are often computationally intensive and can suffer from issues such as singularities. This study leverages ANFIS, combining the learning capability of neural networks with the logic-based structure of fuzzy systems, to predict IK solutions for 5-DOF and 7-DOF manipulators. The Denavit-Hartenberg (D-H) notation is employed for the kinematic modeling of robot links. Comparative analysis of the predicted IK values using ANFIS demonstrates its high accuracy and efficiency, as validated by surface and residual plots. The results show that ANFIS can be effectively utilized for fast and reliable IK predictions, offering a robust alternative to traditional approaches. This method can be applied in robotic systems requiring high precision in constrained or dynamic environments.

A Comparative Study of Handheld Augmented Reality Interaction Techniques for Developing AR Instructions using AR Authoring Tools

Augmented Reality (AR) instructions offer companies tremendous savings potential. However, developing these AR instructions has traditionally been challenging due to the need for programming skills and spatial knowledge. To address this complexity, industry and academia are working to simplify AR development. A crucial aspect of this process is the accurate positioning of AR content within the physical environment, which requires effective AR interaction techniques that enable full 3D manipulation of AR elements. In this study, we conducted an experimental comparison of three different AR interaction techniques with 55 participants to empirically assess their performance, workload, and user satisfaction across tasks related to AR instruction development. Our findings contribute to the design of future AR instructions and AR authoring tools, emphasizing the importance of evaluating AR interaction techniques that can be utilized by users without programming experience tailored to the specific needs of the intended application domain.

Smooth and Time-Optimal Trajectory Planning for Robots Using Improved Carnivorous Plant Algorithm

To improve the safety and reliability of robotic manipulators during high-speed precision movements, this paper proposes a method for smooth and time-optimal trajectory planning incorporating kinodynamic constraints. The primary objective is to use an evolutionary algorithm to determine a trajectory by considering time and jerk within the feasible path-pseudo-velocity phase plane region. Firstly, the path parameterization theory extracted the maximum pseudo-velocity projection curve from the kinodynamic constraints. Subsequently, the feasible region in the phase plane was defined through reachability analysis of discrete linear systems. Thereafter, we constructed the trajectory function using a cubic B-spline curve, optimizing its control points with an improved carnivorous plant optimization algorithm. Finally, the effectiveness and practicality of this method were verified through simulations on a 6-DOF manipulator.

3D manipulation of small multicellular organisms in soft microtubes

Small multicellular organisms, such as C. elegans and zebrafish larvae, are essential models in biological research, but their 3D manipulation poses challenges due to their small size. Manual positioning is labor-intensive and imprecise. To address this, we developed a soft PDMS microtube that can be gently stretched and released, immobilizing organisms without anesthetization and damage. This tube enables easy rotation and adjustment of orientation, facilitating comprehensive imaging. Functionality testing showed effective immobilization as well as precise imaging and microinjection. Zebrafish larvae were successfully injected with fluorescein in the hindbrain without anesthesia. This technique offers a simple, efficient, and non-damaging method for 3D manipulation and imaging of small multicellular organisms and provides a versatile tool for biological research practices.

Translational and rotational drives of micro-droplets of nematic liquid crystal with chiral dopant

The translational and rotational control of liquid crystal micro-droplets by electric fields has been explored for the future use in MEMS and lab-on-a-chip devices. The liquid crystal droplets are generated by the phase transition process of a liquid crystalline material, 4-n-4’-pentylcyanobiphenyl (5CB), from the isotropic to the nematic phase, and are thus suspended in the isotropic phase of 5CB. The addition of a chiral dopant to 5CB induces the symmetry breaking of the molecular orientation configuration within the droplet, leading to the formation of helical molecular orientation configurations. The helical configuration of the molecular orientation field is the key to enabling the translational and rotational drives of the droplet. Under electric fields, the translational drive of the droplets occurs in a direction perpendicular to both the electric field and the helical axis, and the rotational drive of the droplets occurs to align the helical axis perpendicular to the electric field. Finally, we propose the method for 2D and 3D manipulation of the droplets by combining the translational and rotational drives.

Selectively Tunable Joints With Variable Stiffness for a Magnetically-Steerable 6-DOF Manipulator

Selectively Tunable Joints With Variable Stiffness for a Magnetically-Steerable 6-DOF Manipulator

Micromotor based on single fiber optical vortex tweezer

Optical micromotors are powerful tools for trapping and rotating microparticles in various fields of bio-photonics. Conventionally, optical micromotors are built using bulk optics, such as microscope objectives and SLMs. However, optical fibers provide an attractive alternative, offering a flexible photon platform for optical micromotor applications. In this paper, we present an optical micromotor designed for 3D manipulation and rotation based on a single fiber optical vortex tweezer. A tightly focused vortex beam is excited by preparing a spiral zone plate with an ultrahigh numerical aperture of up to 0.9 at the end facet of a functionalized fiber. The focused vortex beam can optically manipulate and rotate a red blood cell in 3D space far from the fiber end facet. The trapping stiffness in parallel and perpendicular orientations to the fiber axis are measured by stably trapping a standard 3-µm silica bead. The rotational performance is analyzed by rotating a trimer composed of silica beads on a glass slide, demonstrating that the rotational frequency increases with rising optical power and the rotational direction is opposite to the topological charge of the spiral zone plate. The proposed fiber micromotor with its flexible manipulation of microparticle rotation circumvents the need for the precise relative position control of multiple fiber combinations and the use of specialized fibers. The innovations hold promising potential for applications in microfluidic pumping, biopsy, micromanipulation, and other fields.

A Force Control Method Integrating Human Skills for Complex Surface Finishing

Force control is one of the core modules for surface finishing such as grinding, polishing and sanding. However, the current force control methods based on human skills lack in-depth analysis of data patterns or are only applicable to flat surfaces. In addition, surface finishing is mainly performed by hand, resulting in low processing efficiency and poor product consistency. Therefore, this paper proposes a force control method that incorporates human skills to achieve relatively accurate force skill transfer and complex surface finishing. Firstly, human skills consisting of the force skill and the motion skill are learned. The force skill is used to generate the desired force. Then, a series of discrete poses are obtained based on human demonstration and combined with the motion skill to generate the desired trajectory. Finally, a computed-torque impedance control method is proposed to achieve relatively accurate force skill transfer and complex surface finishing by incorporating the desired trajectory and the desired force. The experiments are conducted on a platform composed of a 7-DOF collaborative robot manipulator from Franka Emika and a complex violin surface. The results demonstrate that the proposed force control method can achieve relatively accurate force skill transfer and improve the surface quality of the workpiece.

Acoustography by Beam Engineering and Acoustic Control Node: BEACON.

Acoustic manipulation has emerged as a valuable tool for precision controls and dynamic programming of cells and particles. However, conventional acoustic manipulation approaches lack the finesse necessary to form intricate, configurable, continuous, and 3D patterning of particles. Here, this study reports acoustography by Beam Engineering and Acoustic Control Node (BEACON), which delivers intricate, configurable patterns by guiding particles along custom paths with independent phase modulation. Leveraging analytical methods of orbital angular momentum beam via iterative Wirtinger hologram algorithm, this study accomplish acoustography by facilitating orbital angular momentum traps, enabling continuous 2D and 3D acoustic manipulation of microparticles in any desired geometry, with phase modulation independent of intensity. Utilizing on-chip acoustography, the BEACON platform markedly increases the space-bandwidth product to 31000 while attaining an enhanced resolution with a pixel size of ≈25µm, surpassing the typical resolution of over 200µm in previous holographic particle manipulation methods. The capabilities of BEACON are demonstrated in creating intricate triple helical tracing structures using microdroplets (20µm in diameter) and those carrying DNA to validate the effectiveness of the acoustography and phase control methods. This study offers new particle manipulation opportunities, paving the way for next-generation biomedical systems and the future of contact-free precision manufacturing.

Tunable Acoustic Tweezer System for Precise Three-Dimensional Particle Manipulation

This study introduces a tunable acoustic tweezer system designed for precise three-dimensional particle trapping and manipulation. The system utilizes a dual-liquid-layer acoustic lens, which enables the dynamic control of the focal length through the adjustable curvature of a latex membrane. This tunability is essential for generating the acoustic forces necessary for effective manipulation of particles, particularly along the direction of acoustic wave propagation (z-axis). Experiments conducted with spherical particles as small as 1.5 mm in diameter demonstrated the system’s capability for stable trapping and manipulation. Performance was rigorously evaluated through both z-axis and 3D manipulation tests. In the z-axis experiments, the system achieved a manipulation range of 33.4–53.4 mm, with a root-mean-square error and standard deviation of 0.044 ± 0.045 mm, which highlights its precision. Further, the 3D manipulation experiments showed that particles could be accurately guided along complex paths, including multilayer rectangular and helical trajectories, with minimal deviation. A visual feedback-based particle navigation system significantly enhanced positional accuracy, reducing errors relative to open-loop control. These results confirm that the tunable acoustic tweezer system is a robust tool for applications requiring precise control of particles with diameter of 1.5 mm in three-dimensional environments. Considering its ability to dynamically adjust the focal point and maintain stable trapping, this system is well suited for tasks demanding high precision, such as targeted particle delivery and other applications involving advanced material manipulation.

In-field performance evaluation of robotic arm developed for harvesting cotton bolls

Robotic harvesting of cotton bolls combines the advantages of manual picking and mechanical harvesting, including selective picking and reduced labor dependency. Thus, we designed, developed, and tested a cotton-picking robotic arm equipped with a depth camera, a 4-DoF manipulator, and a vacuum-type end-effector. Convolutional neural networks were implemented for cotton boll recognition, while an artificial neural network-assisted particle swarm optimization-based model was utilized for solving inverse kinematics. Linear actuators, controlled by a microcontroller and relays, positioned the end-effector for cotton boll harvesting. Laboratory testing resulted in picking 44–49 cotton bolls per hour, taking approximately 73.0 s per boll. During field testing, the robotic arm picked 40–42 bolls per hour with an average time of 86.0 s per boll. Picking efficiency ranged from 76.19% to 85.71% in the lab and 66.32% to 72.04% in the field. The complex structure of cotton plants sometimes hindered the end-effector’s path, resulting in lower picking efficiency during field testing. As observed in present study, manual pickers achieved a significantly higher picking rate, harvesting 1.381 kg of cotton per hour, nearly 4.7 times more than the robotic arm. Manual pickers achieved 98.04% picking efficiency, significantly higher than the robotic arm’s 68.25%. This significant difference in efficiency is due to the agility, adaptability, and decision-making abilities of human pickers. The developed robotic arm demonstrates competitive efficiency in the domain of harvesting robots, contributing to advancements in robotic agriculture. Future research and development efforts in this field hold the potential to enhance the efficiency and productivity of robotic cotton harvesters.

Integrated Intelligent Control of Redundant Degrees-of-Freedom Manipulators via the Fusion of Deep Reinforcement Learning and Forward Kinematics Models

Redundant degree-of-freedom (DOF) manipulators offer increased flexibility and are better suited for obstacle avoidance, yet precise control of these systems remains a significant challenge. This paper addresses the issues of slow training convergence and suboptimal stability that plague current deep reinforcement learning (DRL)-based control strategies for redundant DOF manipulators. We propose a novel DRL-based intelligent control strategy, FK-DRL, which integrates the manipulator’s forward kinematics (FK) model into the control framework. Initially, we conceptualize the control task as a Markov decision process (MDP) and construct the FK model for the manipulator. Subsequently, we expound on the integration principles and training procedures for amalgamating the FK model with existing DRL algorithms. Our experimental analysis, applied to 7-DOF and 4-DOF manipulators in simulated and real-world environments, evaluates the FK-DRL strategy’s performance. The results indicate that compared to classical DRL algorithms, the FK-DDPG, FK-TD3, and FK-SAC algorithms improved the success rates of intelligent control tasks for the 7-DOF manipulator by 21%, 87%, and 64%, respectively, and the training convergence speeds increased by 21%, 18%, and 68%, respectively. These outcomes validate the proposed algorithm’s effectiveness and advantages in redundant manipulator control using DRL and FK models.

Pick and Place Control of a 3-DOF Robot Manipulator Based on Image and Pattern Recognition

Board games like chess serve as an excellent testbed for human–robot interactions, where advancements can lead to broader human–robot cooperation systems. This paper presents a chess-playing robotic system to demonstrate controlled pick and place operations using a 3-DoF manipulator with image and speech recognition. The system identifies chessboard square coordinates through image processing and centroid detection before mapping them onto the physical board. User voice input is processed and transcribed into a string from which the system extracts the current and destination locations of a chess piece with a word error rate of 8.64%. Using an inverse-kinematics algorithm, the system calculates the joint angles needed to position the end effector at the desired coordinates actuating the robot. The developed system was evaluated experimentally on the 3-DoF manipulator with a voice command used to direct the robot movement in grasping a chess piece. Consideration was made involving both the own pieces as well as capturing the opponent’s pieces and moving the captured piece outside the board workspace.

Guaranteed real-time cooperative collision avoidance for n-DOF manipulators

Abstract This paper presents a decentralized, cooperative, real-time avoidance control strategy for robotic manipulators. The proposed avoidance control law builds on the concepts of artificial potential field functions and provides tighter bounds on the minimum safe distance when compared to traditional potential-based controllers. Moreover, the proposed avoidance control law is given in analytical, continuous closed form, avoiding the use of optimization techniques and discrete algorithms, and is rigorously proven to guarantee collision avoidance at all times. Examples of planar and 3D manipulators with cylindrical links under the proposed avoidance control are given and compared with the traditional approach of modeling links and obstacles with multiple spheres. The results show that the proposed avoidance control law can achieve, in general, faster convergence, smaller tracking errors, and lower control torques than the traditional approach. Furthermore, we provide extensions of the avoidance control to robotic manipulators with bounded control torques.

Learning complex nonlinear dynamics of a 3D translational parallel manipulator using neural network

This study investigates modeling the dynamics of a 3D translational parallel manipulator with closed chains using feedforward neural networks (FFNNs). The dataset exceeds 50,000 samples, incorporating experimental data collected from a robot prototype using MATLAB® real-time workshop and the National InstrumentsTM DAQ toolbox, as well as CAD simulation data from MSC ADAMS software. While achieving satisfactory mean squared error (MSE), some predictions did not fully capture the manipulator's dynamics, with small overfitting observed. A Deep Neural Network (DNN) was tested but faced overfitting and high computational costs, despite being trained on a subset of the dataset. This highlighted the limitations of DNNs for modeling such complicated parallel robots with closed chains and parallelograms. FFNNs were preferred for their simplicity and lower overfitting risk. L2 regularization and k-fold validation were applied to improve performance. Transfer learning (TL) was also employed, fine-tuning a new network with weights from pre-trained FFNNs using a smaller, unseen dataset. This approach significantly reduced MSE and completely eliminated overfitting, demonstrating the effectiveness of TL in refining model performance for forward and inverse dynamics. These findings suggest that FFNNs, combined with TL, L2 regularization, and k-fold validation, offer a robust method for accurately modeling complex robotic dynamics, enhancing control and optimization strategies for complicated robotic systems. Training for all networks was conducted within the MATLAB® environment.

Inverse kinematic solution and singularity avoidance using a deep deterministic policy gradient approach

<p>The robotic arm emerges as a subject of paramount significance within the industrial landscape, particularly in addressing the complexities of its kinematics. A significant research challenge lies in resolving the inverse kinematics of multiple degree of freedom (M-DOF) robotic arms. The inverse kinematics of M-DOF robotic arms pose a challenging problem to resolve, thus it involves consideration of singularities which affect the arm robot movement. This study aims a novel approach utilizing deep reinforcement learning (DRL) to tackle the inverse kinematic problem of the 6-DOF PUMA manipulator as a representative case within the M-DOF manipulator. The research employs Jacobian matrix for the kinematics system that can solve the singularity, and deep deterministic policy gradient (DDPG) as the kinematics solver. This chosen technique offers enhancing speed and ensuring stability. The results of inverse kinematic solution using DDPG were experimentally validated on a 6-DOF PUMA arm robot. The DDPG successfully solves inverse kinematic solution and avoids the singularity with 1,000 episodes and yielding a commendable total reward of 1,018.</p>

Analysis of Polynomial Interpolation Characteristics Based on 6DOF Manipulator Trajectory Planning

Aiming at the problems of choosing and using polynomials of different times and mixed polynomials in the process of manipulator trajectory planning, such as difficulty, wide range of light and low accuracy, in this paper, the application of different polynomials in manipulator trajectory planning is systematically analyzed by using MATLAB Robot Toolbox simulation platform and Polynomial interpolation algorithm. Using a 6-dof robotic arm Puma560 as a simulation object, the effects of three-, five-, and seven-Polynomial interpolation on joint position and pose in joint space trajectory planning were analyzed, a preliminary understanding of the application of different polynomials, and secondly, the more complex application of the fifth and seventh degree polynomials in Descartes space for the trajectory planning of the manipulator, the influence of trajectory planning with different degree polynomials on the position and pose of manipulator joint is clarified. The results show that the higher the number of polynomials, the smoother the transition of the joint at the critical path points, but the greater the maximum angular velocity and angular acceleration of the joint, the Polynomial interpolation of joint space and Descartes space trajectory planning are 14% and 23% lower than the average of absolute maximum angular velocity and absolute maximum angular acceleration of the Polynomial interpolation, respectively. Based on the analysis of the research results, this paper presents the applicable conditions and applications of Polynomial interpolation method in trajectory planning, and further summarizes the application conditions and methods of mixed polynomials.

Motion Planning for Dynamic Three-Dimensional Manipulation for Unknown Flexible Linear Object

Generally, deformable objects have large and nonlinear deformations. Because of these characteristics, recognition and estimation of their movement are difficult. Many studies have been conducted aimed at manipulating deformable objects at will. However, they have been focused on situations wherein a rope’s properties are already known from prior experiments. In our previous work, we proposed a motion planning algorithm to manipulate unknown ropes using a robot arm. Our approach considered three steps: motion generation, manipulation, and parameter estimation. By repeating these three steps, a parameterized flexible linear object model that can express the actual rope movements was estimated, and manipulation was realized. However, our previous work was limited to 2D space manipulation. In this paper, we extend our previously proposed method to address casting manipulation in a 3D space. Casting manipulation involves targeting the flexible linear object tips at the desired object. While our previous studies focused solely on two-dimensional manipulation, this work examines the applicability of the same approach in 3D space. Moreover, 3D manipulation using an unknown flexible linear object has never been reported for dynamic manipulation with flexible linear objects. In this work, we show that our proposed method can be used for 3D manipulation.