Image Jacobian Matrix Research Articles

Adaptive visual servoing control for uncalibrated robot manipulator with uncertain dead-zone constraint

This paper addresses the adaptive visual tracking control problem of an uncalibrated camera robot system with unknown dead-zone inputs. The uncertainties include camera extrinsic and intrinsic parameters, robot dynamic parameters, and feature depth parameters. The control achieves a better performance by establishing a smooth inverse model of the dead-zone input, which eliminates the nonlinear effect of the actuator constraint. A novel adaptive control scheme is presented which does not require a priori knowledge of the parameter intervals of dead-zone model, to update the parameter values online. Furthermore, adaptive laws are provided to estimate the uncertainties that exist both in robot and in camera models. Meanwhile, in order to avoid the singularity of the image Jacobian matrix, a projection algorithm is also introduced. Subsequently, a novel robust adaptive controller together with dead-zone compensation is established so that the tracking error image space converges to the origin. Simulations and experimental studies are conducted to verify the effectiveness of the proposed method.

Adaptive positioning of large-scale aircraft components using a simplified image-based visual servoing via online measurement of ball-head center

In digital alignment systems, ensuring fast and accurate alignment between the ball head on large aircraft components and the ball socket of the positioner is the key to aircraft assembly efficiency and quality. Image-based visual servoing is one of the most advanced techniques with high efficiency and positioning accuracy, but it does not account for measuring the ball-head position. Moreover, variable target positions, similar feature interference, and poor lighting conditions pose great challenges to the fast and robust visual measurement of ball-head position. Hence, an online visual measurement method is introduced to measure the absolute position of the ball head in real time. An adaptive region of interest extraction algorithm and an improved edge following method are introduced to extract the elliptical contour from the ball-head image, and ellipse detection is achieved through a simple triplet selection algorithm. A geometric model based on the perspective projection of the sphere is established for estimating the ball-head center. Then, the simplified image-based visual servoing method is developed by constantizing the image Jacobian matrix. Next, an adaptive positioning method of large-scale aircraft components with multiple ball heads is proposed from a holistic perspective. Finally, experiments show that the final positioning accuracy of each ball head in the X/Y plane is less than ± 0.05 mm, with an efficiency almost ten times that of the conventional method. After adaptive adjustment, the position and angle errors of the simulated component are reduced within 0.6 mm and 0.3°, respectively.

A visual servo reinforcement learning control of uncalibrated manipulators with multi-channel gain decision

A technology based on Kalman filtering method combined with multi-channel gain training reinforcement learning for uncalibrated camera visual servo tasks is proposed in this paper. First, a dynamic system with state variables formed from the elements of the image Jacobian matrix is constructed to describe the mapping relationship between two-dimensional images and three-dimensional poses. Kalman filter is used to estimate the state variables of the constructed system online. Next, the Jacobian matrix estimation and depth determination strategy gradient (DDPG) methods are combined to jointly train multi-channel gains by setting a reasonable segmented reward and punishment mechanism. Through training, a more effective gain decision can be obtained. The robustness of Kalman filtering to interference to a certain extent reduces the precise dependence of reinforcement learning models, thereby achieving higher robustness in intelligent visual servo control. Finally, the effectiveness and advantages of the Kalman-DDPG method have been demonstrated through simulation comparison and six-degree-of-freedom (DOF) uncalibrated manipulator physical experiments.

IHUVS: Infinite Homography-Based Uncalibrated Methodology for Robotic Visual Servoing

The construction of task functions in robotic manipulation is of paramount importance for uncalibrated visual servoing. The existing methods generally use image information as control variables and estimate the image Jacobian matrix online, thus possessing issues relating to convergence, and image Jacobian matrix singularities. Therefore, this work proposes a novel methodology dubbed infinite homography-based uncalibrated visual servoing (IHUVS), in which the visual control of the robot end-effector pose is decomposed into its rotational and translational components. The corresponding rotational controller designs the visual servoing task function using the relationship between the infinite homography matrix and rotation matrix, and employs the Kronecker product to derive linear equations for rotational control, as well as to conduct the associated task error analysis. Meanwhile, the translational controller utilizes Kalman filtering for online estimation of the Jacobian matrix that is required by the proportional control scheme. The robot end-effector motion in Cartesian space is generated via the IHUVS method, without knowing the camera's intrinsic parameters and the robot hand-eye relationship. A simulation analysis is carried out to assess the algorithm's numerical performance, while robotic visual servoing experiments are also conducted to verify the accuracy and efficacy of the proposed IHUVS method.

Robotic Inserting a Moving Object Using Visual-Based Control With Time-Delay Compensator

Tracking-and-inserting a moving peg using a robot manipulator is a challenging task in manufacturing. In the past decades, various visual-based methods have been proposed for robotic manipulating static targets, which usually ignore the time delay in robot command transmission and image processing. However, for tracking and inserting a moving peg, time delays cannot be overlooked because they can reduce the tracking performance and even cause manipulations to fail. In this paper, a robot visual-based control with a time-delay compensator is presented to solve the problem of inserting a moving peg. The time-delay compensator was designed using RBFNNs, a feedback compensator aimed at eliminating the tracking errors caused by the time delays. Thus, we could manipulate a moving object using a commercial industry robot, even with the time-variant delays in the control loop. Furthermore, the visual-based controller with the pseudo-inverse image Jacobian matrix was designed using a linearization model. Thus, the matrix could be efficiently updated using the model. In the experiment, we inserted a peg into a moving hole using an eye-in-hand robot with precision.

Visual Servo Control for Workspace Navigation of Nanorobot End-Effector Inside SEM

Despite of the promising advancements in the last decade, the majority of scanning electron microscope (SEM)-based nanomanipulation tasks remain manually performed. Even in automated tasks, human intervention is required, at least during the task preparatory stage, where both the object of interest and robot end-effector are adequately enclosed in the field of view (FOV) of SEM at moderate magnification. This paper proposes a fully automated visual servo control method for workspace navigation of nanorobot end-effector that actively maintains the end-effector position offset from the center of FOV during passive working scene zooming and translation operations. We also propose using the scaling image Jacobian matrix theory to adaptively establish the hand-eye relationship of the nanorobotic system at uncalibrated magnifications, without the need for hardware regulation. This proposed method for workspace navigation is applicable to almost all commercial nanomanipulation systems. To the authors’ best knowledge, there has been no dedicated research on this problem in the literature. Experiments show that the proposed method significantly improves workspace navigation efficiency by about two-thirds, even for a skilled operator. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Workspace navigation refers to the process of moving the FOV to visually enclose the object or feature of interest at a moderate magnification, which is prerequisite for initializing either an automated or manual nanomanipulation task. This process involves consecutive zooming of the SEM in and out, as well as translating the sample stage, which can cause the end-effector to move out of the FOV. Therefore, it is necessary to move the robot end-effector with visual assistance along workspace navigation to avoid unexpected collisions. Typically, skilled operators perform this process manually in almost all nanomanipulation tasks, which is tedious and time-consuming. Therefore, developing an automated visual servo control method for workspace navigation of end-effector will significantly contribute to the nanomanipulation community.

Data-Driven Model Predictive Control for Uncalibrated Visual Servoing

This paper addresses the image-based visual servoing (IBVS) control problem with an uncalibrated camera, unknown dynamics, and constraints. A novel data-driven uncalibrated IBVS (UIBVS) strategy is proposed, incorporated with the Koopman-based model predictive control (KMPC) algorithm and the adaptive robust Kalman filter (ARKF). First, to alleviate the need for calibration of the camera’s intrinsic and extrinsic parameters, the ARKF with an adaptive factor is utilized to estimate the image Jacobian matrix online, thereby eliminating the laborious camera calibration procedures and improving robustness against camera disturbances. Then, a data-driven MPC strategy is proposed, wherein the unknown nonlinear dynamic model is learned using the Koopman operator theory, resulting in a linear Koopman prediction model. Only input–output data are used to construct the prediction model, and hence, the proposed approach is robust against model uncertainties. Furthermore, with a symmetric quadratic cost function, the proposed approach solves the quadratic programming problem online, and visibility constraints as well as joint torque constraints are taken into account. As a result, the proposed KMPC scheme can be implemented in real time, and the UIBVS performance degradation which arises from the control torque constraints can be avoided. Simulations and comparisons for a 2-DOF robotic manipulator demonstrate the feasibility of the proposed approach. Simulation results further validate that the computation time of the proposed approach is comparable to the one of kinematic-based methods.

Large gear structure assembly method based on uncalibrated image visual servo guidance.

The reliability of the transmission system hinges significantly on the assembly quality of its main component, the large gear structures. However, the traditional approach of employing manual lifting presents a host of challenges, such as high assembly complexity and lowered efficiency, rendering the overall assembly process notably arduous. In this study, a large gear structure assembly method based on uncalibrated image visual servo guidance is proposed. Comprising three modules, the approach involves constructing a task function for projective homography, estimating the image Jacobian matrix, and designing an adaptive servo controller. This methodology facilitates the mapping of changes in gear images to the motion of the end-effector in the parallel mechanism. Consequently, the system dynamically guides the end-effector to achieve the required attitude adjustments in the gear assembly in response to changes in the image features. Experimental results demonstrate that the method proposed surpasses alternative approaches, simultaneously exhibiting a significant enhancement in assembly efficiency. The method has a wide application prospect in the field of automated assembly of large gear structures.

A novel multidimensional uncalibration method applied to six-axis manipulators.

This study proposes a multidimensional uncalibrated technique for tracking and grasping dynamic targets by a robotic arm in the eye-in-hand mode. This method avoids complex and cumbersome calibration processes, enabling machine vision tasks to be adaptively applied in a variety of complex environments, which solved the problem of traditional calibration methods being unstable in complex environments. The specific method used in this study is first, in the eye-in-hand mode, the robotic arm moves along the x, y, and z axes in sequence, and images are taken before and after each movement. Thereafter, the image Jacobian matrix is calculated from the three (or more) sets of images collected. Finally, the robotic arm converts the target coordinates in the real-time captured images by the camera into coordinates in the robotic arm coordinate system through the image Jacobian matrix and performs real-time tracking. This study tests the dynamic quasi-Newton method for estimating the Jacobian matrix and optimizes the initialization coupling problem using the orthogonal moving method. This optimization scheme significantly shortens the iteration process, making the uncalibrated technology more fully applied in the field of dynamic object tracking. In addition, this study proposes a servo control algorithm with predictive compensation to mitigate or even eliminate the systematic error caused by time delay in dynamic target tracking in robot visual servo systems.

A Digital Twin-Based Visual Servoing with Extreme Learning Machine and Differential Evolution

The technology of visual servoing, with the digital twin as its driving force, holds great promise and advantages for enhancing the flexibility and efficiency of smart manufacturing assembly and dispensing applications. The effective deployment of visual servoing is contingent upon the robust and accurate estimation of the vision-motion correlation. Network-based methodologies are frequently employed in visual servoing to approximate the mapping between 2D image feature errors and 3D velocities, offering promising avenues for improving the accuracy and reliability of visual servoing systems. These developments have the potential to fully leverage the capabilities of digital twin technology in the realm of smart manufacturing. However, obtaining sufficient training data for these methods is challenging, and thus improving model generalization to reduce data requirements is imperative. To address this issue, we offer a learning-based approach for estimating Jacobian matrices of visual servoing that organically combines an extreme learning machine (ELM) and a differential evolutionary algorithm (DE). In the first stage, the pseudoinverse of the image Jacobian matrix is approximated using the ELM, which solves the problems associated with traditional visual servoing and is resistant to outside influences such as image noise and mistakes in camera calibration. In the second stage, differential evolution is utilized to select input weights and hidden layer bias and to determine ELM’s output weights. Experimental results conducted on a digital twin operating platform for 4-DOF robot with an eye-in-hand configuration demonstrate better performance than classical visual servoing and traditional ELM-based visual servoing in various cases.

Image-based visual servoing of underwater vehicles for tracking a moving target using model predictive control with motion estimation

This paper introduces an image-based visual servoing (IBVS) target-tracking strategy for an underwater vehicle to track a moving target beneath the vehicle using a downward-facing camera. The relative position, orientation, and velocity of the moving target were estimated using a nonlinear unscented Kalman filter (UKF). Based on these estimated values, image Jacobian matrices with respect to the velocities of the vehicle and target were constructed. A nonlinear model predictive controller (MPC) was employed to generate the velocity commands for underwater vehicles by optimising the visual target trajectories predicted by the estimated image Jacobian matrix and the target velocity. To track the velocity commands, an adaptive neural network controller was employed considering the system uncertainties. Simulation tests were performed with a fully actuated underwater robot to verify the efficiency of the designed IBVS target-tracking strategy.

Image moments-based visual servoing control of bagged agricultural materials handling robot

Manual handling is less efficient and sometimes even hazardous to humans in many areas, for example, agriculture. Using robots in those areas not only avoids human contact with such dangerous agricultural materials but also improves working efficiency. The motion of a robot is controlled using a technique called visual servoing that uses feedback information extracted from a vision sensor. In this study, a visual servoing method was proposed based on learning features and image moments for 3D targets to solve the problem of image moment-based visual servoing. A Gaussian process regression model was used to map the relationship between the image moment invariants and the rotational angles around the X- and Y-axes of the camera frame (denoted as γ and β). To obtain maximal decoupled structure and minimal nonlinearities of the image Jacobian matrix, it was assumed two image moment features, which are linearly proportional to γ and β. In addition to the other four standardized image moment features, a 6-DOF image moment-based visual servoing controller for the agricultural material handling robot was designed. Using this method, the problem of visual servoing task failure due to the singularity of the Jacobian matrix was solved, and it also had a better convergence effect for the part of the target image beyond the field of view and large displacement visual servoing system. The proposed algorithm was validated by carrying out experiments tracking bagged flour in a six-degree-of-freedom robotic system. The final displacement positioning accuracy was accurate to the millimeter level and the direction angle positioning accuracy was accurate to the 0.1 level. The method still has a certain convergence effect when the target image is beyond the field of view. The experimental results have been presented to show the adequate behavior of the presented approach in robot handling operations. It provides reference for the application of visual servoing technology in the field of agricultural robots and has important theoretical significance and practical value. Keywords: image moments, visual servoing, Jacobian matrix, agricultural material handling robot DOI: 10.25165/j.ijabe.20231601.7050 Citation: Li H, Zuo Z J, Chi R Q, Du Y F, Mao E R. Image moments-based visual servoing control of bagged agricultural materials handling robot. Int J Agric & Biol Eng, 2023; 16(1): 212–219.

Zooming-Free Hand-Eye Self-Calibration for Nanorobotic Manipulation Inside SEM

Nanorobotic manipulation inside Scanning Electron Microscope (SEM) is widely used in nanomaterial characterization and assembly tasks. However, tedious and time-consuming hardware regulations, like axis alignment, are still critical and mandatory to keep the functionality of nanorobot systems. This paper proposes a novel zooming-free hand-eye self-calibration method for nanorobotic manipulation inside SEM to circumvent the hardware regulation problem, which is applicable to a wide range of system configurations. The hand-eye relationship calibration method is based on the active incremental motion of the nanorobot without referring to expensive and complicated calibration patterns. Both perspective projection at low magnification and affine projection at high magnification are modelled, and unified by the image Jacobian matrix technique. With several times active calibration at different magnification factors in advance, the hand-eye relationship can be, thereafter, determined from the model at arbitrary status. The calibration model can be applied to both accurate SEM visual measurement and precision visual servo control. The corresponding methods regarding two-dimensional (2D) measurement, in-plane positioning, and constrained three-dimension (3D) positioning based on the proposed hand-eye relationship model are also presented in this paper. Experiments show that practical contact between robot end-effector and silicon nanowire under 1500× is well established with standard deviation less than one pixel.

DESIGN AND EXPERIMENTAL RESEARCH OF HAND–EYE COORDINATION CONTROL SYSTEM FOR ACUPUNCTURE ROBOT

Aiming at the problems of the acupuncture robot’s difficulty in positioning acupoints and poor acupuncture accuracy, a hand–eye coordination control method is proposed in this paper. First, according to the specificity of acupuncture, a label-based acupoint positioning method was designed, and then the task of acupuncture acupoints was transformed into acupuncture with specific shapes of labels. Second, the vision subsystem was designed from detection, to label contour matching, to multi-label recognition based on tracking algorithm. Last, according to the control signal amount processed by the vision subsystem, the image Jacobian matrix, and the strategy of monocular hierarchical control, the main hand–eye coordination problem of the fixed-point needling in the acupuncture task is completed. The experimental results show that the method is used for acupuncture and acupuncture puncture with better puncture effect and higher positioning accuracy, which can meet the needs of acupuncture tasks.

Hybrid regression model via multivariate adaptive regression spline and online sequential extreme learning machine and its application in vision servo system

To solve the problems of slow convergence speed, poor robustness, and complex calculation of image Jacobian matrix in image-based visual servo system, a hybrid regression model based on multiple adaptive regression spline and online sequential extreme learning machine is proposed to predict the product of pseudo inverse of image Jacobian matrix and image feature error and online sequential extreme learning machine is proposed to predict the product of pseudo inverse of image Jacobian matrix and image feature error. In MOS-ELM, MARS is used to evaluate the importance of input features and select specific features as the input features of online sequential extreme learning machine, so as to obtain better generalization performance and increase the stability of regression model. Finally, the method is applied to the speed predictive control of the manipulator end effector controlled by image-based visual servo and the prediction of machine learning data sets. Experimental results show that the algorithm has high prediction accuracy on machine learning data sets and good control performance in image-based visual servo.

Design of UAV target tracking controller based on visual servo

The task of tracking a moving target by the UAV has two major challenges: one is that the multi-angle and multi-scale changes caused by the target’s movement make the target detection more difficult, the other is that the stability and accurate of target tracking are difficult to guarantee. In order to solve these problems, using monocular vision to get the image information of the target and YOLOv4 is used to detect multi-angle and multi-scale moving targets. Then based on the image visual servo control method, the PID controller is designed, and the deviation between the target pixel and the ideal pixel is used as the input, the output of the controller is the desired pixel speed of the UAV in image coordinate system. the tracking speed of the UAV under the machine system is solved according to the image Jacobian matrix, and a tracking strategy is designed to improve the tracking accurate and stability. In the process of target tracking, the difference between the x-axis and Y-axis coordinates of the target center point and the image center point is less than 30 pixels. The results show that the proposed algorithm can keep the target in the center of the image.

The Estimation of Image Jacobian Matrix with Time-Delay Compensation

Time delay exists in image-based visual servo system, which will have a certain impact on the system control. To solve the impact of time delay, the time delay compensation of the object feature point image and the image Jacobian matrix is discussed in this paper. Some work is done in this paper: The estimation of the object feature point image under time delay is based on a proposed robust decorrelation Kalman filtering model, for the measurement vectors which cannot be obtained during time delay in the robust Kalman filtering model, a polynomial fitting method is proposed in which the selection of the polynomial includes the position, velocity and acceleration of the object feature point which impact the feature point trajectory, then the more accurate object feature point image can be obtained. From the estimated object feature point image under time delay, the more accurate image Jacobian matrix under time delay can be obtained. Simulation and experimental results verify the feasibility and superiority of this paper method.

Vision servo positioning control of robot manipulator for distribution line insulation wrapping

Traditional visual servo control based on location needs to convert two-dimensional information into three-dimensional space information, and the conversion process is subject to many interference factors, leading to large control errors. Aiming at the above problems, the visual servo positioning control of distribution line insulated wrapping robot and manipulator is studied. The position and pose of the manipulator are described and homogeneous transformed so as to calibrate the camera vision system of the manipulator. The image Jacobian matrix is established to realize the visual servo positioning control of the manipulator. Compared with the traditional control method, the experimental verification shows that the control method can reduce the pose error by at least 39.7%, and the control effect is more stable and the reliability is better.

A Visual Servo Control Method for Tomato Cluster-Picking Manipulators Based on a T-S Fuzzy Neural Network

HighlightsA T-S fuzzy neural network was applied to the visual servo control system of a tomato picking manipulator.The T-S fuzzy neural network structure was designed, and collected data were used to train the neural network model.A visual servo control system for the picking manipulator based on the neural network was designed and tested.The T-S fuzzy neural network was superior to a BP neural network in visual servo control of the picking manipulator.Abstract. To reduce the computational load of image Jacobian matrix estimation and to avoid the appearance of singularity of a Jacobian matrix in the visual servo control of a picking manipulator, a T-S fuzzy neural network algorithm is proposed to replace the image Jacobian matrix. This better fits the hand-eye relationship by combining the knowledge structure of fuzzy reasoning with the self-learning ability of a neural network. The T-S fuzzy neural network was trained and tested by collecting the variation data of image features and joint angles; after training, the T-S fuzzy neural network was used to predict the joint angles of the picking manipulator. Simulation results show that the square sum of training errors and testing errors were 0.017 and 0.032, respectively, after training the T-S fuzzy neural network. A T-S fuzzy neural network controller was applied to the visual servo system of the picking robot, and the test results show that the average difference between the end-effector and the ultimate target location of the visual servo system based on the T-S fuzzy neural network controller was 0.0037 m, which was 79.44% less than that of the visual servo system based on a BP neural network. The final average error of image features was between 0.52 and 3.25 pixels, which was 74.932% less than that of the visual servo system based on the BP neural network. Keywords: Picking manipulator, Tomato clusters, T-S fuzzy neural network, Visual servoing.

Research on Aerial Robot Based on Visual Servo

Towards target grasping by an aerial manipulator, an aerial manipulator system is presented, for which a separated control strategy is adopted. In this paper, a suitable adaptive visual servo controller is proposed based on the original aerial manipulator system, and a visual servo controller is designed based on the depth independent image Jacobian matrix. Thus, the grasping motion of the manipulator is controlled by the visual servo of the monocular camera, which can greatly enhance the functionality and speed of the aerial manipulator system. Furthermore, a unique single degree of freedom joint collocation camera is designed to enable the camera to lock the target object at any time during the motion of aircraft. Finally, this paper designs a unique state machine system, which can make the control system switch into any state.