Image Jacobian Research Articles

Image-Based Visual Servoing of Manipulators With Unknown Depth: A Recurrent Neural Network Approach.

The image-based visual servoing (IBVS) of manipulators is important for intelligent manipulation using visual feedbacks. While the traditional IBVS methods for manipulators require the knowledge of the depth information in the interaction matrix, in this article, we propose a novel IBVS method for manipulators without depth estimation by leveraging the property of the associated image Jacobian. Because of a novel transformation, the IBVS problem is converted into a convex optimization problem subject to the kinematic constraint, joint constraints, and other constraints that are not explicitly related to the depth information. The problem is then solved by developing a recurrent neural network of global asymptotic convergence, and a dynamic neural control law without depth estimation emerges for the IBVS of manipulators. The theoretical guarantee and simulation results are provided to show the efficacy of the proposed method.

Model-Free and Uncalibrated Eye-in-Hand Visual Servoing Approach for Concentric-Tube Robots

This article proposes a model-free and uncalibrated eye-in-hand visual servoing (EiH-VS) approach for controlling concentric-tube robots (CTRs) in minimally invasive surgery (MIS). Traditionally, a closed-loop EiH-VS controller requires an accurate robot kinematic model and hand–eye calibration. However, it is difficult to model CTR accurately when considering torsion, shear, friction, interactive force, and nonlinear constitutive behavior. In this article, we map image deviations to robot actuation variables with numerically calculated image Jacobian. The estimation of dynamic image Jacobian is based on a modified adaptive square-root unscented Kalman filter (MASR-UKF). A customized measurement matrix is constructed to describe the transformation between state and observation vectors. Moreover, an adaptive gain controller is designed to accelerate convergence. As a result, no prior knowledge of the CTR kinematic model and hand–eye calibration is needed in visual servoing tasks. Simulations and experiments have been conducted. The results validated the efficacy and efficiency of the proposed methods.

DIJE: Dense Image Jacobian Estimation and its Application to Visual Servo Control

DIJE: Dense Image Jacobian Estimation and its Application to Visual Servo Control

The Estimation of Image Jacobian Matrix with Time-Delay Compensation

Time delay exists in image-based visual servo system, which will have a certain impact on the system control. To solve the impact of time delay, the time delay compensation of the object feature point image and the image Jacobian matrix is discussed in this paper. Some work is done in this paper: The estimation of the object feature point image under time delay is based on a proposed robust decorrelation Kalman filtering model, for the measurement vectors which cannot be obtained during time delay in the robust Kalman filtering model, a polynomial fitting method is proposed in which the selection of the polynomial includes the position, velocity and acceleration of the object feature point which impact the feature point trajectory, then the more accurate object feature point image can be obtained. From the estimated object feature point image under time delay, the more accurate image Jacobian matrix under time delay can be obtained. Simulation and experimental results verify the feasibility and superiority of this paper method.

Uncalibrated Image-Based Visual Servoing Control with Maximum Correntropy Kalman Filter

Abstract A major challenge in solving robot visual servoing problems with unstructured environments is to obtain image Jacobian matrix, especially non-Gaussian noise always exist in the whole process. However, the standard Kalman Filter (KF) is exhausted to find the optimal value under Gaussian noise assumption. In this paper, a Kalman filter which adopted the maximum correntropy criterion (MCC) instead of the minimum mean square error (MMSE) criterion is proposed to solve the approximation issue of the image Jacobian. The simulation and experiment results using a conventional 6R manipulator are conducted to verify the effectiveness of the proposed method.

Sampling-Based Visual Path Planning Framework for a Multirotor UAV

This paper proposes a sampling-based visual path planning framework for the servoing of a multirotor unmanned aerial vehicle. There are two main problems of the traditional image-based visual servoing: one is that it cannot guarantee the bounded control input because of the high disparity between the initial and desired features, the other is that it could lead to slow convergence rates caused by exponential attenuation over time and insufficient control inputs in the final phase. To overcome these problems, this paper proposes a sampling-based visual path planning framework to generate control inputs using reference features over the interval between the initial and desired features. Our proposed contribution can generate small and bounded control inputs in the large pose difference environments. In detail, it can maintain a small bounded error task function because of preventing singularities, uncertainties, and local minimums during calculating image Jacobian. Therefore, our proposed contribution can not only overcome the deficiencies of traditional image-based visual servoing approaches but also improve control performance. A variety of simulation results conducted to verify the performance of the proposed framework indicate that it can overcome the vulnerabilities of traditional image-based visual servoing with respect to large disparity environment and generate relative body velocity commands.

Visual servoing framework using Gaussian process for an aerial parallel manipulator

This paper proposes a Gaussian process based visual servoing framework for an aerial parallel manipulator. Our aerial parallel manipulator utilizes the on-board eye-in-hand vision sensor system attached on the end-effector of three-degrees-of-freedom parallel manipulator. There are three major advantages: small, light in weight, and linearity with respect to the host vehicle rather than the serial manipulator, but it has a critical drawback that its workspace is too small to perform the mission itself during the hovering. In order to overcome the limited workspace problem and perform the mission more actively, proposed visual servoing framework is proposed to generate relative body velocity commands of the host vehicle by using the interpolated and extrapolated feature path between the initial and desired features to fed into the underactuated aerial parallel manipulator. It can generate not only numerical stable but also feasible control input. Furthermore, it can overcome the weakness of the traditional image-based visual servoing such as singularities, uncertainties, and local minimums during calculating image Jacobian under the large disparity environment between the target and the unmanned aerial vehicle. As a result of the proposed contribution, we show that our contribution is reliable to perform the picking-and-replacement autonomously, and it shows that it can be applied in the large displacement environments throughout the flight test.

Vibration Damping of Flexible Link Robots Using an Eye‐in‐Hand Camera

AbstractDue to their low weight robots with flexible links are ideal candidates for use in dynamically demanding applications. Since the main goal of the robot control is to achieve precise positioning, the damping of oscillations due to low stiffness is imperative. Furthermore classical PD motor position control is insufficient for vibration damping. This paper presents an approach using an eye‐in‐hand camera for vibration damping of an elastic articulated robot with three actuated joints. The idea is to use the position of feature points in the camera image for generating a feedback signal. The image feature position is transformed into the joint space using the image Jacobian. The PD motor control is augmented by a proportional term w.r.t. a virtual arm angle. Experimental results of a comparison with a control concept based on acceleration sensors are presented. (© 2017 Wiley‐VCH Verlag GmbH & Co. KGaA, Weinheim)

Image-Based Visual Servoing With Light Field Cameras

This paper proposes the first derivation, implementation, and experimental validation of light field image-based visual servoing. Light field image Jacobians are derived based on a compact light field feature representation that is close to the form measured directly by light field cameras. We also enhance feature detection and correspondence by enforcing light field geometry constraints, and directly estimate the image Jacobian without knowledge of point depth. The proposed approach is implemented over a standard visual servoing control loop, and applied to a custom-mirror-based light field camera mounted on a robotic arm. Light field image-based visual servoing is then validated in both simulation and experiment. We show that the proposed method outperforms conventional monocular and stereo image-based visual servoing under field-of-view constraints and occlusions.

CLFs-based optimization control for a class of constrained visual servoing systems

In this paper, we use the control Lyapunov function (CLF) technique to present an optimized visual servo control method for constrained eye-in-hand robot visual servoing systems. With the knowledge of camera intrinsic parameters and depth of target changes, visual servo control laws (i.e. translation speed) with adjustable parameters are derived by image point features and some known CLF of the visual servoing system. The Fibonacci method is employed to online compute the optimal value of those adjustable parameters, which yields an optimized control law to satisfy constraints of the visual servoing system. The Lyapunov's theorem and the properties of CLF are used to establish stability of the constrained visual servoing system in the closed-loop with the optimized control law. One merit of the presented method is that there is no requirement of online calculating the pseudo-inverse of the image Jacobian's matrix and the homography matrix. Simulation and experimental results illustrated the effectiveness of the method proposed here.

Image feature command generation of contour following tasks for SCARA robots employing Image-Based Visual Servoing—A PH-spline approach

Recently, visual servoing has been widely employed in industrial robots and has become an invaluable asset to enhance the functionality of the robot. However, the issue of image feature command generation in a visual servoing task receives little attention. In a contour following task that adopts Image-Based Visual Servoing (IBVS), it is crucial to perform motion planning on the desired image trajectory. Without proper motion planning, not only may the discrepancy between the target position and the current position on the image plane not converge, but also the flexibility of exploiting visual servoing for applications such as contour following will be limited. In order to cope with the aforementioned problem, this paper proposes a PH-spline based motion planning approach for systems that adopt IBVS. In particular, the exterior contour of an object is represented by a PH quantic spline. With proper acceleration/deceleration motion planning, a PH quantic spline interpolator is constructed to generate desired image feature commands so that IBVS can be applied to handle contour following problems of an object without a known geometric model. Furthermore, this paper also develops a depth estimation algorithm for the eye-to-hand camera structure, providing a convenient way to estimate the depth value that is essential in computing image Jacobian. Experimental results of several contour following tasks verify the effectiveness of the proposed approach. A PH-spline based motion planning approach for systems that adopt IBVS is proposed.Image feature commands are generated by a PH quantic spline interpolator.The PH quantic spline interpolator takes into account ACC/DEC motion planning and curvature.A depth estimation approach for IBVS in contour following applications is developed.The proposed approach is suitable for objects without geometric models.

Orthogonal Image Features for Visual Servoing of a 6-DOF Manipulator With Uncalibrated Stereo Cameras

We present an approach to control a 6-degree-of-freedom (DOF) manipulator using an uncalibrated visual servoing (VS) approach that addresses the challenges of choosing proper image features for target objects and designing a VS controller to enhance the tracking performance. The main contribution of this paper is the definition of a new virtual visual space (image space). A novel stereo camera model employing virtual orthogonal cameras is used to map 6-D poses from Cartesian space to this virtual visual space . Each component of the 6-D pose vector defined in this virtual visual space is linearly independent, leading to a full-rank $6\times 6$ image Jacobian matrix , which allows avoiding classical problems, such as image space singularities and local minima. Furthermore, the control for rotational and translational motion of robot is decoupled due to the diagonal image Jacobian. Finally, simulation results with an eye-to-hand robotic system confirm the improvement in controller stability and motion performance with respect to conventional VS approaches. Experimental results on a 6-DOF industrial robot are provided to illustrate the effectiveness of the proposed method and the feasibility of using this method in practical scenarios.

Robust uncalibrated visual servoing control based on disturbance observer

In this paper, an uncalibrated visual servoing scheme with optimal disturbance rejection performance is proposed based on disturbance observer (DOB). Comparing with traditional uncalibrated methods, which estimate online the hand-eye relationship characterized by varying image Jacobian, the uncertainty of image Jacobian is eliminated via DOB to approach a given nominal model in this paper. By solving a constrained optimization problem transformed into an H∞ control framework, the disturbance rejection performance is optimized while ensuring the robust stability of the closed-loop visual servoing system. The controller is based on the nominal image Jacobian matrix, thus avoiding singularities and local minima. Simulations and experiments show that the proposed scheme performs better in tracking an object than traditional algorithms. The disturbance and image noise rejection performance is verified.

Research on robot manipulator uncalibrated visual serving under two-camera bi-axial parallel vision configuration

This article presents a two-camera bi-axial parallel vision configuration to realize robot manipulator uncalibrated visual serving. The intrinsic and extrinsic parameters of the camera and the model of robot manipulator are not known, as well as without real-time computes the inverse of image Jacobian when image-based dynamic control of a robot manipulator in this article. First, the proposed vision configuration is applied to transform the Cartesian posture information into the point and angle parameters in the two image planes. After that, a qualitative mathematical model was established which analyzes the relationship between the feature point and line of the robot manipulator in the Cartesian space and the corresponding specific point and angle in the image plane. In addition, the monotonic property of the point and line features in the mathematic model has been proved. Then, the controller was designed based on image and realized the five-degrees of freedom of the robot manipulator uncalibrated visual serving control in the simulation platform. Moreover, the Lyapunov theory was used to prove the global asymptotic stability of proposed method in the image plane. Finally, the proposed method was compared with the method of image Jacobian matrix in the actual platform experiment, and the comparative experiment results show the effectiveness of the robot manipulator uncalibrated visual serving under the proposed vision configuration.

Robotic Adaptive Impedance Control Based On Visual Guidance

Abstract Uncalibrated visual servoing based on SVR-Jacobian estimator is proposed in unknown environment. Multiple support vector regression (SVR) machines are used to estimate the Jacobian matrix of images, and the nonlinear mapping between the image features of the curved line and the robot joint angle is constructed, uncalibrated robot impedance control can be carried out. Image Jacobian matrix expression with Gaussian kernel is put forward, the effectiveness of the presented approach is verified by using a 6 DOF robot with a CCD camera and a force/torque sensor installed in its end effector

Adaptive vision-based control of robot manipulators using the interpolating polynomial

This paper studies the problem of vision-based adaptive control for robot manipulators under a fixed camera configuration, when the camera intrinsic and extrinsic parameters are uncertain. An image-based controller, which requires the image Jacobian for its implementation, is conducted. The image Jacobian is inversely proportional to the depth factor. Hence it is not linearly parameterizable and the depth factor cannot be adapted with uncertain camera parameters. To cope with the inverse dependence of the image Jacobian on the depth factor, the controller employed the polynomial interpolation to parameterize the image Jacobian matrix linearly. The coefficients of the polynomial are obtained by a trained adaptive-network-based fuzzy inference system. A stability analysis for the proposed method is provided by a full consideration of the non-linear dynamics of the robot manipulator. Simulation results are presented to demonstrate the effectiveness of the proposed approach.

카메라 영상기반 전방향 이동 로봇의 제어

본 논문에서는 카메라를 탑재한 전방향 이동 로봇에서의 표적 추종을 위한 영상기반 시각 서보 제어를 다룬다. 기존 연구에서는 카메라 영상에서 추출한 표적의 영상 좌표로부터 표적 추종을 위한 바퀴의 회전 각속도를 구하기 위하여 카메라의 수학적 모델과 이동 로봇의 기구학 특성으로부터 구한 수학적 영상 자코비안을 널리 활용하였다. 본 논문에서는 표적의 영상 좌표 정보를 이용한 단순한 규칙기반 제어 방식과 영상에 포착된 표적의 크기 정보를 조합하여 바퀴의 회전 각속도를 생성하는 새로운 방식을 제안한다. 카메라 영상을 몇 개의 영역으로 분할하고, 표적이 포함된 영역에 따라 미리 정의한 규칙을 적용하는데, 복잡한 수학적 표현을 사용하지 않으면서도 비교적 적은 수의 규칙을 사용하므로 구현이 용이한 장점이 있다. 제안된 방식은 실제 시스템으로 구현하여 실험하고, 전체 실험 시스템에 대한 설명과 함께 실험 결과를 제시하여 제안하는 방식의 타당성을 입증한다. In this paper, an image-based visual servo control strategy for tracking a target object is applied to a camera-mounted omni-directional mobile robot. In order to get target angular velocity of each wheel from image coordinates of the target object, in general, a mathematical image Jacobian matrix is built using a camera model and a mobile robot kinematics. Unlike to the well-known mathematical image Jacobian, a simple rule-based control strategy is proposed to generate target angular velocities of the wheels in conjunction with size of the target object captured in a camera image. A camera image is divided into several regions, and a pre-defined rule corresponding to the target-located image region is applied to generate target angular velocities of wheels. The proposed algorithm is easily implementable in that no mathematical description for image Jacobian is required and a small number of rules are sufficient for target tracking. Experimental results are presented with descriptions about the overall experimental system.

A Simulation Research on the Visual Servo Based on Pseudo-Inverse of Image Jacobian Matrix for Robot

The image Jacobian Matrix must obtain during the course of uncalibrated visual servo for classic algorithms firstly. Then the inverse of image Jacobian Matrix or pseudo-inverse of image Jacobian Matrix can be taken. But when the inverse of image Jacobian Matrix is not exist or pseudo-inverse of image Jacobian Matrix is not easy to get, the uncalibrated visual servo for robot can not realize. In this paper, a research is carried on by simulation between the classic method for uncalibrared visual servo and the strategy by computing pseudo-inverse of image Jacobian Matrix. It is conclusion that the latter not only has advantage of the performance for tracking, but also reduces computational complexity for control.

Uncalibrated Visual Servoing through the Efficient Estimation of the Image Jacobian for Large Residual

An uncalibrated visual servo control method for tracking a target is presented. We define the robot-positioning problem as an unconstrained optimization problem to minimize the image error between the target feature and the robot end-effector feature. We propose a method to find the residual term for more precise modeling using the secant approximation method. The composite image Jacobian is estimated by the proper method for eye-to-hand configuration without knowledge of the kinematic structure, imaging geometry and intrinsic parameter of camera. This method is independent of the motion of a target feature. The algorithm for regulation of the joint velocity for safety and stability is presented using the cost function. Adaptive regulation for visibility constraints is proposed using the adaptive parameter.

Robotic grasping based on efficient tracking and visual servoing using local feature descriptors

In service robotic applications, grasping daily objects is an essential requirement. In this context, object and obstacle detection are used to find the desired object and to plan an obstacle-free path for a robot to successfully manipulate the object. In this paper, we propose a high-speed object tracking method based on a window approach and a local feature descriptor called speeded-up robust features (SURF). Instead of tracking the object in full image, we search and match features in the window of attention that contains only the object. Therefore, the tracked interest points are more repeatable and robust against noise. The visual servo controller uses geometrical features that are computed directly from the set of interest points, which makes the method robust against the loss of features caused by occlusion or changes in the viewpoint. Furthermore, these features decouple the translations and rotations from the image Jacobian, and also keep the object inside the camera’s field of view. Various experiments with a robotic arm equipped with a monocular eye-in-hand camera demonstrate that objects can be grasped safely and in a stable manner in a cluttered environment using the proposed method.