Uncalibrated Environments Research Articles

Resilient adaptive trajectory tracking control for uncalibrated visual servoing systems with unknown actuator failures

A resilient adaptive tracking control scheme for uncalibrated visual servoing system with unknown actuator failures is proposed, where the camera parameters, the robotic physical parameters, and the position information of end-effector cannot be measured accurately. For our scheme, the nonlinear composite Jacobian matrix involving unknown parameters is estimated by an adaptive algorithm based on the depth-independent interaction matrix frame so that the image feature with time-varying depth can be tackled effectively. With the use of projection technique in the adaptive algorithm, the possible singularity of the estimated composite Jacobian matrix can be well circumvented without any stringent assumption, which is critical for establishing the closed-loop stability. Moreover, a novel adaptive resilient control scheme based on dynamic decoupling is developed to separate the design of visual servoing controller and fault compensation, improving the visual servoing system resilience against unknown actuator failures in uncalibrated environment. Through a rigorous Lyapunov argument, the asymptotic convergence of image tracking error to zero is established successfully regardless of the occurrence of unknown actuator failures. The effectiveness of the proposed visual tracking control scheme is confirmed by simulation results.

DeforMoBot: A Bio-Inspired Deformable Mobile Robot for Navigation Among Obstacles

Many animals can move in cluttered environments by conforming their body shape to geometric constraints in their surroundings such as narrow gaps. Most robots are rigid structures and do not possess these capabilities. Navigation around movable or compliant obstacles results in a loss of efficiency—and possible mission failure—compared to progression through them. In this letter, we propose the novel design of a deformable mobile robot; it can adopt a wider stance for greater stability (and possible higher payload capacity), or a narrower stance to become capable of fitting through small gaps and progressing through flexible obstacles. We use a whisker-based feedback control approach to match the amount of the robot's deformation with the compliance level of the obstacle. We present a real-time algorithm which uses whisker feedback and performs shape adjustment in uncalibrated environments. The developed robot was tested navigating among obstacles with varying physical properties from different approach angles. Our results highlight the importance of co-development of environment perception and physical reaction capabilities for improved performance of mobile robots in unstructured environments.

Adaptive fault-tolerant visual control of robot manipulators using an uncalibrated camera

In this paper, we propose an adaptive fault-tolerant visual control scheme for robotic manipulators with possible actuator failures in an uncalibrated environment. Most existing visual control approaches for robot system do not take into account actuator failures, which may prominently affect the transient performance of the system in practice. In order to moderate the detrimental and adverse effects of actuator failures on the system, a new adaptive algorithm is proposed to compensate stuck-type failures occurred in actuators. Moreover, by proposing a decoupling method, the uncertain parameter model of actuator failure is successfully separated from the dynamics model while the two models are coupled in most existing results [e.g., (Rugthum and Tao in Robotica 34(7):1529–1552, 2016; Rugthum in: International conference on engineering, applied sciences, and technology (ICEAST), pp 1–4, 2018)]. And the stability of the dynamic system and the convergence of the image error are proved by the Lyapunov analysis method. Finally, the effectiveness of the proposed control scheme is verified by comparing and analysing the tracking performance of the 3-DOF manipulator under different failure parameters.

Impedance estimation for robot contact with uncalibrated environments

This paper examines the problem of online estimation of the environment impedance parameters and position during the contact of a robot with an uncalibrated environment. Position controllers are commonly used for free motion, while force controllers are used for constrained motion of robots. However, it is not trivial to set controller parameters for practical contact tasks with unknown environments. In particular, the environmental position is uncalibrated, and it is impossible to determine the contact position through force sensor information because of the presence of a force measurement threshold. To this end, an estimation algorithm based on L-BFGS is proposed to obtain the environmental impedance parameters and position. The estimated environment position is updated with the impedance parameters, and the impedance parameter estimation is directly related to the penetration position. The penetration position is determined by the real position of manipulators and environment surface position. The estimated environmental parameters are used to modify the position and force switching controller parameters; therefore, the accuracy of these parameters can be evaluated by the performance of the modified controller. Simulations and experiments are successfully conducted and validated with the proposed impedance estimation method. The accuracy and effectiveness of the algorithm is verified through the force and trajectory tracking performance in simulations and experiments.

Calibration-Free Image-Based Trajectory Tracking Control of Mobile Robots With an Overhead Camera

To make the controller implementation easier and to enhance the system robustness and control performance in the presence of the camera parameter uncertainties, it is very desired to develop vision-based control approaches without any offline or online camera calibration. In this article, we propose a new calibration-free image-based trajectory tracking control scheme for nonholonomic mobile robots with a truly uncalibrated fixed camera. By developing a novel camera-parameter-independent kinematic model, both offline and online camera calibration can be avoided in the proposed scheme, and any knowledge of the camera is not needed in the controller design. The proposed trajectory tracking control scheme can guarantee exponential convergence of the image position and velocity tracking errors. To illustrate the performance of the proposed scheme, experimental results are provided in this article. Note to Practitioners —This article was motivated by the vision-based motion control problem of mobile robots in uncalibrated environments. Existing vision-based motion control approaches for nonholonomic mobile robots generally depend on offline precise/coarse or online numerical/adaptive calibration of the camera intrinsic and extrinsic parameters and require precise or coarse knowledge of the camera in their implementation. This article presents a novel calibration-free image-based trajectory tracking control scheme, which can be implemented easily in real environments without any offline or online calibration of the camera parameters and can be used to efficiently control the motion of nonholonomic mobile robots with an arbitrarily placed and truly unknown overhead camera. Experimental results show that the proposed scheme can achieve satisfactory trajectory tracking control performance despite the lack of any knowledge about the camera intrinsic and extrinsic parameters and the presence of unknown camera lens distortions, and hence, can provide a simple but efficient solution to the vision-based motion control problem of nonholonomic mobile robots.

Model Predictive Control for Constrained Image‐Based Visual Servoing in Uncalibrated Environments

AbstractThis paper presents a novel approach for image‐based visual servoing (IBVS) of a robotic system by considering the constraints in the case when the camera intrinsic and extrinsic parameters are uncalibrated and the position parameters of the features in 3‐D space are unknown. Based on the model predictive control method, the robotic system's input and output constraints, such as visibility constraints and actuators limitations, can be explicitly taken into account. Most of the constrained IBVS controllers use the traditional image Jacobian matrix, the proposed IBVS scheme is developed by using the depth‐independent interaction matrix. The unknown parameters can appear linearly in the prediction model and they can be estimated by the identification algorithm effectively. In addition, the model predictive control determines the optimal control input and updates the estimated parameters together with the prediction model. The proposed approach can simultaneously handle system constraints, unknown camera parameters and depth parameters. Both the visual positioning and tracking tasks can be achieved desired performances. Simulation results based on a 2‐DOF planar robot manipulator for both the eye‐in‐hand and eye‐to‐hand camera configurations are used to demonstrate the effectiveness of the proposed method.

Robots visual servo control with features constraint employing Kalman-neural-network filtering scheme

This paper presents an image-based servo control approach with a Kalman-neural-network filtering scheme for robots manipulation in uncalibrated environment. The image Jacobian on-line identification problems are firstly addressed by introducing the state estimation techniques, which have been incorporated neural network assists Kalman filtering (NNAKF). In fact, this is, the neural network (NN) can serve to play exactly the role of the error estimator, has the task of compensate the errors of Kalman filtering (KF). Then, by employing the NNAKF scheme, the proposed image-based servo control approach has guaranteed the robustness with respect to destabilized system attached dynamic noises, as well as the image features are constrained in field-of-view (FOV) of the camera. Furthermore, it is without requiring the intrinsic and extrinsic parameters of the camera during visual servoing tasks. To demonstrate further the validity and practicality of proposed approach, various simulation and experimental results have been presented using a six-degree-of-freedom robotic manipulator with eye-in-hand configurations.

Robustness analysis of uncalibrated eye‐in‐hand visual servo system in the presence of parametric uncertainty

PurposeThe purpose of this paper is to present the methodology to the robust stability analysis of a vision‐based control loop in an uncalibrated environment. The type of uncertainties considered is the parametric uncertainties. The approach adopted in this paper utilizes quadratic Lyapunov function to determine the composite Jacobian matrix and ensures the robust stability using linear matrix inequality (LMI) optimization. The effectiveness of the proposed approach can be witnessed by applying it to two‐link robotic manipulator with the camera mounted on the end‐effector.Design/methodology/approachThe objective of this research is the analysis of uncertain nonlinear system by representing it in differential‐algebraic form. By invoking the suitable system representation and Lyapunov analysis, the stability conditions are described in terms of linear matrix inequalities.FindingsThe proposed method is proved robust in the presence of parametric uncertainties.Originality/valueThrough a differential‐algebraic equation, LMI conditions are devised that ensure the stability of the uncertain system while providing an estimate of the domain of attraction based upon quadratic Lyapunov function.

Visual tracking of robots in uncalibrated environments

This paper presents a new adaptive controller for visual tracking control of a robot manipulator in 3D general motion with a fixed camera whose intrinsic and extrinsic parameters are uncalibrated. In addition to camera parameters, the feature positions are also assumed unknown. Based on the fact that the unknown parameters appears linearly in the closed-loop dynamics of the system if the depth-independent interaction matrix is adopted to map the image errors onto the joint space of the manipulator, a new adaptive algorithm was developed to estimated the unknown parameters on-line. With a full consideration of dynamic responses of the robot manipulator, the Lyapunov method is employed to prove asymptotic convergence of the image errors. Simulation and experiment results are used to demonstrate the performance of the proposed approach.

Visual servoing of robots with uncalibrated robot and camera parameters

This paper presents a new controller to regulate a set of feature points on the image plane to desired positions by controlling motion of a robot manipulator in uncalibrated environments. This controller is designed to cope with both the unknown camera parameters and the unknown robot parameters. The controller employs the depth-independent image Jacobian matrix to map the image errors onto the joint space of the manipulator. Based on the fact that the unknown camera and robot parameters appear linearly in the closed-loop dynamics of the system if the depth-independent image Jacobian is used, an adaptive algorithm was developed to estimate the unknown parameters on-line. With a full consideration of dynamic responses of the robot manipulator, the convergence of the image errors on the image plane to zero was proved by the Lyapunov method. Simulation and experiments have been conducted to demonstrate good convergence of the trajectory errors under the control of the proposed method.

Dynamic visual servoing of a small scale autonomous helicopter in uncalibrated environments

This paper presents a novel adaptive controller for image-based visual servoing of a small autonomous helicopter to cope with uncalibrated camera parameters and unknown 3D geometry of the feature points. The controller is based on the back-stepping technique, but its design has two new features. First, it incorporates the visual feedback into the last step of the backstepping procedure, while existing backsteppingbased methods employ the visual feedback at the early steps. Second, the controller maps the image errors onto the actuator space via a depth-independent interaction matrix to avoid estimation the depths of the feature points. The new design method makes it possible to linearly parameterize the closed-loop dynamics by the unknown camera parameters and coordinates of the feature points in the 3D space so that an adaptive algorithm can be developed to estimate the unknown parameters and coordinates on-line. Two potential functions are introduced in the controller to guarantee convergence of the image errors and to avoid trivial solutions of the estimated parameters. The Lyapunov method is used to prove the asymptotic stability of the proposed controller based on the nonlinear dynamics of the helicopter. Simulations have been also conducted to demonstrate the performance of the proposed method.

Uncalibrated eye‐in‐hand visual servoing: an LMI approach

PurposeThe purpose of this paper is to present the control of six degrees of freedom (PUMA560) robotic arm using visual servoing, based upon linear matrix inequality (LMI). The aim lies in developing such a method that neither involves camera calibration parameters nor inverse kinematics. The approach adopted in this paper includes transpose Jacobian control; thus, inverse of the Jacobian matrix is no longer required. By invoking the Lyapunov's direct method, closed‐loop stability of the system is ensured. Simulation results are shown for three different cases, which exhibit the system stability and convergence even in the presence of large errors.Design/methodology/approachThe paper presents LMI‐based visual servo control of PUMA560 robotic arm.FindingsThe proposed method is implementable in the dynamic environment due to its independence to camera and object model.Research limitations/implicationsVisibility constraint is not included during servoing – this may cause features to leave the camera field of view (fov).Originality/valueLMI optimization is employed for visual servo control in an uncalibrated environment. Lyapunov's direct method is utilized which ensures system stability and convergence.

Binocular vision-based 3-D trajectory following for autonomous robotic manipulation

SUMMARYRobotic manipulators that have interacted with uncalibrated environments typically have limited positioning and tracking capabilities, if control tasks cannot be appropriately encoded using available features in the environments. Specifically, to perform 3-D trajectory following operations employing binocular vision, it seems necessary to have a priori knowledge on pointwise correspondence information between two image planes. However, such an assumption cannot be made for any smooth 3-D trajectories. This paper describes how one might enhance autonomous robotic manipulation for 3-D trajectory following tasks using eye-to-hand binocular visual servoing. Based on a novel encoded error, an image-based feedback control law is proposed without assuming pointwise binocular correspondence information. The proposed control approach can guarantee task precision by employing only an approximately calibrated binocular vision system. The goal of the autonomous task is to drive a tool mounted on the end-effector of the robotic manipulator to follow a visually determined smooth 3-D target trajectory in desired speed with precision. The proposed control architecture is suitable for applications that require precise 3-D positioning and tracking in unknown environments. Our approach is successfully validated in a real task environment by performing experiments with an industrial robotic manipulator.

Hybrid Force and Vision-Based Contour Following of Planar Robots

Manipulators interacted with uncalibrated environments have limited dexterity due to constraints imposed by unknown environments. However, to perform science or industrial operations, it is necessary to be able to position and orient these manipulators on targets in order to accomplish required control tasks. This article describes how one might enhance manipulator dexterity for planar contour following tasks using hybrid force and vision-based control. The proposed control approach can guarantee task precision employing only a single-axis force sensor and an imprecisely calibrated CCD camera whose optical axis is perpendicular to the planar workspace. The goal of the autonomous task is to drive an instrument mounted on the end-effector of a planar robotic manipulator to follow a visually determined planar contour and continue tracking the contour in desired pose, contact force, and speed, all demanding time-varying, with precision. The proposed control architecture is suitable for applications that require simultaneous force and pose control in unknown environments. Our approach is successfully validated in a real task environment by performing experiments with an industrial robotic manipulator.

Extracting 3D facial animation parameters from multiview video clips

We propose an accurate and inexpensive procedure that estimates 3D facial motion parameters from mirror-reflected multiview video clips. We place two planar mirrors near a subject's cheeks and use a single camera to simultaneously capture a marker's front and side view images. We also propose a novel closed-form linear algorithm to reconstruct 3D positions from real versus mirrored point correspondences in an uncalibrated environment. Our computer simulations reveal that exploiting mirrors' various reflective properties yields a more robust, accurate, and simpler 3D position estimation approach than general-purpose stereo vision methods that use a linear approach or maximum-likelihood optimization. Our experiments show a root mean square (RMS) error of less than 2 mm in 3D space with only 20-point correspondences. For semiautomatic 3D motion tracking, we use an adaptive Kalman predictor and filter to improve stability and infer the occluded markers' position. Our approach tracks more than 50 markers on a subject's face and lips from 30-frame-per-second video clips. We've applied the facial motion parameters estimated from the proposed method to our facial animation system.

Sensor-based hybrid position/force control of a robot manipulator in an uncalibrated environment

The paper is devoted to the problem of controlling a robot manipulator for a class of constrained motions. The task under consideration is to control the manipulator, such that the end-effector follows a path on an unknown surface, with the aid of a single camera assumed to be uncalibrated with respect to the robot coordinates. To accomplish a task of this kind, we propose a new control strategy based on multisensor fusion. We assume that three different sensors-that is, encoders mounted at each joint of the robot with six degrees of freedom, a force-torque sensor mounted at the wrist of the manipulator, and a visual sensor with a single camera fixed to the ceiling of the workcell-are available. Also, we assume that the contact point between the tool grasped by the end-effector and the surface is frictionless. To describe the proposed algorithm that we have implemented, first we decouple the vector space of control variables into two subspaces. We use one for controlling the magnitude of the contact force on the surface and the other for controlling the constrained motion on the surface. This way, the control synthesis problem is decoupled and we are able to develop a new scheme that utilizes sensor fusion to handle uncalibrated parameters in the workcell, wherein the surface on which the task is to be performed is assumed to be visible, but has an a priori unknown position.

Robotic motion planning and manipulation in an uncalibrated environment

The authors analyze planning and control problems in manipulation in an uncalibrated environment consisting of a PUMA 560 robotic manipulator, a rotating turntable equipped with an encoder and a CCD camera based vision sensor fixed permanently on the ceiling. It is assumed that a part with a known shape but unknown orientation is placed on the turntable which is rotating with an unknown motion dynamics. Furthermore, the calibration parameters are a priori assumed to be unknown. The objective is to track the rotating part with an a priori specified relative orientation. The task considered is of importance in various problems concerning industrial automation, such as part-feeding and tool-changing.

Eye-in-hand robotic tasks in uncalibrated environments

Flexible operation of a robotic agent in an uncalibrated environment requires the ability to recover unknown or partially known parameters of the workspace through sensing. Of the sensors available to a robotic agent, visual sensors provide information that is richer and more complete than other sensors. In this paper we present robust techniques for the derivation of depth from feature points on a target's surface and for the accurate and high-speed tracking of moving targets. We use these techniques in a system that operates with little or no a priori knowledge of object- and camera-related parameters to robustly determine such object-related parameters as velocity and depth. Such determination of extrinsic environmental parameters is essential for performing higher level tasks such as inspection, exploration, tracking, grasping, and collision-free motion planning. For both applications, we use the Minnesota robotic visual tracker (MRVT) (a single visual sensor mounted on the end-effector of a robotic manipulator combined with a real-time vision system) to automatically select feature points on surfaces, to derive an estimate of the environmental parameter in question, and to supply a control vector based upon these estimates to guide the manipulator.