Uncalibrated Stereo Vision Research Articles

Uncalibrated stereo vision with deep learning for 6-DOF pose estimation for a robot arm system

This paper proposes a novel method for six degrees of freedom pose estimation of objects for the application of robot arm pick and place. It is based on the use of a stereo vision system, which does not require calibration. Using both cameras, four corner points of the object are detected. A deep-neural-network (DNN) is trained for the prediction of the 6 DOF pose of the object from the four detected corner points’ coordinates in each image of both cameras. The stereo vision used is a low-end vision system placed in a custom-made setup. Before the training phase of the DNN, the robot is set to auto collect data in a predefined workspace. This workspace is defined dependently on the spatial feasibility of the robot arm and the shared field of view of the stereo vision system. The collected data represent images of a 2D marker attached to the robot arm gripper. The 2D marker is used for data collection to ease the detection of the four corner points. The proposed method succeeds in estimating the six degrees of freedom pose of the object, without the need for the determination of neither the intrinsic nor the extrinsic parameters of the stereo vision system. The optimum design of the proposed DNN is obtained after comparing different activation functions and optimizers associated with the DNN.The proposed uncalibrated DNN-based method performance is compared to that of the traditional calibration-based method. In the calibration-based method, the rotational matrix relating the robot coordinates to the stereo vision coordinates is computed using two approaches. The first approach uses Singular Value Decomposition (SVD) while the second approach uses a novel proposed modification of particle swarm optimization (PSO) called Hyper particle Scouts optimization (HPSO). HPSO outperforms other metaheuristic optimization algorithms such as PSO and genetic algorithm (GA).Exhaustive tests are performed, and the proposed DNN-based method is shown to outperform all tested alternatives.

Direct 3D Information Determination in an Uncalibrated Stereovision System by Using Evolutionary Algorithms

This paper proposes a 3D panoramic shape reconstruction method based on an uncalibrated stereovision system (USS) composed of five cameras circularly located around the object to be analysed. First, some interesting points are detected from markers placed on the object such that they are visible by two successive cameras of the USS. These points are then matched on both images acquired by a couple of successive cameras. This process is repeated for all the couples of cameras. Second, by using an evolutionary algorithm, the depth values of the different interesting points are calculated. A comparison with a traditional method based on calibrated cameras validates the accuracy of 3D information provided by the proposed method. Finally, by combining all the interesting points, a panoramic view of the object is obtained.

Efficient hierarchical matching algorithm for processing uncalibrated stereo vision images and its hardware architecture

In motion estimation, the sub-pixel matching technique involves the search of sub-sample positions as well as integer-sample positions between the image pairs, choosing the one that gives the best match. Based on this idea, this work proposes an estimation algorithm, which performs a 2-D correspondence search using a hierarchical search pattern. The intermediate results are refined by 3-D cellular automata (CA). The disparity value is then defined using the distance of the matching position. Therefore the proposed algorithm can process uncalibrated and non-rectified stereo image pairs, maintaining the computational load within reasonable levels. Additionally, a hardware architecture of the algorithm is deployed. Its performance has been evaluated on both synthetic and real self-captured image sets. Its attributes, make the proposed method suitable for autonomous outdoor robotic applications.

Evolutionary-based 3D reconstruction using an uncalibrated stereovision system: application of building a panoramic object view

In this paper, we propose an original evolutionary-based method for 3D panoramic reconstruction from an uncalibrated stereovision system (USS). The USS is composed of five cameras located on an arc of a circle around the object to be analyzed. The main originality of this work concerns the process of the calculation of the 3D information. Actually, with our method, 3D coordinates are directly obtained without any prior estimation of the fundamental matrix. The method operates in two steps. Firstly, points of interest are detected in pairs of images acquired by two consecutive cameras of the USS are matched. And secondly, using evolutionary algorithms, we jointly compute the transformed matrix between the two images and the respective depth of the points of interest. The accuracy of the proposed method is validated through a comparison with the depth values obtained using a traditional method. In order to perform 3D panoramic object reconstruction, the process is repeated for all the pairs of consecutive cameras. The 3D points thus obtained throughout the successive steps of the process which correspond to the different points of interest, are combined in order to obtain a set of 3D points all around the analyzed object.

The method of virtual points for autonomous, image-based robot control

Robots require sensor information for autonomous manipulation in an unknown unstructured environment. The accuracy of the action depends directly on the calibration of the system. The visual-servoing approach allows manipulator actions independently of calibration errors and a world model. Up to now image-based visualservoing needs image features of the real scene. Here the method of virtual points, i.e. points without detectable image features in the real scene is introduced. With this method autonomous manipulator actions in an unknown environment including object-localization, image-based grasp-point determination, image-based dynamic obstacle avoidance and image-based manipulation tasks based on uncalibrated stereo vision system become possible.

What Tasks can be Performed with an Uncalibrated Stereo Vision System

This article studies the following question: “When is it possible to decide, on the basis of images of point-features observed by an imprecisely modeled two-camera stereo vision system, whether or not a prescribed robot positioning task has been precisely accomplished?” Results are shown for three camera model classes: injective cameras, weakly calibrated projective cameras, and uncalibrated projective cameras. In particular, given a weakly calibrated stereo pair, it is shown that a positioning task can be precisely accomplished if and only if the task specification is invariant to projective transformations. It is shown that injective and uncalibrated projective cameras can accomplish fewer tasks, but are still able to accomplish tasks involving point coincidences. The same formal framework is applied to the problem of determining the set of tasks which can be precisely accomplished with the well-known position-based control architecture. It is shown that, for any class of camera models, the set of tasks which can be precisely accomplished using a position-based control architecture is a subset of the complete set of tasks which can be decided on the set, but includes all positioning tasks based on point coincidences. Two ways of extending the idea of position-based control to accomplish more tasks are also presented.

Computational Vision at Yale

We present a brief introduction to the five articles that make up this special issue: Shock Graphs and Shape Matching, The Bas-Relief Ambiguity, Incremental Focus of Attention for Robust Vision Based Tracking, What Tasks can be Performed with an Uncalibrated Stereo Vision System, and Volumetric Deformation Analysis Using Mechanics-Based Data Fusion: Application in Cardiac Motion Recovery. Tjis introduction and accompanying articles provide a by no means exhaustive, but hopefully representative sampling of the computational vision at Yale University.

Automatic feature matching in uncalibrated stereo vision through the use of color

One difficult problem in stereo vision is how to establish correspondence between features extracted from a pair of images. The difficulty is due to ambiguities or inconsistencies of available information on images. In this paper, we invetigate stereo correspondence problem in the framework of color stereo vision. We propose the use of a matching consistence (MC) constraint in RGB color space and the generalized epipolar geometry to develop an automatic feature matching algorithm.

Visually guided grasping in unstructured environments

We present simple and robust algorithms which combine uncalibrated stereo vision and a robot manipulator to enable it locate, reach and grasp unmodelled objects in unstructured environments. In the first stage, an operator indicates the object to be grasped by simply pointing at it. Next, the vision system segments the indicated object from the background, and plans a suitable grasp strategy. Finally, the robotic arm reaches out towards the object and executes the grasp. Uncalibrated stereo vision allows the system to continue to operate in the presence of errors in the kinematics of the robot manipulator and unknown changes in the position, orientation and intrinsic parameters of the stereo cameras during operation.

Human-robot interface by pointing with uncalibrated stereo vision

Here we present the results of an investigation into the use of a pointing-based interface for robot guidance. The system requires no physical contact with the operator, but uses uncalibrated stereo vision with active contours to track the position and pointing direction of a hand in real time. With a ground plane constraint, it is possible to find the indicated position in the robot's workspace, by considering only two-dimensional collineations. Experimental and simulation data show that a resolution of within 1 cm can be achieved in a 40 cm workspace, allowing simple pick-and-place operations to be specified by finger pointing.