Online Camera Calibration Research Articles

ROLL-SENSITIVE ONLINE CAMERA ORIENTATION DETERMINATION ON THE STRUCTURED ROAD

Abstract. Online camera calibration technology can estimate the pose of the camera onboard in real time, playing an important role in many fields such as HD map production and autonomous vehicles. Some researchers use one vanishing point (VP) to calculate the pitch and yaw angle of the onboard camera. However, this method assumes that the roll angle is zero, which is impractical because of the inevitable installation error. This paper proposes a novel online camera orientation determination method based on a longitudinal vanishing point without the zero-roll hypothesis. The orientation of the camera is determined in two steps: calculating the pitch and yaw angles according to vanishing point theory, and then obtaining the roll angle with lane widths constraint which is modeled as an optimization problem. To verify the effectiveness of our algorithm, we evaluated it on the nuScenes dataset. As a result, the rotation error of the roll and pitch angle can achieve 0.154° and 0.116° respectively. Also, we deployed our method in the “Tuyou”, an autonomous vehicle developed by Wuhan University, and then tested it in the urban structured road. Our proposed method can reconstruct the ground space accurately compared with previous methods with zero-roll hypothesis.

Purely Image-Based Pose Stabilization of Nonholonomic Mobile Robots With a Truly Uncalibrated Overhead Camera

Although many vision-based control methods have been proposed for nonholonomic mobile robots, in their implementation, it is usually necessary to calibrate the camera intrinsic and/or extrinsic parameters using offline/online parameter estimation algorithms or online adaptation laws. To avoid the tediousness of camera calibration and to make the system performance highly robust to camera parameter uncertainties, in this article, we propose novel image-based pose stabilization control approaches for nonholonomic mobile robots with a truly uncalibrated overhead fixed camera. In the proposed approaches, only image position information of three feature points from an overhead camera is used for controller design, while information from other sensors (such as wheel encoders) is not required. Furthermore, either offline or online camera calibration is not necessary, and no knowledge about the camera intrinsic and extrinsic parameters is needed, which also can greatly simplify the controller implementation. Simulation and experimental results are given to demonstrate the feasibility and effectiveness of the proposed purely image-based pose stabilization approaches.

Calibration-Free Image-Based Trajectory Tracking Control of Mobile Robots With an Overhead Camera

To make the controller implementation easier and to enhance the system robustness and control performance in the presence of the camera parameter uncertainties, it is very desired to develop vision-based control approaches without any offline or online camera calibration. In this article, we propose a new calibration-free image-based trajectory tracking control scheme for nonholonomic mobile robots with a truly uncalibrated fixed camera. By developing a novel camera-parameter-independent kinematic model, both offline and online camera calibration can be avoided in the proposed scheme, and any knowledge of the camera is not needed in the controller design. The proposed trajectory tracking control scheme can guarantee exponential convergence of the image position and velocity tracking errors. To illustrate the performance of the proposed scheme, experimental results are provided in this article. Note to Practitioners —This article was motivated by the vision-based motion control problem of mobile robots in uncalibrated environments. Existing vision-based motion control approaches for nonholonomic mobile robots generally depend on offline precise/coarse or online numerical/adaptive calibration of the camera intrinsic and extrinsic parameters and require precise or coarse knowledge of the camera in their implementation. This article presents a novel calibration-free image-based trajectory tracking control scheme, which can be implemented easily in real environments without any offline or online calibration of the camera parameters and can be used to efficiently control the motion of nonholonomic mobile robots with an arbitrarily placed and truly unknown overhead camera. Experimental results show that the proposed scheme can achieve satisfactory trajectory tracking control performance despite the lack of any knowledge about the camera intrinsic and extrinsic parameters and the presence of unknown camera lens distortions, and hence, can provide a simple but efficient solution to the vision-based motion control problem of nonholonomic mobile robots.

Online stereo camera calibration for automotive vision based on HW-accelerated A-KAZE-feature extraction

Abstract Nowadays ongoing integration of camera based advanced driver assistance systems (ADAS) in vehicles demands increasingly complex digital image processing in order to interpret the surrounding situations. To ensure a timely reaction for obstacles lying ahead, far reaching depth information of the observed scene is necessary. Using stereo camera systems, this is achievable by enlarging the camera base line. With rapid driving, the exact alignment of the stereo images is no longer ensured due to vibrations of the vehicle. Based on detection, extraction and matching of Accelerated-KAZE image features (A-KAZE), the geometric distortions are compensable by estimating the external camera parameters for image rectification. The indispensable frame rate for applications in vehicles and the limited power budget in combination with the SW-flexibility demanded for future ADAS applications requires the usage of optimized hardware architectures. Thus, an online camera calibration based on an HW-accelerated A-KAZE extraction is introduced in this work. The suitability of A-KAZE features for an online camera calibration is proven. Furthermore, the Tensilica Vision P5-processor is evaluated regarding its suitability for real-time A-KAZE feature extraction. This processor provides a comprehensive instruction-set extension for high-performance digital image processing. While preserving the initial A-KAZE accuracy, a feature descriptor length reduction of factor × 3.8 is attained compared to the initial descriptor size and an estimated frame rate of 20 fps is achieved for A-KAZE feature extraction on the Tensilica Vision P5-processor.

Continuous localization and mapping of a pan–tilt–zoom camera for wide area tracking

Pan---tilt---zoom (PTZ) cameras are well suited for object identification and recognition in far-field scenes. However, the effective use of PTZ cameras is complicated by the fact that a continuous online camera calibration is needed and the absolute pan, tilt and zoom values provided by the camera actuators cannot be used because they are not synchronized with the video stream. So, accurate calibration must be directly extracted from the visual content of the frames. Moreover, the large and abrupt scale changes, the scene background changes due to the camera operation and the need of camera motion compensation make target tracking with these cameras extremely challenging. In this paper, we present a solution that provides continuous online calibration of PTZ cameras which is robust to rapid camera motion, changes of the environment due to varying illumination or moving objects. The approach also scales beyond thousands of scene landmarks extracted with the SURF keypoint detector. The method directly derives the relationship between the position of a target in the ground plane and the corresponding scale and position in the image and allows real-time tracking of multiple targets with high and stable degree of accuracy even at far distances and any zoom level.

Camera self-calibration from translation by referring to a known camera.

This paper presents a novel linear method for camera self-calibration by referring to a known (or calibrated) camera. The method requires at least three images, with two images generated by the uncalibrated camera from pure translation and one image generated by the known reference camera. We first propose a method to compute the infinite homography from scene depths. Based on this, we use two images generated by translating the uncalibrated camera to recover scene depths, which are further utilized to linearly compute the infinite homography between an arbitrary uncalibrated image, and the image from the known camera. With the known camera as reference, the computed infinite homography is readily decomposed for camera calibration. The proposed self-calibration method has been tested with simulation and real image data. Experimental results demonstrate that the method is practical and accurate. This paper proposes using a "known reference camera" for camera calibration. The pure translation, as required in the method, is much more maneuverable, compared with some strict motions in the literature, such as pure rotation. The proposed self-calibration method has good potential for solving online camera calibration problems, which has important applications, especially for multicamera and zooming camera systems.

Cell-based visual surveillance with active cameras for 3D human gaze computation

Capturing fine resolution and well-calibrated video images with good object visual coverage in a wide space is a tough task for visual surveillance. Although the use of active cameras is an emerging method, it suffers from the problems of online camera calibration difficulty, mechanical delay handling, image blurring from motions, and algorithm un-friendly due to dynamic backgrounds, etc. This paper proposes a cell-based visual surveillance system by using N (N???2) active cameras. We propose the camera scan speed map (CSSM) to deal with the practical mechanical delay problem for active camera system design. We formulate the three mutually-coupled problems of camera layout, surveillance space partition with cell sequence, and camera parameter control, into an optimization problem by maximizing the object resolution while meeting various constraints such as system mechanical delay, full visual coverage, minimum object resolution, etc. The optimization problem is solved by using a full searching approach. The cell-based calibration method is proposed to compute both the intrinsic and exterior parameters of active cameras for different cells. With the proposed system, the foreground object is detected based on motion and appearance features and tracked by dynamically switching the two groups of cameras across different cells. The proposed algorithms and system have been validated by an in-door surveillance experiment, where the surveillance space was partitioned into four cells. We used two active cameras with one camera in one group. The active cameras were configured with the optimized pan, tilt, and zooming parameters for different cells. Each camera was calibrated with the cell-based calibration method for each configured pan, tilt, and zooming parameters. The algorithms and system were applied to monitor freely moving peoples within the space. The system can capture good resolution, well-calibrated, and good visual coverage video images with static background in support of automatic object detection and tracking. The proposed system performed better than traditional single or multiple fixed camera system in term of image resolution, surveillance space, etc. We further demonstrated that advanced 3D features, such as 3D gazes, were successfully computed from the captured good-quality images for intelligent surveillance.

Adaptive visual servoing scheme free of image velocity measurement for uncertain robot manipulators

This work addresses the visual tracking problem of robot manipulators with non-negligible dynamics using a fixed camera, when the camera–robot system parameters are uncertain. An adaptive strategy is developed for visual servoing systems based on the image-based look-and-move structure to allow the tracking of a 2D reference trajectory, without using image velocity measurements. The adaptive visual servoing problem free of image velocity information is formulated as a relative degree two MIMO adaptive control problem. As a solution, we employ a recently proposed Lyapunov/passivity-based adaptive control scheme based on the SDU factorization method. From a cascade control strategy, the resulting online camera calibration scheme is combined with a direct adaptive motion controller for the robot manipulator, which takes into account its uncertain nonlinear dynamics. The overall stability of the adaptive visual servoing system can be proved thanks to the explicit Lyapunov-like functions of both adaptation schemes.

Approximation of the center and radius of C-arm images for automatic on-line camera calibration

This paper presents a new robot autonomous calibration method using a trigger probe. The robot grips a simple probe (which was manufactured as a standard end-effector tool) automatically to touch constraint planes in a workspace (the locations of the constraint planes are not necessarily known exactly). The robot internal sensor measurements are recorded for kinematic calibration while the tip-point of the probe is in contact with the constraint plane. The kinematic constraint conditions are obtained from the known shape of the constraint surface, rather than from the measured reference locations in a workspace. The new method eliminates any use of external measuring devices for robot end-effector location measurements for robot calibration; thus it is suitable for a periodic robot re-calibration in a shop-floor environment. Both simulation and experimental results for a six degree-of-freedom (DOF) PUMA robot are given in this paper. The evaluation results using an external precision measuring device — Coordinate Measuring Machine(CMM) — are also presented.

Real-time tracking of complex structures with on-line camera calibration

This paper presents a novel three-dimensional model-based tracking system which has been incorporated into a visual servoing system. The tracking system combines modern graphical rendering technology with constrained active contour tracking techniques to create wire-frame-snakes. It operates in real-time at video frame rate (25 Hz) and is based on an internal CAD model of the object to be tracked. This model is rendered using a binary space partition tree to perform hidden line removal and the visible features are identified on-line at each frame and are tracked in the video feed. The tracking system has been extended to incorporate real-time on-line calibration and tracking of internal camera parameters. Results from on-line calibration and visual servoing experiments are presented.

A new approach for nonlinear distortion correction in endoscopic images based on least squares estimation.

Images captured with a typical endoscope show spatial distortion, which necessitates distortion correction for subsequent analysis. In this paper, a new methodology based on least squares estimation is proposed to correct the nonlinear distortion in the endoscopic images. A mathematical model based on polynomial mapping is used to map the images from distorted image space onto the corrected image space. The model parameters include the polynomial coefficients, distortion center, and corrected center. The proposed method utilizes a line search approach of global convergence for the iterative procedure to obtain the optimum expansion coefficients. A new technique to find the distortion center of the image based on curvature criterion is presented. A dual-step approach comprising token matching and integrated neighborhood search is also proposed for accurate extraction of the centers of the dots contained in a rectangular grid, used for the model parameter estimation. The model parameters were verified with different grid patterns. The distortion-correction model is applied to several gastrointestinal images and the results are presented. The proposed technique provides high-speed response and forms a key step toward online camera calibration, which is required for accurate quantitative analysis of the images.

Technique of distortion correction in endoscopic images using a polynomial expansion.

A new technique to correct non-linear distortion of endoscopic images based on L2-norm approximation is proposed. A mathematical model is defined that maps the endoscopic images from the distorted image plane onto an undistorted image plane. A set of model parameters is defined, consisting of the image distortion centre, corrected centre and expansion polynomial coefficients representing radial distortion correction. A new method to find the image distortion centre based on a curvature criterion is also developed. The expansion coefficients are estimated on the basis of the degree of straightness of the grid lines in an image of a test grid consisting of dots in a rectangular pattern. The corrected image centre is computed by ensuring that the distances between neighbouring test dots of different grid lines in the corrected image are the same. The quantitative data providing the results and errors of the expansion characteristics are presented. The performance of the proposed distortion correction algorithm is validated with grid patterns of different orientations. The algorithm is also applied to typical endoscopic images, and the results are presented. The high-speed response of the proposed technique is a key step towards on-line camera calibration.

3-D polyhedral face computation from two perspective views with the aid of a calibration plate

The 3-D reconstruction of visible polyhedral faces from a pair of general perspective views with the aid of a calibration plate is addressed. A polyhedron is placed on a planar calibration plate and two side views of both the polyhedron and the calibration plate are taken. Through proper arrangements we may assume that in the two views a number of polyhedral edges lying on the calibration plate and the whole calibration plate boundary are visible. We present an online camera calibration technique with the aid of the calibration plate and a two-stage process to find the vertex/edge correspondences without encountering the ambiguity problem of the conventional epipolar line technique. We then give a closed form solution to the 3-D polyhedral vertices visible in both images. We also describe other advantages of using our method for the 3-D polyhedron reconstruction. Experimental results show that the obtained 3-D polyhedral vertex coordinates are rather accurate.

SOME ASPECTS OF RGB VISION AND ITS APPLICATIONS IN INDUSTRY

RGB machine vision overlaps both colorimetry instrumentation and gray scale machine vision, and offers advantages over both. Proper calibration of the color camera which makes measurements independent of changes in illumination is essential for the reliability of color machine vision. A practical approach to on-line color camera calibration under unstable illumination conditions is presented and the performance of the procedure is evaluated. New potential applications of RGB machine vision are discussed from the perspective of physics-based vision and the calibration procedure developed here.