Camera Calibration Errors Research Articles

Visual Servoing for Aerial Vegetation Sampling Systems

This research describes a vision-based control strategy that employs deep learning for an aerial manipulation system developed for vegetation sampling in remote, dangerous environments. Vegetation sampling in such places presents considerable technical challenges such as equipment failures and exposure to hazardous elements. Controlling aerial manipulation in unstructured areas such as forests remains a significant challenge because of uncertainty, complex dynamics, and the possibility of collisions. To overcome these issues, we offer a new image-based visual servoing (IBVS) method that uses knowledge distillation to provide robust, accurate, and adaptive control of the aerial vegetation sampler. A convolutional neural network (CNN) from a previous study is used to detect the grasp point, giving critical feedback for the visual servoing process. The suggested method improves the precision of visual servoing for sampling by using a learning-based approach to grip point selection and camera calibration error handling. Simulation results indicate the system can track and sample tree branches with minimum error, demonstrating that it has the potential to improve the safety and efficiency of aerial vegetation sampling.

Noise-Robust 3D Pose Estimation Using Appearance Similarity Based on the Distributed Multiple Views.

In this paper, we present a noise-robust approach for the 3D pose estimation of multiple people using appearance similarity. The common methods identify the cross-view correspondences between the detected keypoints and determine their association with a specific person by measuring the distances between the epipolar lines and the joint locations of the 2D keypoints across all the views. Although existing methods achieve remarkable accuracy, they are still sensitive to camera calibration, making them unsuitable for noisy environments where any of the cameras slightly change angle or position. To address these limitations and fix camera calibration error in real-time, we propose a framework for 3D pose estimation which uses appearance similarity. In the proposed framework, we detect the 2D keypoints and extract the appearance feature and transfer it to the central server. The central server uses geometrical affinity and appearance similarity to match the detected 2D human poses to each person. Then, it compares these two groups to identify calibration errors. If a camera with the wrong calibration is identified, the central server fixes the calibration error, ensuring accuracy in the 3D reconstruction of skeletons. In the experimental environment, we verified that the proposed algorithm is robust against false geometrical errors. It achieves around 11.5% and 8% improvement in the accuracy of 3D pose estimation on the Campus and Shelf datasets, respectively.

Multi-View People Detection in Large Scenes via Supervised View-Wise Contribution Weighting

Recent deep learning-based multi-view people detection (MVD) methods have shown promising results on existing datasets. However, current methods are mainly trained and evaluated on small, single scenes with a limited number of multi-view frames and fixed camera views. As a result, these methods may not be practical for detecting people in larger, more complex scenes with severe occlusions and camera calibration errors. This paper focuses on improving multi-view people detection by developing a supervised view-wise contribution weighting approach that better fuses multi-camera information under large scenes. Besides, a large synthetic dataset is adopted to enhance the model's generalization ability and enable more practical evaluation and comparison. The model's performance on new testing scenes is further improved with a simple domain adaptation technique. Experimental results demonstrate the effectiveness of our approach in achieving promising cross-scene multi-view people detection performance.

ANALYSIS OF CALIBRATION ERRORS IN AUTOMOBILE CONTROL TELEVISION SYSTEM

Camera calibration is a critical method for 3D measurement applications based on 2D images. During calibration, internal camera parameters such as principal point and focal length are determined. The principal point is the intersection of the camera's optical axis and the image plane, while the focal length is measured from the lens center to the principal point.This article theoretically investigates calibration errors in two reference planes considering various conditions, such as shifts in the reference plane and changes in focal lengths. Errors in determining the coordinates of reference points in images were used to evaluate these calibration errors.The error in estimating the focal length and principal point is formulated using analytical equations, and the results are compared with simulation outcomes to validate the effectiveness of the proposed formulas. The research reveals that calibration errors arise due to shifts in the reference plane and changes in focal lengths. The obtained results are crucial for enhancing camera calibration methods and improving the accuracy of 3D measurements based on 2D images.This article extends the understanding of camera calibration errors and provides new theoretical approaches for their assessment. The research findings may be beneficial for scientists and practitioners working in the fields of computer vision, image processing, and 3D measurements.Further research in this area may contribute to the development of new calibration methods and approaches that will improve the performance of automotive systems and ensure high accuracy and reliability in the future.

Adaptive visual servoing for the robot manipulator with extreme learning machine and reinforcement learning

AbstractIn this study, a novel image‐based visual servo (IBVS) controller for robot manipulators is investigated using an optimized extreme learning machine (ELM) algorithm and an offline reinforcement learning (RL) algorithm. First of all, the classical IBVS method and its difficulties in accurately estimating the image interaction matrix and avoiding the singularity of pseudo‐inverse are introduced. Subsequently, an IBVS method based on ELM and RL is proposed to solve the problem of the singularity of the pseudo‐inverse solution and tune adaptive servo gain, improving the servo efficiency and stability. Specifically, the ELM algorithm optimized by particle swarm optimization (PSO) was used to approximate the pseudo‐inverse of the image interaction matrix to reduce the influence of camera calibration errors. Then, the RL algorithm was adopted to tune the adaptive visual servo gain in continuous space and improve the convergence speed. Finally, comparative simulation experiments on a 6‐DOF robot manipulator were conducted to verify the effectiveness of the proposed IBVS controller.

Towards a Guideline for UAV-Based Data Acquisition for Geomorphic Applications

Recent years have seen a rapid rise in the generation of high-resolution topographic data using custom-built or commercial-grade Unmanned Aerial Vehicles (UAVs). Though several studies have demonstrated the application potential of UAV data, significant knowledge gaps still exist in terms of proper documentation of protocols for data acquisition, post-flight data processing, error assessments, and their mitigation. This work documents and provides guidelines for UAV data acquisition and processing from several years of field experience in diverse geomorphic settings across India, including undulating topography (~17 km2), alluvial plains (~142 km2), lowland-river basin (~66 km2), and a highly urbanized area (~5 km2). A total of 37,065 images with 16 and 20 Megapixels and 604 ground control points (GCPs) were captured with multiple UAV systems and processed to generate point clouds for a total area of ~230 km2. The Root Mean Square Error (RMSE) for each GCP for all sites ranged from 6.41 cm to 36.54 cm. This manuscript documents a comprehensive guideline for (a) pre-field flight planning and data acquisition, (b) generation and removal of noise and errors of the point cloud, and (c) generation of orthoimages and digital elevation models. We demonstrate that a well-distributed and not necessarily uniformly distributed GCP placement can significantly reduce doming error and other artifacts. We emphasize the need for using separate camera calibration parameters for each flight and demonstrate that errors in camera calibration can significantly impact the accuracy of the point cloud. Accordingly, we have evaluated the stability of lens calibration parameters between consumer-grade and professional cameras and have suggested measures for noise removal in the point cloud data. We have also identified and analyzed various errors during point cloud processing. These include systematic doming errors, errors during orthoimage and DEM generation, and errors related to water bodies. Mitigation strategies for various errors have also been discussed. Finally, we have assessed the accuracy of our point cloud data for different geomorphic settings. We concluded that the accuracy is influenced by Ground Sampling Distance (GSD), topographic features, and the placement, density, and distribution of GCPs. This guideline presented in this paper can be extremely beneficial to both experienced long-term users and newcomers for planning the UAV-based topographic survey and processing the data acquired.

Logarithmic Observation of Feature Depth for Image-Based Visual Servoing

Due to the robustness to robot modeling and camera calibration errors and avoidance of complete target geometry, image-based visual servoing has always been an important topic in the fields such as robotics, computer vision and so forth. When the image information obtained by the camera is mapped to the robotic task space to design the servoing control law, the resulting interaction matrix, which links the spatial velocity of the camera to the temporal variation of the selected image features, depends on the unknown feature depths. The use of inaccurate feature depths may influence the stability and robustness of the controller, and even cause the failure of the task. In this article, based on the perspective camera model, by employing the principle of reduced order observer, a novel logarithmic observer is presented for on-line recovery of feature depth. Compared with the typical observers now available, the presented observer offers several advantages: global convergence, a faster convergence rate of error structure than exponential error structure, a less restricted observability condition and greater robustness against measurements with noise. The comparison results of numerical simulations indicate the superiority of the presented observer, and real experiments with Kinect v2 sensor further validate the effectiveness of the presented observer in practical situation. Note to Practitioners—This article was motivated by the depth problem in the image-based visual servoing scheme, but it can also be used in other situations where the image depth information is needed, such as 3D reconstruction, robot navigation, etc. The existing depth acquisition methods include TOF sensors, stereo vision, depth observers and so on. However, TOF sensors are sensitive to light conditions, and the mounting space of stereo vision is slightly large, and there is contradiction between observation performance and computational complexity in most existing observers. In this article, a novel structure of logarithmic reduced order observer is described in detail, which can be utilized to estimate the depth information of images easily. The simulations and experiments verify the good performance of the observer. The limitations of the given observer are that the estimation accuracy is not very good under weak excitation, and the camera needs to be calibrated in advance. Future work will focus on overcoming these two limitations.

Stability and Convergence Analysis of 3D Feature-Based Visual Servoing

Visual control based on image features has received much attention for its inherent robustness against camera calibration errors, modelling uncertainties and the capability to keep the object in the Field of View (FoV) of the camera. Nevertheless, some drawbacks related to the basin of convergence and the existence of local minima, which make the camera to get stuck in undesired equilibrium points, are still worth being investigated. Nowadays, the availability of cheap and lightweight RGB-D cameras makes the use of three-dimensional features natural. By using an RGB-D camera in an eye-in-hand configuration, this letter proposes an in-depth stability and convergence analysis of 3D feature-based visual servoing. It will be proved that the visual control system is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">almost globally asymptotically stable</i> in the sense that the only trajectories not converging to the desired equilibrium point are those belonging to a zero Lebesgue measure set in the feature space. Moreover, a sufficient condition guaranteeing that the feature trajectories remain in the camera FoV is derived and an algorithm to prevent feature loss caused by violation of the camera FoV constraint is proposed.

Image Based Visual Servoing for Floating Base Mobile Manipulator Systems with Prescribed Performance under Operational Constraints

This paper presents a novel Image-Based Visual Servoing (IBVS) control approach for Floating Base Mobile Manipulator Systems (FBMMSs) that imposes prescribed transient and steady-state response on the image feature coordinate errors while satisfying the visibility constraints that arise owing to the camera’s limited field of view. The proposed control strategy does not incorporate any knowledge on either the FBMMS dynamic model, the exogenous disturbances, or the inevitable camera calibration and depth measurement errors. More specifically, it guarantees: (i) predefined behavior in terms of overshoot, convergence rate, and maximum steady-state error value of the image features and system velocities tracking errors; (ii) satisfaction of camera field of view constraints; (iii) bounded closed-loop control signals, and (iv) reduced design and implementation complexity. Additionally, the performance of the developed scheme is solely determined by certain designer-specified performance functions/parameters, and it is fully decoupled by the control gains selection. The efficiency of the proposed scheme is demonstrated via a realistic simulation study, using an eye-in-hand Underwater Vehicle Manipulator System (UVMS) as a test-bed FBMMS platform.

Dynamic visual servoing with Kalman filter-based depth and velocity estimator

Camera calibration error, vision latency, nonlinear dynamics, and so on present a major challenge for designing the control scheme for a visual servoing system. Although many approaches on visual servoing have been proposed, surprisingly, only a few of them have taken into account system dynamics in the control design of a visual servoing system. In addition, the depth information of feature points is essential in the image-based visual servoing architecture. As a result, to cope with the aforementioned problems, this article proposes a Kalman filter-based depth and velocity estimator and a modified image-based dynamic visual servoing architecture that takes into consideration the system dynamics in its control design. In particular, the Kalman filter is exploited to deal with the problems caused by vision latency and image noise so as to facilitate the estimation of the joint velocity of the robot using image information only. Moreover, in the modified image-based dynamic visual servoing architecture, the computed torque control scheme is used to compensate for system dynamics and the Kalman filter is used to provide accurate depth information of the feature points. Results of visual servoing experiments conducted on a two-degree of freedom planar robot verify the effectiveness of the proposed approach.

3D measurement system based on divergent multi-line structured light projection, its accuracy analysis

This paper proposes an effective three-dimensional(3D) measurement system based on divergent multi-line structured light projection, and analyzes the measurement accuracy of the system. In the system, the previous methods have been improved. The accuracy of system calibration and 3D measurement can be improved effectively by changing the calibration board, changing the relative distance between the camera and the laser. Firstly, the camera calibration and the light plane calibration are realized by projecting a circular calibration board and a blank board. Secondly, the Steger centerline extraction algorithm is used to complete the extraction of the centerline of the stripe. Finally, based on the system calibration parameters and the centerline extraction results, the surface 3D measurement of the object is realized, and the camera calibration, light plane calibration and 3D measurement accuracy are analyzed. Experiments show that the calibration method proposed in this paper can obtain higher accuracy than the checkerboard calibration method, which can reduce the camera calibration error by more than 3 times and the light plane calibration error by more than 1 times. In addition, compared with the previous calibration method, the blank board avoids the influence of the calibration pattern on the light stripe. Finally, the relative error of light plane calibration is controlled within 1% by changing the relative distance between the camera and the laser.

Distorted pinhole camera modeling and calibration.

Camera modeling and calibration are essential tasks in modern optics. Conventionally, the pinhole model is adopted with a further extension for lens distortion. However, pinhole and distortion models are mutually dependent; thus, the standard approach induces systematic camera calibration errors. This research presents a unifying distorted pinhole camera model that includes a telecentric, distortion-free pinhole, and radial lens distortion as particular cases. An iterative calibration method based on the derived distorted pinhole model is proposed, and experimental evaluation by calibrating a camera with high radial distortion is performed. The calibration results are compared with the standard and fisheye models using a well-known commercial camera calibrator software. The proposed method outperforms the standard model and achieves accuracy comparable to the fisheye model, and the proposed approach is a versatile and accurate tool for diverse optical metrology applications.

Dynamic pose estimation of uncooperative space targets based on monocular vision.

A novel algorithm of dynamic pose estimation for monocular visual sensor is proposed in this paper. The sensor is principally composed of two 1D turntables, one collimated laser, and one industrial camera. In particular, the proposed algorithm is suitable for the cases of uncooperative targets. By analyzing the motion of a laser beam based on quaternion, the functional detection algorithm is derived from the position information of multiple scanning points. Furthermore, the depth recovery based on a nonparametric model is a key step in the pose calculation, which is unnecessary to make use of the calibration parameters of an industrial camera. It is, however, effective to avoid the influence of camera distortion and calibration error. After establishing a test platform, simulation and experiments for pose estimation are carried out. The experimental results show that the maximum error is 0.98° at a range of 500 mm, which proves that the proposed algorithm is accurate and effective.

Adaptive visual servoing with an uncalibrated camera using extreme learning machine and Q-leaning

In this paper, a novel image-based visual servoing (IBVS) method using Extreme Learning Machine (ELM) and Q-learning is proposed to solve the problems of complex modeling and selection of the servo gain. First, the pseudoinverse of the interaction matrix is approached by ELM which avoids the singularity of the interaction matrix effectively and is robust to interferences such as feature noises and camera calibration errors. Second, a reinforcement learning method, Q-learning, is adopted to adaptively adjust the servo gain in order to improve the convergence speed and stability. Compared with other methods, ELM has better generalization performance, faster operation speed and a unique optimal solution. Also, Q-learning has self-learning ability without experience in advance. The effectiveness of the proposed method is validated by simulations and experiment on a 6-DOF robot with eye-in-hand configuration.

Hand–eye calibration and grasping pose calculation with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot

Due to motion constraint of 4-R(2-SS) parallel robot, it is difficult to calculate the translation component of hand–eye calibration based on the existing model solving method accurately. Additionally, the camera calibration error, robot motion error, and invalid calibration motion poses make it difficult to achieve fast and accurate online hand–eye calibration. Therefore, we propose a hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot by improving the existing eye-to-hand model and solving method. Firstly, the eye-to-hand model of single camera is improved and the robot motion error in the improved model is compensated to reduce the influence of camera calibration error and robot motion error on model accuracy. Secondly, the vertical-component of hand–eye calibration is corrected based on vertical constraint between calibration plate and end effector in parallel robot to calculate the pose and motion error in calibration of 4-R(2-SS) parallel robot accurately. Thirdly, the nontrivial solution constraint of eye-to-hand model is constructed and adopted to remove invalid calibration motion poses and plan calibration motion. Finally, the proposed method was verified by experiments with a fruit sorting system based on 4-R(2-SS) parallel robot. Compared with random motion, the existing model, and solving method, the average time of online calibration based on planned motion decreases by 29.773 s and the average error of calibration based on the improved model and solving method decreases by 151.293. The proposed method can improve the accuracy and efficiency of hand–eye calibration of 4-R(2-SS) parallel robot effectively and further realize accurate and fast grasping.

Dynamic Granularity Matrix Space Model Based High Robust Multi-Ellipse Center Extraction Method for Camera Calibration

Ellipse center extraction is the important basis of camera calibration with circle arrays calibration board, which directly affects the measurement accuracy of machine vision system. Aiming at the problem of high noise sensitivity of the most ellipse center extraction methods, a high robust multi-ellipse center extraction method is proposed. The idea is to transform the problem of multi-ellipse center extraction into the problems of ellipse sub-pixel edge segmentation and edge based ellipse center clustering. Firstly, the flowchart of the proposed method is introduced. Then, the theory of fuzzy quotient space is introduced to establish the dynamic granularity matrix space model, which describes image segmentation problem as the jumping and transformation of the image by hierarchical structure. Finally, an adaptive multi-ellipse sub-pixel edge segmentation method and an optimal ellipse center extraction method are proposed based on the dynamic granularity matrix space model. Experimental results show that the proposed method has higher accuracy, robustness and lower camera calibration error.

Simulation and Analysis of Photogrammetric UAV Image Blocks—Influence of Camera Calibration Error

Unmanned aerial vehicles (UAV) are increasingly used for topographic mapping. The camera calibration for UAV image blocks can be performed a priori or during the bundle block adjustment (self-calibration). For an area of interest with flat scene and corridor configuration, the focal length of camera is highly correlated with the height of the camera. Furthermore, systematic errors of camera calibration accumulate on the longer dimension and cause deformation. Therefore, special precautions must be taken when estimating camera calibration parameters. In order to better investigate the impact of camera calibration errors, a synthetic, error-free aerial image block is generated to simulate several issues of interest. Firstly, the erroneous focal length in the case of camera pre-calibration is studied. Nadir images are not able to prevent camera poses from drifting to compensate for the erroneous focal length, whereas the inclusion of oblique images brings significant improvement. Secondly, the case where the focal length varies gradually (e.g., when the camera subject to temperature changes) is investigated. The neglect of this phenomenon can substantially degrade the 3D measurement accuracy. Different flight configurations and flight orders are analyzed, the combination of oblique and nadir images shows better performance. At last, the rolling shutter effect is investigated. The influence of camera rotational motion on the final accuracy is negligible compared to that of the translational motion. The acquisition configurations investigated are not able to mitigate the degradation introduced by the rolling shutter effect. Other solutions such as correcting image measurements or including camera motion parameters in the bundle block adjustment should be exploited.

Image-Based Visual Servoing of Helical Microswimmers for Planar Path Following

Magnetically actuated microswimmers have attracted researchers to investigate their swimming characteristics and controlled actuation. Although plenty of studies on actuating helical microswimmers have been carried out, robust closed-loop controls should be still explored for practical applications. In this paper, we proposed a data-driven model-free method using Image-Based Visual Servoing (IBVS), which uses features directly extracted in the image space as feedbacks. The IBVS method can eliminate camera calibration errors. We have demonstrated with experiments that the proposed IBVS method can enable velocity-independent path following of an arbitrarily given path on the plane, which permits a better experience of user interaction. The proposed control method is successfully applied to obstacle avoidance tasks and has the potential for the application in complex circumstances. This approach is promising for biomedical applications. Note to Practitioners —This paper is motivated by the problem of driving a small-scale swimming robot with a helical body by magnetic fields along a predefined path. The proposed new closed-loop control uses features directly extracted in the image space as feedbacks. We demonstrated with experiments that the helical swimming robot can follow an arbitrarily given path on the plane using the proposed control method. The proposed control method is also successfully applied to obstacle avoidance tasks.

Plane chessboard-based calibration method for a LWIR ultra-wide-angle camera.

A method is proposed to calibrate the long-wave infrared ultra-wide-angle camera in this paper. In addition, a novel calibration chessboard is designed and an advanced chessboard corner positioning method is adopted to improve the calibration precision. The designed calibration chessboard can achieve high thermal infrared contrast and exhibits outstanding stability, which is made of a thermoelectric semiconductor refrigeration device. The proposed subpixel corner positioning method can accurately locate the corners on the calibration chessboard according to the characteristics of the infrared image and the checkerboard pattern. Both the principle of the proposed infrared chessboard and the subpixel corner positioning procedure were presented, and the calibration experiment showed that the mean reprojection error and the root mean square error were reduced to 0.32 pixel and 0.39 pixel, respectively. Comparison studies were also performed to verify the calibration effect of the proposed method, and the possibilities of camera calibration error of the proposed method were analyzed.

Calibrating projector flexibly for a real-time active 3D scanning system

A novel method for projector calibration is proposed that is supported by an uncalibrated camera. The use of an uncalibrated camera is motivated by the fact that camera calibration errors may negatively influence the projector calibration results. The principles of camera calibration are used to calibrate the projector by considering it as a camera that acts in a reverse manner. A regular shaped pattern with known dimensions such as a chessboard is projected by the projector, captured by the camera and then used for projector calibration. The re-projection error analysis indicates that the proposed method performs on par with the calibration results of a camera.