Camera Coordinate System Research Articles

Adaptive Positioning Repair Method for Aero-Engine Blades by Using Speckle Vision Measurement

The repair of damaged aero-engine blades has an extremely high economic value. Due to the complexity of the blade surface and various kinds of deformation during the lifetime, blade positioning in the repair process has always been one of the key problems in the aviation industry. In this study, a method of blade self-adaptive positioning repair method based on speckle vision measurement is proposed. Firstly, a speckle vision measurement system is established to measure the speckles coated on the damaged blade surface to obtain its surface point cloud. Meanwhile, the coordinates of the reference points on the fixture are obtained to build the relative pose model between the blade and the fixture. Secondly, through the solution of posture relations among the mark point coordinate system, measuring camera coordinate system, probe coordinate system, and machine coordinate system, the self-adaptive positioning of the under-positioning clamping blade in the machine coordinate system is achieved. Then based on the positioning model, the machining repair parameters are selected reasonably to complete the CNC (Computer numerical control) path planning of tool repair. Finally, the DMU five-axis numerical control center was used to carry out adaptive positioning repair experiments on the damaged blades. The experimental results showed that this method could achieve satisfying repair effect in under positioning clamping, and provide a new method for repairing damaged aero-engine blades.

Extrinsic Calibration between Camera and LiDAR Sensors by Matching Multiple 3D Planes.

This paper proposes a simple extrinsic calibration method for a multi-sensor system which consists of six image cameras and a 16-channel 3D LiDAR sensor using a planar chessboard. The six cameras are mounted on a specially designed hexagonal plate to capture omnidirectional images and the LiDAR sensor is mounted on the top of the plates to capture 3D points in 360 degrees. Considering each camera–LiDAR combination as an independent multi-sensor unit, the rotation and translation between the two sensor coordinates are calibrated. The 2D chessboard corners in the camera image are reprojected into 3D space to fit to a 3D plane with respect to the camera coordinate system. The corresponding 3D point data that scan the chessboard are used to fit to another 3D plane with respect to the LiDAR coordinate system. The rotation matrix is calculated by aligning normal vectors of the corresponding planes. In addition, an arbitrary point on the 3D camera plane is projected to a 3D point on the LiDAR plane, and the distance between the two points are iteratively minimized to estimate the translation matrix. At least three or more planes are used to find accurate external parameters between the coordinate systems. Finally, the estimated transformation is refined using the distance between all chessboard 3D points and the LiDAR plane. In the experiments, quantitative error analysis is done using a simulation tool and real test sequences are also used for calibration consistency analysis.

Calibration based on ray-tracing for multi-line structured light projection system.

Multi-line Structured Light Projection System (MSLPS) is widely used with the advantages of high-contrast stripes and high robustness in high-speed dynamic reconstruction in comparison with Phase Modulation System (PMS). However, existing calibration methods are either sophisticated or low-accuracy in large visual field application because the bending of line structured light plane caused by lens distortion is inevitable and projector calibration is necessary. Therefore, in this paper, we present an accurate and robust calibration method based on ray-tracing, which establishes accurate relationship between subpixel coordinates of each bending line structured light plane and 3D coordinates based on camera coordinate system without projector calibration. As long as placing a checkerboard at random positions, MSLPS can be calibrated and hence is more suitable in practical application. Experimental results demonstrated that, compared with existing line structured light strategies, the proposed method could achieve more accurate and robust 3D reconstruction and the calibration process is simpler and easier to implement.

Optical distortion evaluation based on the Integrated Colour CCD Moiré Method.

Camera calibration is an important part of high-precision optical measurement, which is especially difficult in the micro-nano field. Based on the integrated correlation calculation and CCD moiré method, this paper describes the development of a lens calibration technique called the Integrated Colour CCD Moiré Method (ICCM). The CCD moiré fringes, formed by superimposing a periodic optical signal of a specimen grating with a CCD target array or a Bayer filter array, significantly enlarges the deformation modulated by lens distortion and the calibration plate attitude (i.e. the rotation angle relative to the camera coordinate system). To measure lens distortion using CCD moiré, the deformation pattern that is governed by the lens distortion, specimen grating attitude and carrier was used to construct a CCD fringe image. If the constructed CCD fringe image based on the trial lens distortion and rotation angles have a maximum similarity to the captured CCD moiré image, the lens distortion and rotation angles are correctly inversed. Particle swarm optimisation algorithm was selected to search for the true value so that the accuracy and robustness could be improved. Numerical experiments verified that the ICCM method developed in this work can simultaneously inverse the lens distortion, rotation angle and the grating pitch with high precision. The lens distortion of the metallographic microscope has been successfully characterised by the developed method with an 833 nm pitch grating. Simulations and experiments showed that ICCM is an intuitive, accurate, anti-noise and robust distortion calibration method.

Hand-Eye Calibration With a Remote Centre of Motion

In the eye-in-hand robot configuration, hand-eye calibration plays a vital role in completing the link between the robot and camera coordinate systems. Calibration algorithms are mature and provide accurate transformation estimations for an effective camera-robot link but rely on a sufficiently wide range of calibration data to avoid errors and degenerate configurations. This can be difficult in the context of keyhole surgical robots because they are mechanically constrained to move around a remote centre of motion (RCM) which is located at the trocar port. The trocar limits the range of feasible calibration poses that can be obtained and results in ill-conditioned hand-eye constraints. In this letter, we propose a new approach to deal with this problem by incorporating the RCM constraints into the hand-eye formulation. We show that this not only avoids ill-conditioned constraints but is also more accurate than classic hand-eye calibration with a free 6DoF motion, due to solving simpler equations that take advantage of the reduced DoF. We validate our method using simulation to test numerical stability and a physical implementation on an RCM constrained KUKA LBR iiwa 14 R820 equipped with a NanEye stereo camera.

Calibration of mobile robot odometry and camera based on hand-eye calibration

The relative pose transformation among several sensors in mobile robots has a certain impact on the pose estimation of the robot. The traditional hand-eye calibration works out the matrix transformation relationship between the camera coordinate system and the robot end effector or its base coordinate system. As for the household sweeping robot, which use monocular camera for localization and mapping. A novel method is proposed calibrating the chassis odometry and the top-view camera. First, we get chassis poses from wheel encoder and camera poses from Perspective-n-Point(PnP).Then we use two types of poses to solve the formula we build which is similar to hand-eye calibration. We make some tests to ensure this method is feasible.

A Novel Multi-Camera Global Calibration Method for Gaze Tracking System

In a 3-D gaze tracking system, the system calibration is used to determine the positions of the light sources and the screen in the camera coordinate system, which plays an important role in the estimation of the 3-D gaze and the point of regard (PoR). The traditional way for the system calibration is the mirror-based method, in which a mirror is used to make the light sources and the screen can be captured by the camera through specular reflection. However, the mirror-based method is imprecise and complex. In this paper, a novel system-calibration method for the 3-D gaze tracking system is proposed. On the basis of the system camera, a calibration camera is placed on the opposite of the screen to constitute a multi-camera system. The images are captured by the two cameras to estimate the light sources and the screen of the system camera coordinate frame. Simulations and experiments were carried out to illustrate the validation of the method. This method solves the problem that the light sources and the screen are not in view of the system camera. In addition, it is easy to implement while ensuring high accuracy and providing a new method for other similar calibration systems.

Adaptive Trajectory Tracking of Wheeled Mobile Robots Based on a Fish-eye Camera

This paper presents a novel adaptive trajectory tracking control method, which can precisely control wheeled mobile robots only using an uncalibrated fish-eye camera fixed on the ceiling. Different from existing approaches, the inertial device, distorted image correction, and the trajectory expression are not required in the control system. The position and orientation of the mobile robot in the camera coordinate system are estimated by the extended POSIT (Pose from Orthography and Scaling with Iteration) algorithm in real-time. Based on estimation results, the controller considering both tracking errors and parameter estimated errors is designed by linear parameterization, where the camera intrinsic parameters are online updated. The asymptotic convergence of the tracking error and the estimated error to zero is proved by the Barbalat lemma. Circular trajectory and irregular trajectory tracking experiments have been conducted to verify the performance of our controller.

A Calibration Method for System Parameters in Direct Phase Measuring Deflectometry

Phase measuring deflectometry has been widely studied as a way of obtaining the three-dimensional shape of specular objects. Recently, a new direct phase measuring deflectometry technique has been developed to measure the three-dimensional shape of specular objects that have discontinuous and/or isolated surfaces. However, accurate calibration of the system parameters is an important step in direct phase measuring deflectometry. This paper proposes a new calibration method that uses phase information to obtain the system parameters. Phase data are used to accurately calibrate the relative orientation of two liquid crystal display screens in a camera coordinate system, by generating and displaying horizontal and vertical sinusoidal fringe patterns on the two screens. The results of the experiments with an artificial specular step and a concave mirror showed that the proposed calibration method can build a highly accurate relationship between the absolute phase map and the depth data.

Dual-station intelligent welding robot system based on CCD

Welding is a manufacturing technique that joins metal or other thermoplastic materials in a heated, high-temperature, or high-pressure manner. Harmful dust and the light from welding can cause great harm to the human body. In addition, to improve welding quality and efficiency, intelligent welding has become an urgent need in the manufacturing industry. Herein, a dual-station intelligent welding strategy is designed based on an industrial charge-coupled device (CCD) visual detection system and welding control system. This solves the problem of expensive laser vision sensors and the poor detection effect of the visual system under the conditions of a short distance and strong light intensity. In the actual environment, there is the problem of radial distortion that affects the edge of the image, and the problem of the optical axis of the camera not being consistent with the installation plane. In this study, the camera coordinate system and hand–eye coordinate system are calibrated separately. The abovementioned problems are solved, and the images obtained by the CCD are acquired and processed in real-time. Image segmentation was performed using neighbourhood average filtering, iterative threshold segmentation, and local information extraction. The edge amplitude image was obtained by the Prewitt operator. Under the Hough transform method, the recognition of the weld seam and the extraction of weld feature points are realised. We designed an intelligent weld detection system that contains a friendly human–computer interface. Through numerous repeated experiments on circular and square workpieces, the control error of this intelligent welding system is within 0.2 mm, and the time of single seam feature extraction is 0.8 s.

Automatic calibration of an arbitrarily-set near-infrared camera for patient surface respiratory monitoring.

A patient's respiratory monitoring is one of the key techniques in radiotherapy for a moving target. Generally, such monitoring systems are permanently set to a fixed geometry during the installation. This study aims to enable a temporary setup of such a monitoring system by developing a fast method to automatically calibrate the geometrical position by a quick measurement of calibration markers. One calibration marker was placed on the isocenter and the other six markers were placed at positions 5-cm apart from the isocenter to the left, right, anterior, posterior, superior, and inferior directions. A near-infrared (NIR) camera (NIC) [Kinect v2 (Microsoft Corp.)] was arbitrarily set with ten different angles around the calibration phantom with a fixed tilting-down angle at approximately 45° in a linear accelerator treatment vault. The three-dimensional (3D) coordinates in the camera (Cam) coordinate system (CS; and are the horizontal and vertical coordinates of the image, respectively, and is a coordinate along the NIR time-of-flight) were taken for 1min with 30 frames per second. The data corresponding to the measurement times of 1, 3, 10, 30, and 60s were created to mimic various measurement times. These data were used to calculate the initial matrix elements, which included six parameters of the pitching, yawing, and rolling angles; horizontal two-dimensional translation in the treatment room; and the source-to-axis distance of NIC, for a conversion from the Cam CS to the treatment room CS for which the origin was defined at the isocenter (Iso coordinate). The six parameters were then optimized to minimize the displacements of the calculated marker coordinates from the actual positions in the Iso CS. The 3D positional accuracy and angular accuracy of the conversion were evaluated. The random error of the Iso coordinates was analyzed through a relation with the angle of each measurement setup. Three angles of NIC and relative translation vectors were successfully calculated from the measurement data of the calibration markers. The achieved spatial and angular accuracies were 0.02mm and 1.6°, respectively, after the optimization. Among the mimicked measurement times investigated in this study, both spatial and angular accuracies had no dependence on the measurement time. The average random error of a static marker was 0.46mm after the optimization. We developed an automatic method to calibrate the 3D patient surface monitoring system. The procedure developed in this study enabled a quick calibration of NIC, which can be easily repeated multiple times for a frequent and quick setup of the monitoring system.

Image Uncertainty-Based Absolute Camera Pose Estimation with Fibonacci Outlier Elimination

Estimating the pose of a camera is a vital requirement in real-world applications like virtual reality, structure from motion, vision-assisted robot localization and manipulation. The existing Perspective-n-Point (PnP) based pose estimation algorithms have poor accuracy in presence of noise and outliers. Hence, they are combined with the Random Sample Consensus strategies to eliminate the outliers prior to pose estimation and to produce accurate results at the expense of computation time. With this concern, an Image Uncertainty-based Perspective-n-Point (IUPnP) with Fibonacci-based outlier rejection is proposed to accurately estimate the absolute pose of a calibrated camera with a minimum computation load. The uncertainties of the spherically normalized camera coordinates are formulated in the tangent space of the camera coordinate system and the initial pose is estimated using Singular Value Decomposition. The correspondences with tangent space residual exceeding the threshold values, are classified as outliers and then rejected iteratively. In order to prevent the inlier rejections and to improve the pose estimates, the threshold values are updated eventually using the Fibonacci technique. Finally, the estimated pose values are refined, using Gauss-Newton optimization. The proposed IUPnP algorithm is tested with synthetic data and real image data to validate its performance by comparing with the existing PnP algorithms in terms of accuracy. The results show that the proposed technique produces better pose estimates for correspondences with 50% outliers, than the state-of-art techniques do.

Research on Robot Control Technology Based on Vision Localization

Based on the understanding of machine vision localization technology at home and abroad, this paper outlines the overall design of the system, and analyses the working principle and workflow of the robot with vision system in workpiece grinding. The hardware design of the system is introduced. The process of image processing is analyzed in detail, and the results of image processing are given. The basic parameters of camera imaging are taken as internal parameters. The camera calibration is obtained by rotation matrix R and translation parameter T. The coordinate transformation of camera coordinate system and world coordinate system is analyzed. Finally, the positions and postures of the actual workpiece and the end-effector in the world coordinate system are given respectively, and the robot with the vision system is used to grasp the actual workpiece. The difficulties of this project are visual calibration, image processing and coordinate transformation. Robot vision technology can directly grasp the location. Compared with the manual mechanical positioning, the robot that realizes autonomous vision localization has more flexibility, better quality and higher efficiency.

Master-slave motion alignment for an open surgical console.

Open surgical consoles widely employed in minimally invasive surgery have better ergonomics than closed consoles. To enhance surgical robots' ergonomics, operational efficiency, and safety, an effective master-slave motion alignment model should be established. The kinematic model of the robot system based on laparoscopic camera coordinate system is built in the first place. Then, the relative pose between the operator's eyes and the display is measured by Tobii Eye Tracking Sensor and is subsequently used to improve the master-slave motion alignment model. Robot threading experiments are conducted by two doctors and three testers to verify the kinematic model. As a result, in contrast to the original model, the improved model reduces both operation time and the number of collisions. The improved master-slave motion alignment model, in which the transformation matrix between the operator's eyes and the display is employed, raises the ergonomics, operational efficiency, and safety.

A gait skeleton model extraction method based on the fusion between vision and tactility

In the traditional gait skeleton model, the fixed length proportions among different bones cause the loss of model personality, and the shoulder and hip points that are blocked by the body are difficult to be extracted. For those problems, a method based on the spatial and temporal fusion between vision and tactility is proposed, by which a equal proportion 3D gait skeleton model can be restructured in the camera coordinate system accurately through a single-frame image. In the geometric analysis process, some logical assumptions are proposed according to human anatomy and the laws of human movement, and an effective method is proposed for the extraction of thigh slope. The experimental result shows that the consistent equal proportion models, in which single bone length error is eventually controlled within $$\pm \,5\,{\text {mm}}$$, can be extracted in different position under the premise of rapidity with the aid of this method. The extracted shoulder and hip points also meet the real human body skeleton structure, which lays the foundation for the integrity and rationality of the entire skeletal model.

Method for Measuring Clearance of Automobile Body Panel with Double Structured Light System

According to the changes of the autobody measuring clearance often face the shooting angle problem caused by internal decline caused by the lack of data acquisition accuracy, a high precision and robust measurement of binocular gap line structured light and the system parameter calibration method of system integration. First the mathematical measurement model of the system is established according to the Zhang Zhengyou camera calibration model to calculate the camera parameters, the extrinsic parameters of the camera calibration of the light plane, while the introduction of auxiliary line structured light laser by solving the intersection of two camera coordinate system, finally gives the calculation model of the door gap and combined with the calibration data for the gap size. Compared with a three-dimensional scanning measuring device, the system can measure only two images at the same time. The experimental results show that when the shooting angle is less than 45 degrees at the working distance of 400mm and 50*50 field of view, the accuracy reaches 0.08mm, which meets the needs of fast and real-time measurement gap.

Active solution of homography for pavement crack recovery with four laser lines

An active solution method of the homography, which is derived from four laser lines, is proposed to recover the pavement cracks captured by the camera to the real-dimension cracks in the pavement plane. The measurement system, including a camera and four laser projectors, captures the projection laser points on the 2D reference in different positions. The projection laser points are reconstructed in the camera coordinate system. Then, the laser lines are initialized and optimized by the projection laser points. Moreover, the plane-indicated Plücker matrices of the optimized laser lines are employed to model the laser projection points of the laser lines on the pavement. The image-pavement homography is actively determined by the solutions of the perpendicular feet of the projection laser points. The pavement cracks are recovered by the active solution of homography in the experiments. The recovery accuracy of the active solution method is verified by the 2D dimension-known reference. The test case with the measurement distance of 700 mm and the relative angle of 8° achieves the smallest recovery error of 0.78 mm in the experimental investigations, which indicates the application potentials in the vision-based pavement inspection.

Global calibration of multi-cameras with non-overlapping fields of view based on photogrammetry and reconfigurable target

Multi-camera vision systems are often needed to achieve large-scale and high-precision measurement because these systems have larger fields of view (FOV) than a single camera. Multiple cameras may have no or narrow overlapping FOVs in many applications, which pose a huge challenge to global calibration. This paper presents a global calibration method for multi-cameras without overlapping FOVs based on photogrammetry technology and a reconfigurable target. Firstly, two planar targets are fixed together and made into a long target according to the distance between the two cameras to be calibrated. The relative positions of the two planar targets can be obtained by photogrammetric methods and used as invariant constraints in global calibration. Then, the reprojection errors of target feature points in the two cameras’ coordinate systems are calculated at the same time and optimized by the Levenberg–Marquardt algorithm to find the optimal solution of the transformation matrix between the two cameras. Finally, all the camera coordinate systems are converted to the reference coordinate system in order to achieve global calibration. Experiments show that the proposed method has the advantages of high accuracy (the RMS error is 0.04 mm) and low cost and is especially suitable for on-site calibration.

Methods based on 1D homography for camera calibration with 1D objects.

This paper focuses on camera calibration with one-dimensional (1D) objects, and novel methods are proposed in this paper. Different from the known 1D object-based camera calibration algorithms, which define the camera coordinate system as the world coordinate system, we assume that the 1D calibration object is located along the X axis of the world coordinate system. Based on this new model, a 3×2 1D homography is defined to relate the points in the 1D objects to the perspective image points thereof. Then, the basic constraint for camera calibration using 1D objects from a single image is derived. Subsequently, two existing motions, namely, rotating around a fixed point and moving on a plane, are discussed, and new algorithms are proposed. In our methods, if the number of points in the 1D objects is more than three, more compact constraints can be obtained when the 1D objects rotate around a fixed point. In the case of planar motion, the estimation of vanishing points is not needed, and the calibration accuracy is significantly improved. Finally, both computer simulations and experiments are performed to validate the effectiveness and robustness of our algorithms.

Optimization reconstruction method of object profile using flexible laser plane and bi-planar references

An optimization method to reconstruct the object profile is performed by using a flexible laser plane and bi-planar references. The bi-planar references are considered as flexible benchmarks to realize the transforms among two world coordinate systems on the bi-planar references, the camera coordinate system and the image coordinate system. The laser plane is confirmed by the intersection points between the bi-planar references and laser plane. The 3D camera coordinates of the intersection points between the laser plane and a measured object are initially reconstructed by the image coordinates of the intersection points, the intrinsic parameter matrix and the laser plane. Meanwhile, an optimization function is designed by the parameterized differences of the reconstruction distances with the help of a target with eight markers, and the parameterized reprojection errors of feature points on the bi-planar references. The reconstruction method with the bi-planar references is evaluated by the difference comparisons between true distances and standard distances. The mean of the reconstruction errors of the initial method is 1.01 mm. Moreover, the mean of the reconstruction errors of the optimization method is 0.93 mm. Therefore, the optimization method with the bi-planar references has great application prospects in the profile reconstruction.