Structured Light Vision System Research Articles

Two-phase visual servoing for capturing tumbling non-cooperative satellites with a space manipulator

In this paper, a visual servoing approach is developed to capture the docking rings of tumbling non-cooperative satellites with a space manipulator. The primary challenge addressed is the potential for the docking ring to leave the monocular camera’s field-of-view as the manipulator approaches the target, due to the ring’s large size. To solve this issue, a two-phase visual servoing scheme combining a monocular camera and a three-line structured light vision system is proposed. In an effort to augment the success rate and safety of capture operations, several constraints are formulated, encompassing manipulator’s kinematics, monocular camera’s field-of-view, obstacle avoidance, structured light’s breakpoints and smooth capture. Subsequently, a nonlinear model predictive controller is proposed to manage these constraints in real-time and regulate the system. System models are established based on image moments and pose for each phase, selecting these features as visual feedback to simplify the formulation of servo constraints and avoid the complex circle-based pose measurement. Furthermore, to ensure unbiased predictions, the model disturbances arising from the imprecise estimation of target motion parameter are observed using an extended Kalman filter, which are then incorporated into the predictive control framework. The simulation results demonstrate the effectiveness of this scheme.

Improving Robotic Bin-Picking Performances through Human–Robot Collaboration

The automation of bin-picking processes has been a research topic for almost two decades. General-purpose equipment, however, still does not show adequate success rates to find application in most industrial tasks. Human–robot collaboration in bin–picking tasks can increase the success rate by exploiting human perception and handling skills and the robot ability to perform repetitive tasks. The aim of this paper, starting from a general-purpose industrial bin picking equipment comprising a 3D–structured light vision system and a collaborative robot, consists in enhancing its performance and possible applications through human–robot collaboration. To achieve successful and fluent human–robot collaboration, the robotic workcell must meet some hardware and software requirements that are defined below. The proposed strategy is tested in some sample tests: the results of the experimental tests show that collaborative functions can be particularly useful to overcome typical bin picking failures and to improve the fault tolerance of the system, increasing its flexibility and reducing downtimes.

Development, Calibration, and Image Processing of Underwater Structured Light Vision System: A Survey

The underwater structured light vision system has the characteristics of a large amount of information, high measurement accuracy, and good robustness, which has attracted the attention of researchers and is widely used in the underwater 3-D measurement. In this article, the state-of-the-art techniques of the underwater structured light vision system are comprehensively summarized, which provides valuable guidance for relevant researchers. First, underwater structured light vision systems are classified and introduced, and their advantages and disadvantages are analyzed. Then, the calibration methods of the underwater structured light vision system are introduced, including underwater camera calibration, underwater laser calibration, and underwater projector calibration. Besides, the image processing methods of the underwater structured light vision system are analyzed, including laser structured light image processing and coded structured light image processing. Moreover, the applications of the underwater structured light vision system in underwater archeology, seabed mapping, and underwater structure inspection are introduced. On this basis, the research directions of the underwater structured light vision system are prospected.

An Underwater Structured Light Vision Calibration Method Considering Unknown Refractive Index Based on Aquila Optimizer

Underwater structured light vision systems have attracted the attention of researchers because of their high measurement accuracy and robustness. At present, most underwater structured light vision measurement systems consider the refractive index as a known parameter. However, the refractive index of water will change because of environmental changes such as salinity and temperature. Inaccurate refractive index will have serious influence on the measurement accuracy of the underwater structured light vision system. Therefore, an underwater structured light vision calibration method considering unknown refractive index is proposed. To avoid complex calibration process and complex solution process of a nonlinear underwater structured light vision model, a novel checkerboard usage calibration method based on Aquila Optimizer (AO) is proposed. The calibration solution process is regarded as a nonlinear optimization problem, and the optimization objective is established with the distance between the feature points intersected by the checkerboard and the laser. Finally, calibration and measurement experiments are carried out to verify the effectiveness and superiority of the proposed method.

A Three-Dimensional Structured Light Vision System by Using a Combination of Single-Line and Three-Line Lasers

A multi-line structured light measurement method that combines a single-line and a three-line laser, in which precision sliding rails and displacement measurement equipment are not required, is proposed in this paper. During the measurement, the single-line structured light projects onto the surface of an object and the three-line structured light remains fixed. The single-line laser is moved and intersects with the three-line laser to form three intersection points. The single-line light plane can be solved using the camera coordinates of three intersection points, thus completing the real-time calibration of the scanned light plane. The single-line laser can be scanned at any angle to determine the overall complete three-dimensional (3D) shape of the object during the process. Experimental results show that this method overcomes the difficulty of obtaining information about certain angles and locations and can effectively recover the 3D shape of the object. The measurement system's repetition error is under 0.16 mm, which is sufficient to measure the 3D shapes of complicated workpieces.

Improvement of calibration method for multi-camera line structured light vision system

Improvement of calibration method for multi-camera line structured light vision system

A fully automatic surgical registration method for percutaneous abdominal puncture surgical navigation

Surgical registration that maps surgical space onto image space plays an important role in surgical navigation. Accurate surgical registration can help surgeons efficiently locate surgical instruments. The complicated marker-based surgical registration method is highly accurate, but it is time-consuming. Therefore, a marker-less surgical registration method with high-precision and high-efficiency is proposed without human intervention. Firstly, the surgical navigation system based on the multi-vision system is calibrated by using a specially-designed calibration board. When extracting the abdominal point cloud acquired by the structured light vision system, the constraint is constructed by using Computed Tomography (CT) image to filter out the points in irrelevant areas to improve the computational efficiency. The Coherent Point Drift (CPD) algorithm based on Gaussian Mixture Model (GMM) is applied in the registration of abdominal point cloud with lack of surface features. To enhance the efficiency of the CPD algorithm, firstly, the system calibration result is used in rough registration of the point cloud, and then the proper point cloud pretreatment method and its parameters are studied through experiments. Finally, the puncturing simulation experiments were carried out by using the abdominal phantom. The experimental results show that the proposed surgical registration method has high accuracy and efficiency, and has potential clinical application value.

Research on rail end face slope algorithm based on Bisquare fitting

The slope of rail end face needs to be measured by the base staffs before the rail is welded. Most of them are trained to manually measure, using a feeler gauge and an angle ruler. The measuring steps are complicated, making the whole process less than efficient and impossible to be applied at an automated pipeline inspection. This paper proposes a new type of non-contact rail end face slope measurement method, based on structured light vision system. Two structural light sensors are placed on the mechanical platform, specifically on the left and bottom of the rail, to obtain the contour data of the end and bottom surfaces. The data is transmitted to the host computer, while all contour data is converted into the same 3D world coordinate system. Next, the data is processed, using Bisquare fitting method. Finally, the end face slope result is obtained, by combining the mathematical descriptions of the end face slope. Experimental tests are conducted, proving that the proposed method of measuring the slope of the end face shows good stability and accuracy, while it can be used to complete the slope inspection of the rail end face, meeting the actual testing requirements.

Dimension Measurement and Key Point Detection of Boxes through Laser-Triangulation and Deep Learning-Based Techniques

Dimension measurement is of utmost importance in the logistics industry. This work studies a hand-held structured light vision system for boxes. This system measures dimension information through laser triangulation and deep learning using only two laser-box images from a camera and a cross-line laser projector. The structured edge maps of the boxes are detected by a novel end-to-end deep learning model based on a trimmed-holistically nested edge detection network. The precise geometry of the box is calculated by the 3D coordinates of the key points in the laser-box image through laser triangulation. An optimization method for effectively calibrating the system through the maximum likelihood estimation is then proposed. Results show that the proposed key point detection algorithm and the designed laser-vision-based visual system can locate and perform dimension measurement of measured boxes with high accuracy and reliability. The experimental outcomes show that the system is suitable for portable automatic box dimension online measurement.

Three-line structured light vision system for non-cooperative satellites in proximity operations

Adapter ring is a commonly used component in non-cooperative satellites, which has high strength and is suitable to be recognized and grasped by the space manipulator. During proximity operations, this circle feature may be occluded by the robot arm or limited field of view. Moreover, the captured images may be underexposed when there is not enough illumination. To address these problems, this paper presents a structured light vision system with three line lasers and a monocular camera. The lasers project lines onto the surface of the satellite, and six break points are formed along both sides of the adapter ring. A closed-form solution for real-time pose estimation is given using these break points. Then, a virtual structured light platform is constructed to simulate synthetic images of the target satellite. Compared with the predefined camera parameters and relative positions, the proposed method is demonstrated to be more effective, especially at a close distance. Besides, a physical space verification system is set up to prove the effectiveness and robustness of our method under different light conditions. Experimental results indicate that it is a practical and effective method for the pose measurement of on-orbit tasks.

Dual surface structured light vision system based on multidimensional parameter coding.

A dual surface structured light vision system based on the theory of multidimensional parameter coding is proposed in this paper that can provide a new high-performance and adaptive encoding robot vision perception system. Specifically, the dual surface structured light consists of an auxiliary light source and a principal structured light. Scene images from the auxiliary structured light determine the parameters of the principal structured light. The vision process system can adaptively encode the principal structured light according to types and numbers of parameters. The image formed by the encoded principal structured light can reflect the depth change details of the target object and guide the robot to locate the target object quickly and accurately. The simulation and industrial experiments show that the dual surface structured light vision system has wider application scenarios and higher accuracy than the traditional structured light vision system. According to different industrial scenes, the positioning accuracy of the vision system can reach 0.1mm to 20μm in 0.8s.

A robust weld seam recognition method under heavy noise based on structured-light vision

Abstract Structured-light vision systems are widely used in robotic welding. The key to improving the robotic visual servo performance and weld quality is the weld seam recognition accuracy. Common detection algorithms are likely to be disturbed by the noise of spatter and arc during the welding process. In this paper, a weld seam recognition algorithm is proposed based on structured light vision to overcome this challenge. The core of this method is fully utilizing information of previous frames to process the current frame, which can make weld seam extraction both more robust and effective. The algorithm can be divided into three steps: initial laser center line recognition, online laser center line detection, and weld feature extraction. A Laplacian of Gaussian filter is used for recognizing the laser center line in the first frame. Afterwards, an algorithm based on the NURBS-snake model detects the laser center line online in a dynamic region of interest (abbreviated ROI). The center line obtained from first step is set as the initial contour of the NURBS-snake model. Using the line obtained from the previous step, feature points are determined by segmentation and straight-line fitting, while the position of the weld seam can be calculated according to the feature points. The accuracy, efficiency and robustness of the recognition algorithm are verified by experiments.

Flexible Three-Dimensional Reconstruction via Structured-Light-based Visual Positioning and Global Optimization.

Three-dimensional (3D) reconstruction using line structured light vision system commonly cooperates with motion restraint devices, such as parallel guide rail push-broom devices. In this study, we propose a visual positioning method to eliminate the motion constraint. An extended orthogonal iteration algorithm for visual positioning is proposed to obtain the precise position of the line structured light binocular camera system during movement. The algorithm uses the information acquired by the binocular camera, and produces a better positioning accuracy than the traditional vision localization algorithm. Furthermore, a global optimization method is proposed to calculate the poses of the camera relative to the world coordinate system at each shooting position. This algorithm effectively reduces the error accumulation and pose drift during visual positioning, and 3D information of the surface can be measured via the proposed free-moving line structured light vision system. The simulation and physical experiments performed herein validate the proposed method and demonstrate the significant improvement in the reconstruction accuracy: when the test distance is 1.5 m, the root mean square error of the point cloud is within 0.5 mm.

线结构光视觉系统标定新法及其振动测控应用

线结构光视觉系统标定新法及其振动测控应用

A High-Speed Seam Extraction Method Based on the Novel Structured-Light Sensor for Arc Welding Robot: A Review

In the modern manufacturing, the teaching-playback mode and the off-line programming mode of the welding robots still play the important roles. However, these modes cannot meet the autonomous and adaptive capabilities of the welding robots. To improve the flexibility of the welding robots, fast and accurate seam extraction is the key link to realize the intelligent welding robots. A fast and accurate seam extraction algorithm based on the novel structured light vision system is proposed in this paper. First, the digital light processing projector is used to construct the multi-function structured light vision sensor in this paper. It could generate different pattern images to adapt different welding tasks. Second, during the current research work about seam extraction, the seam extraction algorithm based on morphological image processing has been widely used and improved. However, these algorithms include much machine vision algorithms and the speed of these algorithms is too slow to meet real-time requirements. Meanwhile, these algorithms are difficult to adapt to different types of weld seam. In order to achieve fast and accurate extraction of various types of weld seams, the target tracking algorithm based on the kernelized correlation filters algorithm is applied into this paper. Experimental results show that the proposed method could well realize fast and accurate seam extraction of various types of weld seams.

Calibration of Structured Light Vision System using Multiple Vertical Planes

Calibration of Structured Light Vision System using Multiple Vertical Planes

A flexible and high precision calibration method for the structured light vision system

Calibration is the process of estimating the intrinsic and extrinsic parameters of the device. In the field of photogrammetry, structured light measurement system is one of the frequently used active vision measurement systems. The accuracy of the estimated parameters of the system determines the measurement results in the later experiments and applications. Traditional methods were often not easy to implement and time-consuming. In this paper, a flexible and accurate method for calibrating the structured light system and a hybrid pattern which takes both the advantages of time multiplexing method and spatial neighborhood method is proposed. To make full use of the available open source of the camera calibration toolbox, this method treats the projector as a virtual camera. By analyzing the projected patterns and establishing the relationship between the projector and the camera, the projector of the structured light system is able to “capture” images on its own and thus can be calibrated easily and accurately like the usual camera as well. The experimental results show that the intrinsic and extrinsic parameters of the structured light system were estimated with high precision.

Multi-camera and structured-light vision system (MSVS) for dynamic high-accuracy 3D measurements of railway tunnels.

Railway tunnel 3D clearance inspection is critical to guaranteeing railway operation safety. However, it is a challenge to inspect railway tunnel 3D clearance using a vision system, because both the spatial range and field of view (FOV) of such measurements are quite large. This paper summarizes our work on dynamic railway tunnel 3D clearance inspection based on a multi-camera and structured-light vision system (MSVS). First, the configuration of the MSVS is described. Then, the global calibration for the MSVS is discussed in detail. The onboard vision system is mounted on a dedicated vehicle and is expected to suffer from multiple degrees of freedom vibrations brought about by the running vehicle. Any small vibration can result in substantial measurement errors. In order to overcome this problem, a vehicle motion deviation rectifying method is investigated. Experiments using the vision inspection system are conducted with satisfactory online measurement results.

A flexible calibration method using the planar target with a square pattern for line structured light vision system.

A flexible calibration approach for line structured light vision system is proposed in this paper. Firstly a camera model is established by transforming the points from the 2D image plane to the world coordinate frame, and the intrinsic parameters of camera can be obtained accurately. Then a novel calibration method for structured light projector is presented by moving a planar target with a square pattern randomly, and the method mainly involves three steps: first, a simple linear model is proposed, by which the plane equation of the target at any orientations can be determined based on the square’s geometry information; second, the pixel coordinates of the light stripe center on the target images are extracted as the control points; finally, the points are projected into the camera coordinate frame with the help of the intrinsic parameters and the plane equations of the target, and the structured light plane can be determined by fitting these three-dimensional points. The experimental data show that the method has good repeatability and accuracy.

High-Frame-Rate Structured Light 3-D Vision for Fast Moving Objects

<div class=""abs_img""><img src=""[disp_template_path]/JRM/abst-image/00260003/04.jpg"" width=""300"" />HFR 3D vision system</span></div> This paper presents a fast motion-compensated structured-light vision system that realizes 3-D shape measurement at 500 fps using a high-frame-rate camera-projector system. Multiple light patterns with an 8-bit gray code, are projected on the measured scene at 1000 fps, and are processed in real time for generating 512 × 512 depth images at 500 fps by using the parallel processing of a motion-compensated structured-light method on a GPU board. Several experiments were performed on fast-moving 3-D objects using the proposed method. </span>