Structured Light Sensor Research Articles

Improving 3D Reconstruction Through RGB-D Sensor Noise Modeling

High-resolution RGB-D sensors are widely used in computer vision, manufacturing, and robotics. The depth maps from these sensors have inherently high measurement uncertainty that includes both systematic and non-systematic noise. These noisy depth estimates degrade the quality of scans, resulting in less accurate 3D reconstruction, making them unsuitable for some high-precision applications. In this paper, we focus on quantifying the uncertainty in the depth maps of high-resolution RGB-D sensors for the purpose of improving 3D reconstruction accuracy. To this end, we estimate the noise model for a recent high-precision RGB-D structured light sensor called Zivid when mounted on a robot arm. Our proposed noise model takes into account the measurement distance and angle between the sensor and the measured surface. We additionally analyze the effect of background light, exposure time, and the number of captures on the quality of the depth maps obtained. Our noise model seamlessly integrates with well-known classical and modern neural rendering-based algorithms, from KinectFusion to Point-SLAM methods using bilinear interpolation as well as 3D analytical functions. We collect a high-resolution RGB-D dataset and apply our noise model to improve tracking and produce higher-resolution 3D models.

A high-precision visual detection method for rail profile based on Scheimpflug condition

Rail profile is an important indicator of rail service status, which should be always kept in good condition. How to efficiently collect rail profiles with high precision is a critical issue for their condition-based maintenance. This paper focuses on the method for accurate visual detection of rail full profile (RFP), including optical modeling, visual calibration, and error compensation. The proposed optical unit, featuring double line structured-light sensors, introduces an innovative method under the Scheimpflug condition to avoid the limitation of depth of field in RFP measurement. Subsequently, double visual sensors are calibrated together using the common reference target to extract the internal and external parameters respectively. The dynamic errors resulting from the impact of vehicle vibrations are ultimately investigated to rectify the distorted profiles to the normal. Both laboratory and field tests are performed to verify the effectiveness of the proposed system in terms of precision and efficiency.

High-efficiency automated triaxial robot grasping system for motor rotors using 3D structured light sensor

High-efficiency automated triaxial robot grasping system for motor rotors using 3D structured light sensor

Efficient data acquisition and reconstruction for air-coupled ultrasonic robotic NDE

Non-destructive evaluation of complex parts using surface scanning techniques, such as ultrasonic testing and eddy current testing, requires complex manipulation of such sensors to ensure quantitative results. A robotic arm may function as a complex manipulator for surface scanning, controlling the position and tilt between the probe and specimen’s surface. To ensure accuracy in probe manipulation, accurate geometric information of the specimen is required. This article explores a methodology that uses structured light for physical-to-virtual reconstruction, providing submillimeter scale and accurate surface geometries. Reconstruction aids in path planning through a novel ray-triangle intersection array algorithm, establishing movements for the NDE probe to orient itself on the specimen at a constant probe to specimen surface distance, or lift-off. The proposed technique is demonstrated and validated through experimental air-coupled ultrasonic inspection of automotive CFRP composite samples with simulated flaws such as interlaminar delamination. The proposed method employs guided waves and a pitch-catch configuration of air-coupled ultrasonic probes, enabling single-side access scans. A Fanuc 100ib robot arm was used to manipulate the ultrasonic probes along a sample reconstructed with a CR-Scan 01 structured light sensor. The probes were excited at 200khz from a SonoAir system, while also recovering defect vs background information synchronized with the probe’s orientation. Additionally, a framework for potential automation is proposed, with further details to be explored in future works.

Hand-eye calibration of line structured-light sensor based on double-sphere constraints

Aiming at the complex problem of solving the positional relationship between the line laser sensor at the end of the robot wrist and the coordinate system of the end flange, a line laser hand-eye calibration method based on a double ball fixed target is proposed. Firstly, a double standard ball with known radius and center distance is used as a reference tool, and the point cloud data of a single contour line is collected several times, and the coordinates of the two ball centers under the camera coordinate system can be obtained according to the center coordinates of the circle obtained from the fitted circle combined with the Pythagoras theorem. Then, according to the principle that the two spherical centers are fixed under the robot base coordinate system, the rigid transformation matrix between the line laser sensor and the robot is successively solved. When solving the rotation matrix, the constraint of the distance between the spherical centers of the two balls is introduced to obtain a solution with better accuracy. Experimental results show that the method has better overall performance compared with the traditional single-ball hand-eye calibration method.

Robust Rail-Track Section Identification With Multiple Structured Light Sensors and Kernel-Based Belief Sensor-Credibility Evaluation

Robust Rail-Track Section Identification With Multiple Structured Light Sensors and Kernel-Based Belief Sensor-Credibility Evaluation

Hand-eye calibration of line structured-light sensor by scanning and reconstruction of a free-placed standard cylindrical target

This paper proposed a hand-eye calibration method for a line structured-light (LSL) scanner mounted on industrial robot. Unlike widely used spherical targets, the target used in our work is a cylinder, which is inexpensive and easy to be manufactured in larger size to cover a wider calibration space. In order to address the profile self-occlusion and drastically shape changing of conic section during calibration, an unconstrained dimensionality reduction method is proposed to fit the scanned profiles. Based on fitted ellipse centers under various robot poses, the hand-eye calibration model is developed based on the difference between cylinder radius and the distance from the profile point to the cylinder axis. Simulation and actual experiments have shown that the proposed ellipse fitting method is more accurate and robust, and the calibration method achieves higher reconstruction accuracy under the same robot positioning error, need less robot operations to obtain more accurate hand-eye parameters.

The Automatic Algorithm of Optimizing the Position of Structured Light Sensors

Optical 3D detection technology has a wide range of applications in industrial detection, agricultural production, and so on. Its advantages are non-contact, efficiency, and high precision. However, the specular reflection problem affects model coverage and measurement accuracy. An optimization algorithm for calculating the number and pose of sensors has been proposed to address this issue. First, the specular reflection problem is viewed as a multi-sensor position search problem. Then, an optimization algorithm is used to find the optimal number and bit positions of sensors to avoid specular reflection. The experiment shows that the optimization results of this algorithm can cover the area to be measured with the least number of sensor combinations while avoiding the influence of specular reflection.

Deep panoramic depth prediction and completion for indoor scenes

We introduce a novel end-to-end deep-learning solution for rapidly estimating a dense spherical depth map of an indoor environment. Our input is a single equirectangular image registered with a sparse depth map, as provided by a variety of common capture setups. Depth is inferred by an efficient and lightweight single-branch network, which employs a dynamic gating system to process together dense visual data and sparse geometric data. We exploit the characteristics of typical man-made environments to efficiently compress multi-resolution features and find short- and long-range relations among scene parts. Furthermore, we introduce a new augmentation strategy to make the model robust to different types of sparsity, including those generated by various structured light sensors and LiDAR setups. The experimental results demonstrate that our method provides interactive performance and outperforms state-of-the-art solutions in computational efficiency, adaptivity to variable depth sparsity patterns, and prediction accuracy for challenging indoor data, even when trained solely on synthetic data without any fine tuning.

Exposure time and point cloud quality prediction for active 3D imaging sensors using Gaussian process regression

Setting an optimal image exposure is crucial for acquiring dense point clouds using 3D active optical sensor systems such as structured light sensors [structured light sensors (SLSs)] and active stereo sensors. One of the most common and seamless ways to optimize the image brightness of an image exposure is to adjust the camera’s exposure time. However, optimizing the image exposure alone is ineffective for acquiring surfaces of large-scale objects with a complex topology if a spatial understanding of the scene is neglected. Hence, the present paper proposes a data-driven approach using two Gaussian processes [Gaussian processes (GPs)] regression models to select a proper exposure time considering the nonlinear correlations between image exposure and the scene spatial relationships. To model these correlations, our study introduces first the generic synthesization of seven inputs and two target variables. Then, based on these inputs, two independent GPs are designed: one for predicting the measurement quality and one for estimating the exposure time. The performance and generalizability of both models are thoroughly evaluated using an SLS and an active stereo sensor. The evaluation demonstrated that the point cloud quality models adequately matched observations with an R2 exceeding 90%. Specifically, the models predicted point cloud quality with an root mean square error (RMSE) of 10%. Additionally, the assessment of the performance of the exposure time models showed a model fit with an R2 above 97%. The exposure time prediction accuracy, as evidenced by the RMSE values, was within 10% of the corresponding exposure time range for each sensor. The present research shows the potential and effectiveness to completely automate the assessment of a point cloud quality and the selection of exposure times with the help of data-driven models.

Measurement of Three-Dimensional Back Shape of Normal Adults Using a Novel Three-Dimensional Imaging Mobile Surface Topography System (MSTS): An Intra- and Inter-Rater Reliability Study

Postural and spinal deformities are major contributing factors to musculoskeletal (MSK) disorders. Posture screening and assessment can help to identify early morphological deformities, thereby preventing progression and reducing or correcting them with effective treatments. The study evaluates both intra- and inter-repeatability of using a mobile structured light sensor with a structured light pattern for building an accurate 3D human model and its use in postural screening. 16 young males (age: 25 ± 5.6 years, height: 172 ± 5.3 cm, mass: 69 ± 8.6 kg) participated without any musculoskeletal pain or pre-existing leg or spinal abnormalities. An iPad-based 3D mobile scanning tool, Structure SensorTM (2018 version), was used to capture the participants’ back and whole-body shape. The collected data (3D model) were realigned and processed in the open-source software, Netfabb BasicTM (7.2 version). For each participant, five trained raters individually measured three trials of standing back and body posture on two separate occasions to calculate both intra- and inter-rater reliability. With the use of this software, nine postural variables and angular displacements were individually measured by the raters. The results indicated good to excellent intra-rater and good to moderate inter-rater reliability for measuring 78% (7 out of 9) of postural variables with an ICC ranging from 0.70 to 0.98. The remaining 22% of variables (2 out of 9; lateral pelvic tilt and right frontal knee angle) showed moderate to low inter- and intra-rater reliability, with ICCs ranging from 0.26 to 0.79.

A lightweight efficient semantic segmentation with encoder-decoder for arc interference in robotic arc welding

Recognizing laser stripes under conditions of arc interference is a persistent challenge faced by linear structured light sensors due to their similarity with noise in morphological characteristics. In this paper, a symmetrical encoder-decoder semantic segmentation model named LSRNet (lightweight stripe recognition network) is proposed that enables real-time detection of laser stripes in environments with strong arc interference. It analyzes the difference between arc and laser stripe from the semantic point of view. Firstly, to facilitate quick and accurate recognition of laser stripes, we designed a Depthwise-separable Factorized Module that is both efficient and scalable with a larger receptive field. Moreover, due to the difficulty in distinguishing laser stripes from arcs in a local context, a Multi-scale Depthwise-separable Factorized Module is designed to capture more context information. Secondly, the Unified Attention Fusion Module is used to enhance the feature representation of the laser stripe. Additionally, The combination of DiceLoss(dice coefficient loss) and CELoss(cross-entropy loss) is introduced to optimize the loss, which addresses the pixel imbalance challenge that the pixels of the laser stripe are much less than the background pixels. Finally, based on the images collected in the industrial field, our ArcDataset is created for model training. Experimental results show that the proposed method achieves the highest intersection over union of 88.0% in laser stripe segmentation and a recognition speed of 101.3 FPS(frames per second).

Optical Characterization of Materials for Precision Reference Spheres for Use with Structured Light Sensors.

Traditionally, 3D digitizing sensors have been based on contact measurement. Given the disadvantages of this type of measurement, non-contact sensors such as structured light sensors have gained the attention of many sectors in recent years. The fact that their metrological performance is affected by the optical properties of the digitized material, together with the lack of standards, makes it necessary to develop characterization work to validate materials and calibration artifacts for the qualification and calibration of these sensors. This work compares and optically characterizes different materials and surface finishes of reference spheres used in the calibration of two structured light sensors with different fields of application, with the aim to determine the most suitable sphere material-sensor combination in each case. The contact measurement system of a CMM is used as a reference and, for the processing of the information from the sensors, the application of two different filters is analyzed. The results achieved point to sandblasted stainless steel spheres as the best choice for calibrating or qualifying these sensors, as well as for use as registration targets in digitizing. Tungsten carbide spheres and zirconium are unsuitable for this purpose.

Surface structured light sensor with simultaneous color mapping

To obtain geometric information and color texture simultaneously, a surface structured light sensor consisting of a monochrome camera, a color camera, and a projector is proposed. The sensor uses a color camera to acquire surface color information while using it as a geometric measurement unit and matching with the monochrome camera to obtain geometric information. Due to the Bayer array and demosaicing algorithm of the color camera, pixel RGB components are always coupled with interference from other channels. However, existing color de-crosstalk in reconstruction is merely applied to the decoupling of color composite patterns, ignoring the intensity errors present in color fringe patterns under monochrome illumination. In our sensor, de-crosstalk of monochromatic patterns is considered to guarantee the reconstruction accuracy. The high-accuracy measurement of the sensor is validated by reconstructing standard steps, yielding a mean absolute error of 0.008 mm for distance measurements. In addition, the reconstruction experiment of a terracotta warrior verifies that the proposed sensor has potential application in the digital preservation of cultural relics.

Three dimensional reconstruction and measurement of underwater spent fuel assemblies

It is an important work to measure the dimensions of underwater spent fuel assemblies in the nuclear power industry during the overhaul, to judging whether the spent fuel assemblies can continue to be used. In this paper, a three dimensional reconstruction method for underwater spent fuel assemblies of nuclear reactor based on linear structured light is proposed, and the topography and size measurement was carried out based on the reconstructed 3D model. Multiple linear structured light sensors are used to obtain contour size data, and the shape data of the whole spent fuel assembly can be collected by one-dimensional scanning motion. In this paper, we also presented a corrected model to correct the measurement error introduced by lead-glass and water is corrected. Then, we set up an underwater measurement system for spent fuel assembly based on this method. Finally, an underwater measurement experiment is carried out to verify the 3D reconstruction ability and measurement ability of the system, and the measurement error is less than ±0.05 mm.

An automatic calibration algorithm for endoscopic structured light sensors in cylindrical environment

ABSTRACT Structured light sensing systems, as one of the most common optical-based nondestructive evaluation techniques, have been widely applied for inline pipeline inspection. The sensor can be inserted inside the pipe to generate 3D visualisation and evaluate the cracks in the materials. The precise calibration of the camera-projector measurement system is of great significance to ensure the measurement accuracy of the 3D sensing system. Conventional calibration methods for structured light sensors involve complicated and time-consuming procedures and are easily affected by ambient light. The paper presents a novel algorithm to automatically calibrate the projection module and estimate the stereo parameters between the camera and the projector. The calibration algorithm exploits the cylindrical nature of the inspected pipe to create a set of geometric constraints and automatically calibrate the sensor without the need for reference calibration points. Experimental and simulation results showed that the algorithm could successfully estimate the projector’s intrinsic and extrinsic parameters by simply acquiring the data inside a cylindrical pipe with a known diameter. The proposed algorithm highly reduces the data collection time for the calibration (only 53 s), improves the accuracy, and simplifies the calibration process.

Research on Welding Guidance System of Intelligent Perception for Steel Weldment

In order to achieve intelligent and flexible welding, complete the extraction and path planning of 3-D complex weld seam of large steel weldment, and solve the problems that in the vision-guided welding process, the scanning path depends on manual teaching, and the offline programming depends on the machining and assembly accuracy of the weldment; a vision-guided method for welding robots based on a line structured-light sensor is proposed. First, the 3-D computer aided design (CAD) model of the weldment is used to build an offline welding knowledge library, and then, the scanning path is planned in combination with the point cloud alignment. Then, the sensor collects the point cloud on the surface of the weldment along the scanning path. Second, different weld seam extraction algorithms are proposed according to different weld types. The random sample consensus (RANSAC) algorithm is used to identify the position of weld feature points, and the three-frame method is used to solve the pose of weld feature points. Finally, the weld seam is generated to guide the welding robot to complete the welding operation. The experimental results show that the proposed method can be used for different types and dimensions of weldments and provides flexible welding operations with better versatility and higher intelligence.

Model-free cable robot control

This paper proposes a technique to control a cable robot in the total absence of a model and its parameters. The cable robot is actuated by three motors whose data, including exact positions, pulley diameters, and nominal cable length, are unknown. We just assume to have a very rough knowledge of lower and upper bounds for the partial derivatives of the relation between the cable lengths and the end-effector space coordinates. A structured-light sensor measures the end-effector position, and the goal is to drive it to a designated point. An algorithm is proposed with guaranteed convergence based on the so-called model-free plant tuning approach. No learning stage is required. Experimental results are reported.

An inspection method of rail head surface defect via bimodal structured light sensors

An inspection method of rail head surface defect via bimodal structured light sensors

Ergonomic risk factors and work-related musculoskeletal disorders in clinical physiotherapy.

The purpose of this study was to objectively quantify and evaluate the ergonomic risk of clinical physiotherapy practices and evaluate physiotherapists for work-related musculoskeletal disorders and pain. Twenty-nine physiotherapists in the rehabilitation department of a large-scale tertiary hospital were recruited in this study. The sampling period lasted for 2 weeks for each physiotherapist and interval sampling was adopted to avoid duplication of cases. Therapist posture during physiotherapy was captured, tracked and analyzed in real time using structured light sensors with an automated assessment program. The quantification of ergonomic risk was based on REBA (Rapid Entire Body Assessment) and the RPE (perceived physical exertion) scores of the therapists were recorded before and after treatment, respectively. Two hundred and twenty-four clinical physiotherapy cases were recorded, of which 49.6% were high risk and 33% were very high risk, with none of the cases presenting negligible risk. The positioning (p < 0.001) of physiotherapist had a considerable impact on ergonomic risk and pediatric physiotherapy presented a higher risk to physiotherapists than adults (p < 0.001). The RPE score of physiotherapist after performing physiotherapy was greater than before physiotherapy and was positively correlated with the REBA distribution. Our study creates an automatic tool to assess the ergonomic risk of physiotherapy practices and demonstrates unacceptable ergonomic risk in common practices. The high prevalence of musculoskeletal disorders and pains recommends that rehabilitation assistance devices should be optimized and standard ergonomic courses should be included in physiotherapists' training plans.