Optical 3D Sensor Research Articles

Highly stable lead-free perovskite glass for real-time 3D surface temperature measurement across the wide temperature range

3D Optical temperature sensors are highly valuable in various fields such as industry, medicine, and aerospace, as they allow for real-time detection of temperature distribution on surfaces. However, existing temperature sensor materials face challenges in achieving accurate detection across a wide range of temperatures from low to high. To overcome these issues, Cs2Zn1-xEuxBr4 perovskite B2O3–BaF2-Ga2O3 (BBG) borate glass was prepared for non-contact 3D optical temperature sensing. Cs2Zn1-xEuxBr4 perovskite B2O3–BaF2-Ga2O3 (BBG) borate glass improves the stability, thus expanding the temperature measurement range. Utilizing the fluorescence intensity ratio (FIR) method, based on the 613 nm (Eu3+: 5D0→7F2) and 592 nm (Eu3+: 5D0→7F1) emission bands, a FIR benchmark database was constructed for the entire temperature range from 10 K to 640 K. Temperature calculation equations related to FIR were derived, with a maximal relative sensitivity of 0.08 % K−1 at 180 K. The method for measuring the surface temperature of 3D curved aircraft is based on the rare earth perovskite borate glass. The temperature measured with this method is close to the thermal imaging temperature and has a good temperature measuring effect. The temperature measurement range is wider and more advantageous than previously reported ranges. These results indicate that Cs2Zn1-xEuxBr4-BBG perovskite borate glass temperature sensors have broad application prospects in many applications.

Infrared-based temperature measurement in three dimensions

A highly accurate and low-cost mobile platform for multiple object temperature measurement in 3D coordinate positions is introduced. The key idea relies on not only a combination of a 3D optical depth sensor and the 2D infrared thermal imager but also a determined distant-dependent compensation model. As a result, a cost-effective yet wider (>100.0∘C) temperature measurement range under the fluctuation of working distant shift is realized with a low standard deviation of 0.16°C at the working distance of 1.0 m. Accurate temperature measurement at different positions of objects along the suitable working distance is also demonstrated.

A novel global calibration method for vision measurement system based on mirror-image stereo target

Using three-dimensional (3D) targets to solve the transformation relationship between the local and global coordinate systems is a common global calibration method. However, 3D targets have complex structures and high costs. And multiple coordinate transformations during global calibration can cause significant cumulative errors. Therefore, based on reflection imaging, this paper designs a new mirror-image stereo target (MST) and analyzes its view area and three structural parameter constraints. A new global calibration method is proposed, as well as three optimization algorithms for pose solution between MST, optical 3D sensor and global camera to improve the global calibration accuracy. The experimental results show that the registration errors of two workpieces are 0.086 mm and 0.114 mm, respectively, confirming the feasibility and precision of the proposed method. Our method is flexible, requiring only a planar target and a mirror, rather than complex 3D targets, to achieve global calibration with non-overlapping field of view.

Exposure time and point cloud quality prediction for active 3D imaging sensors using Gaussian process regression

Setting an optimal image exposure is crucial for acquiring dense point clouds using 3D active optical sensor systems such as structured light sensors [structured light sensors (SLSs)] and active stereo sensors. One of the most common and seamless ways to optimize the image brightness of an image exposure is to adjust the camera’s exposure time. However, optimizing the image exposure alone is ineffective for acquiring surfaces of large-scale objects with a complex topology if a spatial understanding of the scene is neglected. Hence, the present paper proposes a data-driven approach using two Gaussian processes [Gaussian processes (GPs)] regression models to select a proper exposure time considering the nonlinear correlations between image exposure and the scene spatial relationships. To model these correlations, our study introduces first the generic synthesization of seven inputs and two target variables. Then, based on these inputs, two independent GPs are designed: one for predicting the measurement quality and one for estimating the exposure time. The performance and generalizability of both models are thoroughly evaluated using an SLS and an active stereo sensor. The evaluation demonstrated that the point cloud quality models adequately matched observations with an R2 exceeding 90%. Specifically, the models predicted point cloud quality with an root mean square error (RMSE) of 10%. Additionally, the assessment of the performance of the exposure time models showed a model fit with an R2 above 97%. The exposure time prediction accuracy, as evidenced by the RMSE values, was within 10% of the corresponding exposure time range for each sensor. The present research shows the potential and effectiveness to completely automate the assessment of a point cloud quality and the selection of exposure times with the help of data-driven models.

Kinect-Based Assessment of Lower Limbs during Gait in Post-Stroke Hemiplegic Patients: A Narrative Review.

The aim of this review was to present an overview of the state of the art in the use of the Microsoft Kinect camera to assess gait in post-stroke individuals through an analysis of the available literature. In recent years, several studies have explored the potentiality, accuracy, and effectiveness of this 3D optical sensor as an easy-to-use and non-invasive clinical measurement tool for the assessment of gait parameters in several pathologies. Focusing on stroke individuals, some of the available studies aimed to directly assess and characterize their gait patterns. In contrast, other studies focused on the validation of Kinect-based measurements with respect to a gold-standard reference (i.e., optoelectronic systems). However, the nonhomogeneous characteristics of the participants, of the measures, of the methodologies, and of the purposes of the studies make it difficult to adequately compare the results. This leads to uncertainties about the strengths and weaknesses of this technology in this pathological state. The final purpose of this narrative review was to describe and summarize the main features of the available works on gait in the post-stroke population, highlighting similarities and differences in the methodological approach and primary findings, thus facilitating comparisons of the studies as much as possible.

Facial Asymmetry Detected with 3D Methods in Orthodontics: A Systematic Review

Background: Historically, the development of two-dimensional (2D) imaging techniquesforerun that of three-dimensional (3D) ones. Some 2D methods are still considered valid and effective to diagnose facial asymmetry but 3D techniques may provide more precise and accurate measurements. Objective: The aim of this work is to analyze the accuracy and reliability of the imaging techniques available for the diagnosis of facial asymmetry in orthodontics and find the most reliable. Methods: A search strategy was implemented using PubMed (National Library of Medicine, NCBI). Results: A total of 3201 papers were identified in electronic searches. 90 articles, available in full text, were included in the qualitative synthesis consisting of 8 reviews on the diagnosis of facial asymmetry, 22 in vivo and in vitro studies on 2D methods and 60 in vivo and in vitro studies on 3D methods to quantify the asymmetry. Conclusion: 2D techniques include X-ray techniques such as posterior-anterior cephalogram, which still represents the first level exam in the diagnosis of facial asymmetry. 3D techniques represent the second level exam in the diagnosis of facial asymmetry. The most current used techniques are CBCT, stereophotogrammetry, laser scanning, 3D optical sensors and contact digitization. The comparison between bilateral parameters (linear distances, angles, areas, volumes and contours) and the calculation of an asymmetry index represent the best choices for clinicians who use CBCT. The creation of a color-coded distance map seems to represent the most accurate, reliable and validated methods for clinicians who use stereophotogrammetry, laser scanning and 3D optical sensors.

Optically smooth and optically rough surface in the view of optical 3D sensors based on imaging

An optical 3D sensor is a device intended to measure the geometric shape of objects using optical methods. When designing an optical 3D sensor, it is necessary to consider whether the surface to be measured is smooth or rough. This is because the signal formation process is different for each of the two cases. Moreover, it turns out that the signal formation process is not only influenced by the roughness of the measured surface but also by other parameters of both the measured surface and the optical measuring system. The additional parameter of the measured surface is the surface correlation length. The parameters of the optical measuring system are the wavelength of the light used and the radius of the resolution cell. Based on the knowledge of the parameter values of the surface to be measured and the measuring system, the surface to be measured can be classified as optically smooth or optically rough. We present an analysis to determine whether a given surface measured by a given optical measuring system will behave as an optically smooth or optically rough surface.

Metrological Characterization and Comparison of D415, D455, L515 RealSense Devices in the Close Range.

RGB-D cameras are employed in several research fields and application scenarios. Choosing the most appropriate sensor has been made more difficult by the increasing offer of available products. Due to the novelty of RGB-D technologies, there was a lack of tools to measure and compare performances of this type of sensor from a metrological perspective. The recent ISO 10360-13:2021 represents the most advanced international standard regulating metrological characterization of coordinate measuring systems. Part 13, specifically, considers 3D optical sensors. This paper applies the methodology of ISO 10360-13 for the characterization and comparison of three RGB-D cameras produced by Intel® RealSense™ (D415, D455, L515) in the close range (100–1500 mm). ISO 10360-13 procedures, which focus on metrological performances, are integrated with additional tests to evaluate systematic errors (acquisition of flat objects, 3D reconstruction of objects). The present paper proposes an off-the-shelf comparison which considers the performance of the sensors throughout their acquisition volume. Results have exposed the strengths and weaknesses of each device. The D415 device showed better reconstruction quality on tests strictly related to the short range. The L515 device performed better on systematic depth errors; finally, the D455 device achieved better results on tests related to the standard.

Comparative study on 3D optical sensors for short range applications

The increasing availability of commercial 3D optical sensors drastically benefits the application fields including the mechatronics where providing affordable sensing means for perception and control is vital. Yet, to our knowledge, there is no thorough comparable study to the state-of-the-art 3D optical sensors, making it difficult for users to select for their specific applications. This paper evaluates the performance of each sensor for short range applications (i.e., ≤ 1 m). Specifically, we present our findings on the measurement accuracy of each sensor under “ideal” situations, compare the influence of various lighting conditions, object surface properties (e.g., transparency, shininess, contrast), and object locations. In addition, we developed software APIs and user instructions that are available for the community to easily use each of the evaluated commercially available 3D optical sensor.

Registration strategy of point clouds based on region-specific projections and virtual structures for robot-based inspection systems

Current requirements resulting from high quality standards, individualized mass production, and cost pressure call for new concepts in manufacturing metrology. Robot-based optical inspection systems have shown high potential to cope with these challenges. Since the kinematics of robots usually exhibit significant inaccuracies, an alternative approach becomes necessary to register multiple views of a sensor. With the advancement of optical 3D sensors in recent years, registration methods based on measured data (e.g. point clouds with image data) have shown tremendous progress and potential. Therefore, we present a new registration strategy which uses region-specific projections (image data) and virtual structures (point data) to support the alignment process of multiple point clouds. The fine registration is done in a pairwise manner and by means of an Iterative Closest Point (ICP) algorithm. We first introduce a procedure to obtain a set of virtual structures. Subsequently, influencing factors, such as the parametrization of the structures, noise, detection methods for the projected shapes, and the underlying surface, are examined. Based on these results, we identify virtual structures and employ them to evaluate the registration strategy in a real-world use case in the context of geometric quality assurance in the automotive industry. Acquired sphere-to-sphere distances are compared to reference measurements obtained from a coordinate measuring machine (CMM). The virtual structure comprising three point-based “columns” with an increasing point spacing in a triangular arrangement showed the best overall performance. The mean error (34 μm for one registration; 70 μm and 54 μm for two registrations respectively) in comparison to the robot kinematics was reduced by a factor of about 14 to 23 for the different reference distances. We demonstrated that our registration strategy enables a highly accurate and robust alignment of point clouds, which benefits manufacturing metrology for the inspection of sheet metal parts.

Inverse design and demonstration of high-performance wide-angle diffractive optical elements.

Diffractive optical elements are ultra-thin optical components required for constructing very compact optical 3D sensors. However, the required wide-angle diffractive 2D fan-out gratings have been elusive due to design challenges. Here, we introduce a new strategy for optimizing such high-performance and wide-angle diffractive optical elements, offering unprecedented control over the power distribution among the desired diffraction orders with only low requirements with respect to computational power. The microstructure surfaces were designed by an iterative gradient optimization procedure based on an adjoint-state method, capable to account for application-dependent target functions while ensuring compatibility with existing fabrication processes. The results of the experimental characterization confirm the simulated tailored power distributions and optical efficiencies of the fabricated elements.

3D Optical Machine Vision Sensors With Intelligent Data Management for Robotic Swarm Navigation Improvement

The optimized communication within robotic swarm, or group (RG) in a tightly obstacled ambient is crucial point to optimize group navigation for efficient sector trespass and monitoring. In the present work the main set of problems for multi-objective optimization in a non-stationary environment is described. It is presented the algorithm of data transfer from 3D optical sensor, based on the principle of dynamic triangulation. It uses the distributed scalable big data storage and artificial intelligence in automated 3D metrology. Two different simulations in order to optimize the fused data base for better path planning aiming the improvement of electric wheeled mobile robots group navigation in unknown cluttered terrain is presented. The optical laser scanning sensor combined with Intelligent Data Management permits more efficient dead-reckoning of the RG.

High-Speed Optical 3D Measurement Sensor for Industrial Application

3D optical sensors are becoming more and more popular in vision guidance of industrial robots and other industrial automation applications. Although research on high-speed 3D shape measurement (or imaging) has experienced tremendous growth over the past decades, simultaneously achieving high-speed and high-accuracy performance remains a major challenge in industrial practice. This paper presents a new FPGA architecture for the PMP algorithm and designs an high-speed embedded binocular structured light 3D measurement sensor. The sensor mainly contains a DLP projector, two cameras and a Xilinx Zynq-7000 UltraScale which make real-time computation possible. Experiments showed the time consuming of processing a set of 9 ×2 images with a resolution of 2048 ×1536 using the sensor proposed in this paper is 31.47 ms that has a better performance than 255ms on GPU. To the best of our knowledge, this is the first high-speed FPGA-based embedded binocular structured light 3D measurement sensor. This proposed 3D sensor can obtain ultra high quality 3D information in real time, it can be widely applied to robot random pin-picking and other industrial applications.

Scene Acquisition with Multiple 2D and 3D Optical Sensors: A PSO-Based Visibility Optimization.

Designing an acquisition system for 2D or 3D information, based on the integration of data provided by different sensors is a task that requires a labor-intensive initial design phase. Indeed, the definition of the architecture of such acquisition systems needs to start from the identification of the position and orientation of the sensors observing the scene. Their placement is carefully studied to enhance the efficacy of the system. This often coincides with the need to maximize the surfaces observed by the sensors or some other metric. An automatic optimization procedure based on the Particle Swarm Optimization (PSO) algorithm, to seek the most convenient setting of multiple optical sensors observing a 3D scene, is proposed. The procedure has been developed to provide a fast and efficient tool for 2D and 3D data acquisition. Three different objective functions of general validity, to be used in future applications, are proposed and described in the text. Various filters are introduced to reduce computational times of the whole procedure. The method is capable of handling occlusions from undesired obstacle in the scene. Finally, the entire method is discussed with reference to 1) the development of a body scanner for the arm-wrist-hand district and 2) the acquisition of an internal environment as case studies.

Accuracy Analysis of Alignment Methods based on Reference Features for Robot-Based Optical Inspection Systems

In recent years, optical 3D sensors have reached a high level of accuracy suitable for many applications involved in the geometric quality assurance in modern production sites. In order to circumvent the tradeoff between the size of the field of view and the accuracy, a fusion of multiple point clouds is often performed by means of data-driven registration algorithms, such as the well-known ICP. These methods require a coarse alignment of point clouds, which also influences the accuracy and robustness of the actual 3D matching process. In the context of robot-based inspection systems, additional reference features are often applied. The references are well detectable and provide key points. This gives rise to the question of whether or not better initial alignments are obtainable from the measurement data, in contrast to the alignment obtained by the robot kinematic. Therefore, we investigated the accuracy of calculated transformations for translational and rotational modifications based on measured data. The results indicate that for mainly translational relative transformations high accuracies are obtainable. An improvement of the coarse alignment for subsequent fine registration processes promises a contribution towards having more accurate and robust alignments of point clouds, and therefore benefits geometric quality assurance applications in manufacturing industries.

White-light interferometer without mechanical scanning

We introduce an optical 3D sensor that can measure the shape of objects with rough surface. The proposed sensor is based on white-light interferometry but it does not require a mechanical movement of the measured object relative to the measuring device. The output of a fiber optical interferometer is used as the light source for the measuring interferometer. The built-in fiber stretcher changes the optical path difference and replaces the mechanical movement. A focus tunable lens is a part of the imaging system. This lens secures that the measured part of the surface is imaged sharply onto the camera during the measurement process.

Assessment of Iterative Closest Point Registration Accuracy for Different Phantom Surfaces Captured by an Optical 3D Sensor in Radiotherapy.

An optical 3D sensor provides an additional tool for verification of correct patient settlement on a Tomotherapy treatment machine. The patient's position in the actual treatment is compared with the intended position defined in treatment planning. A commercially available optical 3D sensor measures parts of the body surface and estimates the deviation from the desired position without markers. The registration precision of the in-built algorithm and of selected ICP (iterative closest point) algorithms is investigated on surface data of specially designed phantoms captured by the optical 3D sensor for predefined shifts of the treatment table. A rigid body transform is compared with the actual displacement to check registration reliability for predefined limits. The curvature type of investigated phantom bodies has a strong influence on registration result which is more critical for surfaces of low curvature. We investigated the registration accuracy of the optical 3D sensor for the chosen phantoms and compared the results with selected unconstrained ICP algorithms. Safe registration within the clinical limits is only possible for uniquely shaped surface regions, but error metrics based on surface normals improve translational registration. Large registration errors clearly hint at setup deviations, whereas small values do not guarantee correct positioning.

A robotic surgery system (da Vinci) with image guided function--system architecture and cholecystectomy application.

We have developed a data fusion system for the robotics surgery system "da Vinci". The data fusion system is composed of an optical 3D location sensor and a digital video processing system. The 3D location sensor is attached to the da Vinci's laparoscope and measures its location and direction. The digital video processing system captures the laparoscope's view and superimposes 3D patient's organ models onto the captured view in real-time. We applied the system to "da Vinci" and examined this system during a cholecystectomy. In this experiment, the surgeon was able to observe the inner conditions of the organs with a stereo view.

Automated and Cycle Time Optimized Path Planning for Robot-Based Inspection Systems

Robot-based inspection systems, consisting of a standard industrial robot and an optical 3D sensor, increasingly gain importance within production in order to quantify the quality of products. These systems show advantages in terms of costs, flexibility and in-line capability. Based on the inspection plan of a product, the robot path for the inspection system is currently planned manually which is a very time consuming process. Consequently, an automated path planning algorithm generating a time optimized and collision-free path would improve the flexibility of robot-based inspection systems. The presented approach shows an automated and cycle time optimized path planning algorithm for robot-based inspection systems. This is realized by the probabilistic roadmap method applied on all measurement poses in combination with the A* search algorithm for the determination of weighted paths. Finally, the optimization of the path is reduced to a traveling salesman problem which is solved by the Christofides heuristic.

X-ray and optical stereo-based 3D sensor fusion system for image-guided neurosurgery.

In neurosurgery, an image-guided operation is performed to confirm that the surgical instruments reach the exact lesion position. Among the multiple imaging modalities, an X-ray fluoroscope mounted on C- or O-arm is widely used for monitoring the position of surgical instruments and the target position of the patient. However, frequently used fluoroscopy can result in relatively high radiation doses, particularly for complex interventional procedures. The proposed system can reduce radiation exposure and provide the accurate three-dimensional (3D) position information of surgical instruments and the target position. X-ray and optical stereo vision systems have been proposed for the C- or O-arm. Two subsystems have same optical axis and are calibrated simultaneously. This provides easy augmentation of the camera image and the X-ray image. Further, the 3D measurement of both systems can be defined in a common coordinate space. The proposed dual stereoscopic imaging system is designed and implemented for mounting on an O-arm. The calibration error of the 3D coordinates of the optical stereo and X-ray stereo is within 0.1 mm in terms of the mean and the standard deviation. Further, image augmentation with the camera image and the X-ray image using an artificial skull phantom is achieved. As the developed dual stereoscopic imaging system provides 3D coordinates of the point of interest in both optical images and fluoroscopic images, it can be used by surgeons to confirm the position of surgical instruments in a 3D space with minimum radiation exposure and to verify whether the instruments reach the surgical target observed in fluoroscopic images.