3D Vision Sensor Research Articles

Predictive Maintenance in Bridge Infrastructure

Predictive maintenance, driven by artificial intelligence and data analytics, represents a transformative approach to infrastructure management. By continuously monitoring asset conditions through sensors (Vibration, Temperature, crack, moisture and corrosion, 3D vision sensor and Ultrasonic & sonic) and advanced algorithms, predictive maintenance can forecast maintenance needs before infrastructure failures occur, leading to efficient scheduling, reduced costs, minimized downtime, and improved safety. Beyond cost savings, this approach enhances sustainability by optimizing resource usage, extending asset service life, and reducing environmental impact. Real-world applications in road infrastructure and the power sector demonstrate its potential to ensure reliability, prevent accidents, and meet the growing demands placed on critical infrastructure. Embracing predictive maintenance technologies is essential for addressing 21st-century infrastructure challenges effectively and safeguarding the well-being of communities

Research on Accuracy Enhancement Scheme for Home 3D Printers Based on Vision Sensors

Nowadays, with the rapid development of technology, more and more new technologies are being developed, but how to make these technologies accessible to factories at an affordable price and at a cost that is acceptable to all parties is a huge and complex comprehensive research. This paper focuses on two new technologies. They are 3D printers and vision sensors. After discussing how they work and the advantages and disadvantages about them. This paper focuses on how these two new technologies can be combined to improve the quality of 3D printing, increase the compatibility of building complex geometries, and reduce the reliance on manual calibration. Finally, a model for combining these two novel technologies is presented along with its operational logic. However, the challenges of cost, sensor integration and algorithm optimization will be addressed in further development and finally a partial analysis of how it can be further developed in the future.

Real-time pose correction and registration of complex workpiece in robotic machining considering normal consistency constrains

Efficient and precision model registration of complex workpiece with arbitrary pose is a challenging task that determines the effects of intelligent robotic machining. Aiming at the problems of limited field of view of 3D vision sensors such as surface structured light and the complexity of scanning-splicing-registration algorithm, in this paper a real-time pose correction and registration algorithm of complex workpiece in robotic machining considering normal consistency constrains is proposed. Specifically, the operations on ellipse feature detection and center point extraction are performed based on the arc-support line segment, and then the actual three-dimensional coordinate of the ellipse is obtained by the camera conversion matrix. Furthermore, considering the constrained normal consistency of the center points set of ellipses on the model surface, the pose of the target workpiece is corrected in real-time by the improved ICP registration algorithm. Taking the automotive flywheel shell as a typical application object, the experimental results demonstrate that the registration correction time of the proposed algorithm is 0.5 s, and the registration accuracy can reach 0.35 mm. The repetitive correction time of the algorithm after arbitrarily changing the workpiece pose is less than 1 s, and the positioning accuracy of the guiding robot is less than 1.5 mm. On the premise of satisfying the robotic machining quality requirements, the pose of the complex workpiece model can be corrected in real-time for subsequent robotic machining.

Measuring the interior of in-use sewage pipes using 3D vision

Sewage pipes may be renovated using tailored linings. However, the interior diameter of the pipes must be measured prior to renovation. This paper investigates the use of 3D vision sensors for measuring the interior diameter of sewage pipes, removing the need for human entry in the pipes. The 3D sensors are residing in a waterproof box that is lowered into the well. A RANSAC-based method is used for cylinder estimation from the acquired point clouds of the pipe and the diameter of these cylinders is used as a measure of the interior pipe diameter. The method is tested in 74 real-world sewage pipes with diameters between 150- and 1100 mm. The diameter of 68 pipes is measured within a tolerance of ±20mm whereas 8 pipes are above. It was found that the faulty estimates can be detected in the field using a combination of human-in-the-loop qualitative and quantitative data-driven measures.

Vision-Based Impedance Control of an Aerial Manipulator Using a Nonlinear Observer

Contact-based aerial interaction control in unknown environments is a challenging issue. To solve this problem, this paper presents an image-based impedance control scheme for an aerial manipulator. Unlike points and image moments, the line features considered in this paper cannot uniquely determine the pose of the onboard camera. Therefore, a nonlinear observer is proposed to online provide supplementary 3D information of the system. After that, a hierarchical tracking controller, which is equipped with an image-space impedance filter, is designed to implement a trajectory tracking task with tunable compliance, while at the same time achieving a subtask that aims to determine the final pose of the aerial manipulator. By planning task-space trajectories, desired interaction behavior can be specified without relying on the global position of the system. Furthermore, the effectiveness criterion of the observer is derived, and the stability of the system is proven. Several experiments, including a push-and-slide task, robustness analysis, and a comparison with the state of the art have been conducted to demonstrate the feasibility of the method. Note to Practitioners—This article presents a method for controlling an aerial manipulator to interact with unknown environments without measuring the global position of both the environment and the aerial manipulator. Furthermore, unlike the existing methods, the proposed visual servoing method is based on line features, which are generally more robust than point features. Moreover, it would be intuitive for users to specify the approaching path and the interaction behavior of the aerial manipulator by just planning the image feature trajectories and the desired distance between the aerial manipulator and the environment. This distance is online estimated using the visual feedback of a monocular camera, instead of relying on any costly 3D-vision sensors or lidars, making it an economical solution for aerial manipulation in field environments.

Digital model reconstruction through 3D Stereo Depth camera: a faster method exploiting robot poses

In industrial robotic contact-based operations it is necessary to have a detailed geometrical knowledge of the work-piece to automate the generation of the working trajectory. In many cases, the digital model is not available or it differs from its as built actual conditions. Vision sensors, especially 3D vision sensors allows to scan the work-piece and reconstruct its digital copy that is used for the generation of the robotic working trajectory. In this paper we compare two algorithms for the generation of the 3D model of an unknown work-piece based on the use of the RGB-D images taken from different perspectives. The first technique, based on standard image reconstruction methods commonly used for reconstruction of indoor scenes where error in the order of few centimeters is negligible, is exploited in this work in order to be exploited in contact-based robotic operations where the geometrical error has to be limited to few millimeters. It is based on the analysis of the images to estimate the camera pose taking each image. Based on the pose, the algorithm integrates then the information contained in the images in a unique volume representing the scene. The second algorithm directly uses the known poses of the robot, stored while capturing each image, to integrate the information to create the model. The two algorithms are compared to evaluate the accuracy, acquisition time and elaboration time. The analysis is done considering two kinds of objects, the first one with regular shape, the second one corresponding to an actual industrial object.

Dense Scene Flow Estimation with the GVF Snake Model for Obstacle Detection Using a 3D Sensor in the Path-Planning Module

Abstract The novel framework for estimating dense scene flow using depth camera data is demonstrated in the article. Using these estimated flow vectors to identify obstacles improves the path planning module of the autonomous vehicle's (AV) intelligence. The primary difficulty in the development of AVs has been thought to be path planning in cluttered environments. These vehicles must possess the intelligence to recognize their surroundings and successfully navigate around obstacles. The AV needs a thorough understanding of the surroundings to detect and avoid obstacles in a cluttered environment. Therefore, when determining the course, it is preferable to be aware of the kinematic behavior (position and the direction) of the obstacles. As a result, by comparing the depth images between different time frames, the position and direction of the obstacles are calculated using a 3D vision sensor. The current study focuses on the extraction of the flow vectors in 3D coordinates from the differential scene flow method. Generally, the evaluation of scene flow algorithms is crucial in determining their accuracy and effectiveness in different applications. The gradient of the vector field snake model, which extracts changes in pixel values in three directions, is combined with the scene flow technique to identify both static and dynamic obstacles. Our goal is to create a single-vision sensor-based real-time obstacle avoidance method based on scene flow estimation. In addition, the common evaluation metrics such as endpoint error (EPE), average angular error (AAE), and standard deviation angular error (STDAE) are used to measure the accuracy of different algorithms in terms of computational errors with the benchmark Middlebury datasets. The proposed technique is validated with different experiments using a Pixel-Mixed-Device (PMD) camera and a Kinect sensor as 3D sensors. Finally, the numerical and experimental results are displayed and reported.

Automated 3D burr detection in cast manufacturing using sparse convolutional neural networks

For automating deburring of cast parts, this paper proposes a general method for estimating burr height using 3D vision sensor that is robust to missing data in the scans and sensor noise. Specifically, we present a novel data-driven method that learns features that can be used to align clean CAD models from a workpiece database to the noisy and incomplete geometry of a RGBD scan. Using the learned features with Random sample consensus (RANSAC) for CAD to scan registration, learned features improve registration result as compared to traditional approaches by (translation error (Delta 18.47 mm) and rotation error(Delta 43 ^circ )) and accuracy(35%) respectively. Furthermore, a 3D-vision based automatic burr detection and height estimation technique is presented. The estimated burr heights were verified and compared with measurements from a high resolution industrial CT scanning machine. Together with registration, our burr height estimation approach is able to estimate burr height similar to high resolution CT scans with Z-statistic value (z=0.279).

Model-Free Adaptive Impedance Control for Autonomous Robotic Sanding

Sanding is a common yet important task in the manufacturing of many wooden objects (e.g. furniture, decoration box), where the coated layer attached on objects is removed after the interaction with sanding belts. Existing sanding operation is heavily dependent on manual works, which is highly labor-intensive and with the low consistency of quality, and the issues of safety and health also arise after continuous working in the noisy and dusty environment. To deal with the aforementioned, this paper presents the development of a new autonomous sanding robot. The autonomous capability of the developed robot is reflected in the whole procedure of sanding. In particular, the CAD model of the target object is automatically constructed with the structured-light technology, and the sanding behavior on the target surface is self-regulated under the desired impedance model. Such feature makes the robot capable of working towards uncertain objects with minimum human involvement. The proposed impedance controller has the model-free advantage, by using the adaptive neural networks (NNs) to compensate the uncertain dynamics and the unknown disturbances online. The stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results on different objects are presented to validate the performance of the developed robot. The implementation of the developed robot can systematically address the problems associated with manual works. Note to Practitioners—The current working environment of sanding is not healthy or safe to humans, due to the nature of noise, dust, high-speed sanding belt. By controlling the robot to autonomously perform sanding tasks can keep humans away from such environment and hence systematically address the issues of health and safety. This paper presents a new impedance control method for sanding robot. In impedance control, the control goal is specified as a dynamic relationship between the contact position and the interaction force. In this paper, the contact position is determined by using the 3D vision sensor, while the interaction force is regulated by referring to the human experience. In addition, the influence caused by unmodeled factors (e.g. unknown dynamics of the sanded object) is also dealt with by using the techniques of NNs. Such setting can effectively guarantee the sanding quality and also avoid the physical damage to the sanded object. Therefore, it lays the foundation for the autonomous robotic sanding.

Extraction of key regions of beef cattle based on bidirectional tomographic slice features from point cloud data

As the world population grows and the increase in food consumption creates unprecedented pressure on beef producers, innovative technologies for sustainable and efficient beef production is quintessential. Therefore, a significant effort is ongoing on developing digital precision technologies for continuous beef cattle monitoring and accurate cattle growth estimation. One such technology relies on 3D visual sensor technologies. 3D point cloud data acquired with 3D cameras can be processed to evaluate the livestock’s growth condition with minimal human intervention. Essentially, the automatic measurement of animal body dimensions highly depends on the location accuracy of the key points or regions in point clouds. However, the location of specific points and regions requires extra assistance and usually leads to semi-automatic and low-accuracy solutions at best. To address these issues, we demonstrated a novel strategy for partial segmentation of beef cattle point clouds, based on the Bidirectional Tomographic Slice Segmentation (BTSS) algorithm. The cattle point clouds can be successfully segmented into the head and neck, front trunk, middle trunk, back trunk, lower leg, and hip and tail regions with high accuracies of 89%, 91%, 94%, 95%, 92%, and 95%, respectively. Consequently, the segmentation completion rate is 96%, and the average segmentation accuracy reaches 92.8%. When compared to traditional approaches, our method efficiently extracted abundant body characteristics from the beef cattle point cloud. Moreover, the proposed algorithm exhibited generalization capabilities to segment the point clouds of other cloven-hoofed livestock species. The accurate localization of key regions enabled body dimensions measurements and non-contact weighing, which can provide a solid support for breeding needs such as health evaluation, production performance measurement, and genetic breeding evaluation.

Fusion of sparse LIDAR and uncertainty-based 3D vision for accurate depth estimation with Bayesian Kriging

Accurate 3D perception is vital for the localization of mobile robots and autonomous vehicles. Sparse 3D LIDAR scanners and 3D vision sensors (stereo and RGB-D cameras) are commonly used environment perception sensors. The former could provide accurate but sparse-range data, while the latter is dense but full of uncertainty. Since the acquired depth map by each sensor alone could not accurately perceive complex surrounding environment, an innovative probabilistic fusion model based on Bayesian Kriging (BK) is proposed for dense high-precision depth map estimation, to fully utilize the complementary characteristics of the two distinct sensors. Combing the geometric piece-wise planar hypotheses of the scene, the noisy disparity map from the 3D vision sensor is segmented into multiple small planes. In each segmented plane, the BK-based fusion estimator is founded to predict depth for each pixel with sparse scattered LIDAR points. In particular, we focus on analyzing the spatial variation prior and correlation between these two types of data, based on regionalized variable theory. Experiments on various scenes of KITTI stereo datasets are carried out to demonstrate the effectiveness of our algorithm, and validate the superiority compared with existing fusion methods.

AI-enhanced vision system for dispensing process monitoring and quality control in manufacturing of large parts

Assembly is one of the cornerstones of today’s manufacturing; thus, efficient automated assembly systems have become a necessity in the recent past. Challenges that have traditionally been present in such systems involve positioning accuracy, stable gripping, accommodation of several variants by the same resources and prevention of error aggregation. In robot-based applications the detection of parts and their components is a common task, but when it comes to large objects which cannot be portrayed in single camera frames, the complexity becomes significantly larger. This paper presents an AI based perception system which employs a 3D vision sensor to implement a smart dispensing application and perform online process quality control of large parts assembly. The aim of the system is to provide execution autonomy for a robotic manipulator that is responsible for glue dispensing on different types of aluminium profiles of varying cross-sections, length, and associated glue patterns. By using an eye-in-hand RGB-D camera and employing deep learning methods (Yolov3 algorithm), the framework recognizes the type of profile that is going to be processed, and depth information is used to extract the profile’s points (starting-ending points) on which glue must be dispensed. Point Cloud manipulation (DBSCAN clustering, plane segmentation) and 2D image-based processing were tested during the quality inspection, in order to validate the success and accuracy of the process (detection of glue discontinuities, calculation of the glue volume level). The system has been deployed on a case study involving the assembly of components on buses and the results indicate that a robust recognition and adhesive application can be obtained throughout each operation cycle.

Real‐Time Capture of Snowboarder’s Skiing Motion Using a 3D Vision Sensor

Due to the influence of environmental interference and too fast speed, there are some problems in ski motion capture, such as inaccurate motion capture, motion delay, and motion loss, resulting in the inconsistency between the actual motion of later athletes and the motion of virtual characters. To solve the above problems, a real‐time skiing motion capture method of snowboarders based on a 3D vision sensor is proposed. This method combines the Time of Fight (TOF) camera and high‐speed vision sensor to form a motion acquisition system. The collected motion images are fused to form a complete motion image, and the pose is solved. The pose data is bound with the constructed virtual character model to drive the virtual model to synchronously complete the snowboarding motion and realize the real‐time capture of skiing motion. The results show that the motion accuracy of the system is as high as 98.6%, which improves the capture effect, and the motion matching proportion is better and more practical. It is also excellent in the investigation of motion delay and motion loss.

Robust human position estimation in cooperative robotic cells

Robots are increasingly present in our lives, sharing the workspace and tasks with human co-workers. However, existing interfaces for human-robot interaction / cooperation (HRI/C) have limited levels of intuitiveness to use and safety is a major concern when humans and robots share the same workspace. Many times, this is due to the lack of a reliable estimation of the human pose in space which is the primary input to calculate the human-robot minimum distance (required for safety and collision avoidance) and HRI/C featuring machine learning algorithms classifying human behaviours / gestures. Each sensor type has its own characteristics resulting in problems such as occlusions (vision) and drift (inertial) when used in an isolated fashion. In this paper, it is proposed a combined system that merges the human tracking provided by a 3D vision sensor with the pose estimation provided by a set of inertial measurement units (IMUs) placed in human body limbs. The IMUs compensate the gaps in occluded areas to have tracking continuity. To mitigate the lingering effects of the IMU offset we propose a continuous online calculation of the offset value. Experimental tests were designed to simulate human motion in a human-robot collaborative environment where the robot moves away to avoid unexpected collisions with de human. Results indicate that our approach is able to capture the human’s position, for example the forearm, with a precision in the millimetre range and robustness to occlusions.

Grasping pose estimation for SCARA robot based on deep learning of point cloud

With the development of 3D measurement technology, 3D vision sensors and object pose estimation methods have been developed for robotic loading and unloading. In this work, an end-to-end deep learning method on point clouds, PointNetRGPE, is proposed to estimating the grasping pose of SCARA robot. In PointNetRGPE model, the point cloud and class number are fused into a point-class vector, and several PointNet-like networks are used to estimate the robot grasping pose, containing 3D translation and 1D rotation. Considering that rotational symmetry is very common in man-made and industrial environments, a novel architecture is introduced into PointNetRGPE to solve the pose estimation problem with rotational symmetry in the z-axis direction. Additionally, an experimental platform is built containing an industrial robot and a binocular stereo vision system, and a dataset with three subsets is set up. Finally, the PointNetRGPE is tested on the dataset, and the success rates of three subsets are 98.89%, 98.89%, and 94.44% respectively.

Intelligent 3D Perception System for Semantic Description and Dynamic Interaction

This work proposes a novel semantic perception system based on computer vision and machine learning techniques. The main goal is to identify objects in the environment and extract their characteristics, allowing a dynamic interaction with the environment. The system is composed of a GPU processing source and a 3D vision sensor that provides RGB image and PointCloud data. The perception system is structured in three steps: Lexical Analysis, Syntax Analysis and finally an Analysis of Anticipation. The Lexical Analysis detects the actual position of the objects (or tokens) in the environment, through the combination of RGB image and PointCloud, surveying their characteristics. All information extracted from the tokens will be used to retrieve relevant features such as object velocity, acceleration and direction during the Syntax Analysis step. The anticipation step predicts future behaviors for these dynamic objects, promoting an interaction with them in terms of collisions, pull, and push actions. As a result, the proposed perception source can assign relevant information to mobile robots, not only about distances as traditional sensors, but about other environment characteristics and object behaviors. This novel perception source introduces a new class of skills to mobile robots. Experimental results obtained with a real robot are presented, showing the proposed perception source efficacy and potential.

Expansion of person detection range using wide FoV 3D LiDAR and vision sensor

Expansion of person detection range using wide FoV 3D LiDAR and vision sensor

Depth Data Error Modeling of the ZED 3D Vision Sensor from Stereolabs

The ZED camera is binocular vision system that can be used to provide a 3D perception of the world. It can be applied in autonomous robot navigation, virtual reality, tracking, motion analysis and so on. This paper proposes a mathematical error model for depth data estimated by the ZED camera with its several resolutions of operation. For doing that, the ZED is attached to a Nvidia Jetson TK1 board providing an embedded system that is used for processing raw data acquired by ZED from a 3D checkerboard. Corners are extracted from the checkerboard using RGB data, and a 3D reconstruction is done for these points using disparity data calculated from the ZED camera, coming up with a partially ordered, and regularly distributed (in 3D space) point cloud of corners with given coordinates, which are computed by the device software. These corners also have their ideal world (3D) positions known with respect to the coordinate frame origin that is empirically set in the pattern. Both given (computed) coordinates from the camera’s data and known (ideal) coordinates of a corner can, thus, be compared for estimating the error between the given and ideal point locations of the detected corner cloud. Subsequently, using a curve fitting technique, we obtain the equations that model the RMS (Root Mean Square) error. This procedure is repeated for several resolutions of the ZED sensor, and at several distances. Results showed its best effectiveness with a maximum distance of approximately sixteen meters, in real time, which allows its use in robotic or other online applications.

Detection of morphology defects in pipeline based on 3D active stereo omnidirectional vision sensor

There are many kinds of defects in pipes, which are difficult to detect with a low degree of automation. In this work, a novel omnidirectional vision inspection system for detection of the morphology defects is presented. An active stereo omnidirectional vision sensor is designed to obtain the texture and depth information of the inner wall of the pipeline in real time. The camera motion is estimated and the space location information of the laser points are calculated accordingly. Then, the faster region proposal convolutional neural network (Faster R-CNN) is applied to train a detection network on their image database of pipe defects. Experimental results demonstrate that system can measure and reconstruct the 3D space of pipe with high quality and the retrained Faster R-CNN achieves fine detection results in terms of both speed and accuracy.

Data fusion for a vision-aided radiological detection system: Calibration algorithm performance

In order to improve the ability to detect, locate, track and identify nuclear/radiological threats, the University of Florida nuclear detection community has teamed up with the 3D vision community to collaborate on a low cost data fusion system. The key is to develop an algorithm to fuse the data from multiple radiological and 3D vision sensors as one system. The system under development at the University of Florida is being assessed with various types of radiological detectors and widely available visual sensors. A series of experiments were devised utilizing two EJ-309 liquid organic scintillation detectors (one primary and one secondary), a Microsoft Kinect for Windows v2 sensor and a Velodyne HDL-32E High Definition LiDAR Sensor which is a highly sensitive vision sensor primarily used to generate data for self-driving cars. Each experiment consisted of 27 static measurements of a source arranged in a cube with three different distances in each dimension. The source used was Cf-252. The calibration algorithm developed is utilized to calibrate the relative 3D-location of the two different types of sensors without need to measure it by hand; thus, preventing operator manipulation and human errors. The algorithm can also account for the facility dependent deviation from ideal data fusion correlation. Use of the vision sensor to determine the location of a sensor would also limit the possible locations and it does not allow for room dependence (facility dependent deviation) to generate a detector pseudo-location to be used for data analysis later. Using manually measured source location data, our algorithm-predicted the offset detector location within an average of 20 cm calibration-difference to its actual location. Calibration-difference is the Euclidean distance from the algorithm predicted detector location to the measured detector location. The Kinect vision sensor data produced an average calibration-difference of 35 cm and the HDL-32E produced an average calibration-difference of 22 cm. Using NaI and He-3 detectors in place of the EJ-309, the calibration-difference was 52 cm for NaI and 75 cm for He-3. The algorithm is not detector dependent; however, from these results it was determined that detector dependent adjustments are required.