Point Cloud Image Research Articles

Transferability of ALS-based forest attribute models when predicting with drone-based image point cloud data

Field measurement of sample plots is a major cost in forest remote sensing. This is also relevant in drone-based forest inventories where the target area is rather small compared to the area used in other remote sensing techniques. Implementation of forest inventories by remote sensing could be streamlined by using models fitted elsewhere in a similar type of forest. The main objective of this study was to investigate the accuracy of forest attribute predictions from drone-based image point clouds (DIPC) without locally fitted models. Instead, the models were fitted in 22 inventory areas across Finland using airborne laser scanning (ALS) data. These models were applied to predict dominant height and stem volume for a separate test area located in eastern Finland. In the test area, the predictors were computed from DIPC data for 15 m × 15 m sub-plots that were finally aggregated to full 30 m × 30 m plots. All dominant height models performed well with the test data: the relative root mean square error (RMSE) varied between 3 and 5% and the relative mean difference (MD) values ranged between 0 and 3%. In contrast, the stem volume models fitted in northern Finland performed poorly with the test data. These models produced RMSE values between 40 and 65%, whereas models fitted in other parts of the country produced RMSE values between 20 and 30%. Similarly, MD values associated with the stem volume models fitted in northern Finland ranged between 24 and 51%, whereas MD values associated with models fitted elsewhere in Finland ranged between 3 and 17%. Regional variations in forest structure are the main reason why models fitted in northern Finland did not perform as well as in the test area.

A Novel Method for Density Analysis of Repaired Point Cloud with Holes Based on Image Data

Repairing point cloud holes has become an important problem in the research of 3D laser point cloud data, which ensures the integrity and improves the precision of point cloud data. However, for the point cloud data with non-characteristic holes, the boundary data of point cloud holes cannot be used for repairing. Therefore, this paper introduces photogrammetry technology and analyzes the density of the image point cloud data with the highest precision. The 3D laser point cloud data are first formed into hole data with sharp features. The image data are calculated into six density image point cloud data. Next, the barycenterization Bursa model is used to fine-register the two types of data and to delete the overlapping regions. Then, the cross-section is used to evaluate the precision of the combined point cloud data to get the optimal density. A three-dimensional model is constructed for this data and the original point cloud data, respectively and the surface area method and the deviation method are used to compare them. The experimental results show that the ratio of the areas is less than 0.5%, and the maximum standard deviation is 0.0036 m and the minimum is 0.0015 m.

3D point cloud generation reconstruction from single image based on image retrieval

Most of the 3D point cloud generation methods from a single image are only applicable to a single target. However, the real image often contains multiple targets, which is difficult to generate 3D point clouds from the network directly. To address this problem, this work proposes an end-to-end point cloud generation network that integrates three parts: image cropping, image retrieval and point cloud reconstruction. The addition of the image cropping module allows network to deal with multi-target situations that appear in image. The image retrieval module can retrieve similar images in the ShapeNet dataset, and input their corresponding ground truth point cloud and target images to the reconstruction network to obtain the three-dimensional reconstruction model. Experiments on a single image containing single and multiple targets show that our method achieves point cloud reconstruction of real image and obtains a more accurate three-dimensional model.

SaccadeFork: A lightweight multi-sensor fusion-based target detector

Commercialization of self-driving applications requires precision and reliability of the perception system due to the highly dynamic and complex road environment. Early perception systems either rely on the camera or on LiDAR for moving obstacle detection. With the development of vehicular sensors and deep learning technologies, the multi-view and sensor fusion based convolutional neural network (CNN) model for detection tasks has become a popular research area. In this paper, we present a novel multi-sensor fusion-based CNN model–SaccadeFork–that integrates the image and upsampled LiDAR point clouds as the input. SaccadeFork includes two modules: (1) a lightweight backbone that consists of hourglass convolution feature extraction module and a parallel dilation convolution module for adaptation of the system to different target sizes; (2) an anchor-based detection head. The model also considers deployment of resource-limited edge devices in the vehicle. Two refinement strategies, i.e., Mixup and Swish activation function are also adopted to improve the model. Comparison with a series of latest models on public dataset of KITTI shows that SaccadeFork can achieve the optimal detection accuracy on vehicles and pedestrians under different scenarios. The final model is also deployed and tested on a local dataset collected based on edge devices and low-cost sensor solutions, and the results show that the model can achieve real-time efficiency and high detection accuracy.

Establishment and Application of Hospital BIM Based on Point Cloud and Image Fusion for Infectious Disease (Novel Corona Virus Pneumonia)

Since neither a single high-precision 3D coordinate laser point cloud nor a oblique image of high-precision spectral information can make the computer understand the data as human, in order to accurately and completely understand the object in the real world, in this paper, the fusion of the point cloud and the oblique image is studied for the understanding and application of the real object buildings by establishing a BIM model. Firstly, point cloud data and image data are acquired by different data acquisition methods. Laser point cloud and oblique image are fused by installation parameters so that geometric elements are consistent and benchmark is unified. Processing level is not limited to element level only, so as to ensure the integrity of information, and the fused mutual information remains consistent in scale, texture and direction. Secondly, under the condition of space-time synchronization, the relationship between point cloud and image in geometric space is established, and the alternating "energy transfer" is used for shape prediction and visual filling by matching under texture mapping constraints to enhance the semantic information of the 3D scene. Finally, a 3D building information model is established. Using the experimental results to track the whole life cycle (design, construction, operation) in real time saves the cost of building rework, reduces the building cycle, and improves the building accuracy. Especially based on BIM during Novel Coronavirus Pneumonia: analyze hospital space resource management and elastic function; construct medical unit model of infectious disease hospital; simulate and optimize medical process in infectious disease hospital; rapidly construct prefabricated infectious disease emergency hospital; simulate hospital infection path and dynamics; the auxiliary design and construction of infectious disease treatment in existing hospitals; The monitoring analysis and automatic control of hospital infection safety operation.

Achieve accurate recognition of 3D point cloud images by studying the scattering characteristics of typical targets

Three-dimensional imaging lidar has been widely used in various fields due to its high measurement accuracy, strong directionality, and three-dimensional visualization. At present, the three-dimensional point cloud imaging method of lidar mainly depends on the distance information of the target, but the intensity information of the target can play an auxiliary role in the recognition of the target, and it has also received extensive attention. The combination of intensity information and distance information enables the system to make more accurate decisions and improve the redundancy of the system. When scanning non-cooperative targets, if there is only distance information, the target will not be accurately identified, which may cause errors in decision-making. This paper studies and analyzes the scattering characteristics of typical lidar targets. The intensity of the scattered echo signal of the target is related to the target material, target distance, gray value and detection angle, so the detector's receiving scattering rate is different. For the extraction of the intensity and the distance signal, we propose a double-scale intensity-weighted centroid algorithm to achieve it, which ensures the accuracy of the signal. To this end, we build a database of the scattering characteristics of related targets, combined with the original distance information to realize the accurate recognition of the target in the lidar 3D point cloud image.

3D terahertz incoherent point-cloud imaging for complex objects

Synthetic Aperture Radar (SAR) Imaging and Time of Flight (ToF) Imaging are two popular approaches for terahertz (THz) 3D imaging nowadays. Both systems above require broadband operations and coherent detection, which requires complicated and costly hardware. Generally, 3D imaging is difficult for those objects with large-curvature features in the case of both optical and THz systems. In this paper, we propose a simple but effective 3D imaging approach, which works in the point-cloud imaging mode. Our in-house THz lenses can focus the THz wave very well and it can gather reflected power within a large angle, thus making the decent 3D imaging feasible. By introducing the entropy concept for threshold optimization, decent figures can be obtained in case of poor raw data with a low signal-to-noise (SNR) ratio. In experiments, our system outperforms the commercial Terahertz Time-Domain Spectroscopy (THz-TDS) system. The proposed method could benefit industrial production, heritage studies, etc.

A hybrid feature-based patient-to-image registration method for robot-assisted long bone osteotomy.

The purpose of this study is to provide a simple, feasible and effective patient-to-image registration method for robot-assisted long bone osteotomy, which has rarely been systematically reported. The practical requirement is to meet the accuracy of 1mm or even higher without bone-implanted markers. A hybrid feature-based registration method termed CR-RAMSICP is proposed. Point-based coarse registration (CR) is accomplished relying on the optical retro-reflective markers attached to the tracked rigid body fixed out of the bone. In surface-based fine registration, an improved iterative closest point (ICP) algorithm based on the range-adaptive matching strategy (termed RAMSICP) is presented to cope with the robust precise matching between the asymmetric patient and image point clouds, which avoidsconverging to a local minimum. A series of registration experiments based on the isolated porcine iliums are carried out. The results illustrate that CR-RAMSICP not only significantly outperforms CR and CR-ICP in the accuracy and reproducibility, but also exhibits better robustness to the CR errors and less sensitiveness to the distribution and number of fiducial points located in the patient point cloud than CR-ICP. The proposed registration method CR-RAMSICP canstably satisfythedesiredregistration accuracy without the use of bone-implanted markers like fiducial screws. Besides, the RAMSICP algorithm used in fine registration is convenient for programming because any complex metrics or models are not involved.

A Field-Tested Harvesting Robot for Oyster Mushroom in Greenhouse

The fully autonomous harvesting of oyster mushrooms in the greenhouse requires the development of a reliable and robust harvesting robot. In this paper, we propose an oyster-mushroom-harvesting robot, which can realize harvesting operations in the entire greenhouse. The two crucial components of the harvesting robot are the perception module and the end-effector. Intel RealSense D435i is adopted to collect RGB images and point cloud images in real time; an improved SSD algorithm is proposed to detect mushrooms, and finally, the existing soft gripper is manipulated to grasp oyster mushrooms. Field experiments exhibit the feasibility and robustness of the proposed robot system, in which the success rate of the mushroom recognition success rate reaches 95%, the harvesting success rate reaches 86.8% (without considering mushroom damage), and the harvesting time for a single mushroom is 8.85 s.

The Use of UAV in Cadastral Mapping of the Czech Republic

The main challenge in the renewal and updating of the Cadastre of Real Estate of the Czech Republic is to achieve maximum efficiency but to retain the required accuracy of all points in the register. The paper discusses the possibility of using UAV photogrammetry and laser scanning for cadastral mapping in the Czech Republic. Point clouds from images and laser scans together with orthoimages were derived over twelve test areas. Control and check points were measured using geodetic methods (RTK-GNSS and total stations). The accuracy of the detailed survey based on UAV technologies was checked on hundreds of points, mainly building corners and fence foundations. The results show that the required accuracy of 0.14 m was achieved on more than 80% and 98% of points in the case of the image point clouds and orthoimages and the case of the LiDAR point cloud, respectively. Nevertheless, the methods lack completeness of the performed survey that must be supplied by geodetic measurements. The paper also provides a comparison of the costs connected to traditional and UAV-based cadastral mapping, and it addresses the necessary changes in the organisational and technological processes in order to utilise the UAV based technologies.

Point cloud target detection and tracking algorithm based on K-means and Kalman

This paper is mainly about point cloud image and data processing, virtualization and visualization. An improved K-means iterative clustering algorithm is proposed. In addition, cloth simulation filtering algorithm is used to extract road surface information. On this basis, bilateral filtering and statistic filtering are used to smooth image denoising. Kalman filter is used to track and predict point cloud targets. Using the variance of innovation sequence, the noise variance and observation noise variance can be estimated and corrected gradually in the process of system calculation. The error is relatively small, and the accuracy and reliability are improved. For disordered, occluded, sparsity and noisy, complex terrain image, how to improve the denoising effect, the accuracy and effectiveness of target extraction and tracking is the focus of this paper.

A Deep Learning Method for Target Pose Estimation of Planetary Surface

The key technical challenge of 6D target attitude estimation from RGB-D image is to make full use of 2d and 3D image point cloud data. Previous traditional target pose estimation methods either extract information from RGB images and depth, respectively, or use expensive post-processing steps that limit their performance in highly cluttered scenarios and real-time applications. Therefore, in this paper, a method is proposed to estimate the 6D attitude of a group of known objects from RGB-D images. The network used is a heterogeneous architecture, which processes two data from different dimensions separately, and uses a new dense fusion network to extract pixel-level dense feature embedding, and estimate the target position and attitude from it. In addition, an end-to-end iterative attitude optimization method is integrated to further improve attitude estimation, while real-time visualization of point clouds and rendering of bounding boxes are realized. Experimental results show that this method is effective on open source data sets.

Deep Learned Quantization-Based Codec for 3D Airborne LiDAR Point Cloud Images.

This paper introduces a novel deep learned quantization-based coding for 3D Airborne LiDAR (Light detection and ranging) point cloud (pcd) image (DLQCPCD). The raw pcd signals are sampled and transformed by applying the Nyquist signal sampling and Min-max signal transformation techniques, respectively for improving the efficiency of the training process. Then, the transformed signals are feed into the deep learned quantization module for compressing the data. To the best of our knowledge, this proposed DLQCPCD is the first deep learning-based model for 3D airborne LiDAR pcd compression. The functions of Mean Squared Error and Stochastic Gradient Descent optimization function enhance the quality of the decompressed image by 67.01 percent on average, compared to other functions. The model’s efficiency has been validated with established well-known compression techniques such as the 7-Zip, WinRAR, and tensor tucker decomposition algorithm on the three inconsistent airborne datasets. The experimental results show that the proposed model compresses every pcd image into constant 16 Number of Neurons of data and decompresses the image with approximately 160 dB of PSNR value, 174.46 s execution time with 0.6 s execution speed per instruction, and proved that it outperforms the other existing algorithms regarding space and time.

Design of 2D LiDAR and camera fusion system improved by differential evolutionary PID with nonlinear tracking compensator

An improved 2D LiDAR and camera fusion system is proposed for the 3D reconstruction of unknown environments. It combines the advantages of dense 2D point cloud and rich color image, adopting a differential evolutionary nonlinear tracking PID to control the pitching motion of LiDAR and camera accurately. The quadratic polynomial transition function is used to optimize the pitching trajectory. The environment was scanned by the system and converted into a 3D colored point cloud by the data fusion algorithm. The experimental results show: the proposed PID control algorithm can accurately control the pitching motion with a small average error (0.0267°) and significantly reduce the point cloud inhomogeneity (0.00698); the processing time for converting each 2D point cloud into the 3D point cloud is about 0.6 ms; combined with the data fusion algorithm, the system can obtain the dense colored 3D point cloud; compared with binocular camera, depth camera and 3D LiDAR under the condition of strong light interference, the fusion system outperforms, with the reconstruction object errors of distance, length and width of 0.23%, 0.17% and 0.46% respectively. In conclusion, the system can obtain homogeneous, and dense colored 3D point cloud in real time while ensuring stable refresh frame rate.

An intelligent and automated 3D surface defect detection system for quantitative 3D estimation and feature classification of material surface defects

To evaluate defects on the surface of the materials at the 3D level accurately and quantitatively, a 3D surface defect detection system based on stereo vision is presented, which can extract the precise 3D defect features of the detected object. The proposed detection system consists of two image capture modules and a turntable to capture the complete 3D information and color texture information from the object surface. More precisely, each image capture module is a binocular stereo vision system containing two monochrome cameras, a color camera, and a speckle projector which is used to reconstruct the 3D point clouds of the object surface based on stereo digital image correlation (stereo-DIC). Furthermore, a point-image mapping relationship between the reconstructed 3D object points and the color images is established. Eventually, the 3D characteristic parameters of defects are calculated by the corresponding 3D point cloud of the defect area obtained by segmenting the defect area using the image segmentation and point cloud segmentation algorithms according to this point-image mapping relationship. A convolutional neural network named DenseNets is employed to identify defect types intelligently. A high-precision multi-camera calibration method based on close-range photogrammetry is applied to ensure system detection accuracy in the proposed system. The experimental results demonstrate that the system has higher accuracy and better performance in system calibration, 3D reconstruction, and defect feature calculation.

Color Interactive Contents System using Kinect Camera Calibration

Recently, media content that interacts in real time is increasing. In this paper, we introduce a real-time color extraction content system that utilizes the Kinect camera used in ‘COLOR’ media art. The Kinect camera used in the work detects and tracks the joints of the visitors that enter the exhibition space. Kinect detected data is mapped to color calibration in a Unity environment to generate a point cloud video. Get the pixel color of the spine shoulder joint coordinates of the visitor in the point cloud image. The color data is output on the screen in the form of color one, and passes through along with the spectators. Color circle decreases as the distance between the visitors and Kinect increases and the distance increases. When visitors come in and the color circles overlap, color of the mixed part will have an intermediate value between the two color circles. This work shows the form of a person's social movement through the colors that each person has and the mixture of the colors. The technology used in this work differs from other media arts in that it extracted the calibrated image colors separately and advanced the interactive media arts. We will improve the accuracy of the point cloud that corrects the color image and the depth image, and improve the color extraction accuracy of the visitors.

On-line detection device for high temperature forgings based on laser triangulation

This paper takes the online detection equipment of high temperature forgings in the forging environment as the research goal, and uses laser triangulation to perform 3D point cloud imaging of high temperature forgings, and then conducts online detection of the defects, temperature, position, size and other parameters of the forgings. Real-time information feedback of measurement results, and timely adjustment of forging equipment and improvement of production process routes based on the feedback results, thereby effectively improving forging accuracy and forging efficiency, and reducing the work intensity of workers.

Manipulator system designs for drawing random objects through point cloud posture estimation

This paper designed a 7-DOF redundant robot manipulator that can flexibly and efficiently pick-up random objects. The developed 7-DOF machine with an additional redundancy achieved great progress in terms of flexibility and efficiency in the operational space. A robot operating system (ROS) was used to configure the manipulator system’s software modules, supporting convenient system interface, appropriate movement control policy, and powerful hardware device management for better regulation of the manipulator’s motions. A 3D type Point Cloud Library (PCL) was utilized to perform a novel point cloud image pre-processing method that did not only reduce the point cloud number but also maintained the original quality. The results of the experiment showed that the estimation speed in object detection and recognition procedure improved significantly. The redundant robot manipulator architecture with the two-stage search algorithm was able to find the optimal null space. Suitable parameters in D-H transformation of forward kinematics were selected to efficiently control and position the manipulator in the right posture. Meanwhile, the reverse kinematics estimated all angles of the joints through the known manipulator position, orientation, and redundancy. Finally, motion panning implementation of manipulator rapidly and successfully reached the random object position and automatically drew it up to approximate the desired target.

Sonar image reconstruction method based on adaptive compensation of very shallow water acoustic channel features

Aiming at the problem of eliminating the influence of channel interference on the detection results in the process of using sonar to detect very shallow water targets, a sonar image reconstruction method based on adaptive compensation of very shallow water acoustic channel features is proposed. Standard acoustic source and remote server are used to complete the adaptive calibration process of channel characteristic parameters before detection. The calibrated channel characteristic parameters are used in the subsequent beamforming algorithm to compensate the channel interference. The time-of-flight parameters calculated by the compensation algorithm are used in the subsequent point cloud generation and image reconstruction. Simulation and field test show that the proposed compensation method can improve sonar target detection ability by 25% in very shallow water and complex acoustic channel environment, and the quality of sonar image reconstruction is significantly improved.

Workspace analysis and kinematics simulation of 6-DOF upper limb exoskeleton robot

In this paper, the outer non-humanoid robot 6-DOF upper limb bone as the research object, the establishment of a forward kinematics model of the robot be modeled, using the DH parameter method in Creo software to get the robot based on Monte Carlo Method Point cloud image of workspace. Based on ADAMS, the dynamic of the joint driving torque is solved, and the torque change curve of each joint during the vertical lifting of the arm is obtained. The simulation results show that the working space at the end of the six-degree-of-freedom upper-limb exoskeleton robot model can meet the working requirements, and the joint torque is relatively small, which lays the foundation for the subsequent trajectory planning and drive device design of the upper-limb exoskeleton robot.