Low-cost 3D Sensors Research Articles

REVOLUTIONIZING PHOTOGRAMMETRY: USING MIRRORS IN SINGLE-IMAGE PHOTOGRAMMETRY

Abstract. The present work introduces a new low-cost prototype® designed for photogrammetric surveys of small objects, using a single photographic shot. The prototype® utilizes the different perspectives of the object presented by a series of mirrors strategically placed in the camera sensor. The set of specular reflections projected on our camera sensor simulates the same effect as taking pictures from equivalent points of each mirror.The v1 prototype® was presented at the 7th International Workshop "Low-Cost 3D Sensors, Algorithms, Applications" in Würzburg on December 15–16, 2022. It consists of a set of eighteen common mirrors conveniently placed around the object to be digitized, making its reflections converge towards the camera lens. The v2 prototype® presented here is an evolution of the previous one, which replaces the common mirrors with precision ones that have a metallic reflective surface on the front. This modification eliminates the double image formed by the reflective layer and the protective glass.These prototypes® allow for faster surveys of small inert objects (such as small cultural heritage artefacts like sculptures, engravings, rings, coins, etc.) as well as photogrammetric surveys of small living beings, such as insects, which cannot remain still during traditional photography.

Real-time automated forest field inventory using a compact low-cost helmet-based laser scanning system

Forest field inventory plays a crucial role in forestry management and the estimation of carbon circular economy, as it provides information on forest parameters, assesses carbon storage, and identifies the impact factors of ecological change. Terrestrial Laser Scanning (TLS) and Mobile Mapping Systems (MMS) are commonly used for forest field inventory, but they are unable to verify the inventory results in real time. With real-time forest inventory, forestry workers can immediately verify if all trees have been correctly extracted without requiring expertise in point clouds. The advences of miniaturized, low-cost 3D sensors (such as solid-state laser scanners and Inertial Measurement Units [IMUs]) and edge computing units have made it possible to achieve real-time forest inventory using a compact, low-cost helmet. To this end, this paper presents a real-time automated forest field inventory method, which is validated on a compact, low-cost helmet-based laser scanning system. Firstly, a Fast Candidate Tree Detection (FCTD) approach is proposed to identify individual trees as candidates by utilizing a novel 2D corner detection technique based on point cloud projection, taking into account point density and geometry features. Secondly, a Spatiotemporal Consistency-based Tree Parameter Estimation (SCTPE) method is proposed to estimate the tree parameters by considering both the current submap and submaps that have been scanned in real-time. The proposed method was tested in three typical forest areas in Wuhan, China, where the main tree species present include Sapindaceae, dawn redwoods, and Platycladus. The results showed that the proposed method achieves high accuracy in tree detection (recall=0.97,precision=0.94,F=0.96). The average error and root-mean-square error (RMSE) of DBH are 0.033 m and 0.038 m, which outperforms the existing one-circle fitting model. The average error and RMSE of tree height are 0.231 m and 0.294 m. Overall, these results demonstrate the high potential of the helmet-based laser scanning system for real-time forest field inventory.

Evaluation of Geometric Data Registration of Small Objects from Non-Invasive Techniques: Applicability to the HBIM Field.

Reverse engineering and the creation of digital twins are advantageous for documenting, cataloging, and maintenance control tracking in the cultural heritage field. Digital copies of the objects into Building Information Models (BIM) add cultural interest to every artistic work. Low-cost 3D sensors, particularly structured-light scanners, have evolved towards multiple uses in the entertainment market but also as data acquisition and processing techniques for research purposes. Nowadays, with the development of structured-light data capture technologies, the geometry of objects can be recorded in high-resolution 3D datasets at a very low cost. On this basis, this research addresses a small artifact with geometric singularities that is representative of small museum objects. For this, the precision of two structured-light scanners is compared with that of the photogrammetric technique based on short-range image capture: a high-cost Artec Spider 3D scanner, and the low-cost Revopoint POP 3D scanner. Data capture accuracy is evaluated through a mathematical algorithm and point set segmentation to verify the spatial resolution. In addition, the precision of the 3D model is studied through a vector analysis in a BIM environment, an unprecedented analysis until now. The work evaluates the accuracy of the devices through algorithms and the study of point density at the submillimeter scale. Although the results of the 3D geometry may vary in a morphometric analysis depending on the device records, the results demonstrate similar accuracies in that submillimeter range. Photogrammetry achieved an accuracy of 0.70 mm versus the Artec Spider and 0.57 mm against the Revopoint POP 3D scanner.

Lowcost 3d—Sensors, Algorithms, Applications

Lowcost 3d—Sensors, Algorithms, Applications

The Potential of Low-Cost 3D Imaging Technologies for Forestry Applications: Setting a Research Agenda for Low-Cost Remote Sensing Inventory Tasks

Limitations with benchmark light detection and ranging (LiDAR) technologies in forestry have prompted the exploration of handheld or wearable low-cost 3D sensors (<2000 USD). These sensors are now being integrated into consumer devices, such as the Apple iPad Pro 2020. This study was aimed at determining future research recommendations to promote the adoption of terrestrial low-cost technologies within forest measurement tasks. We reviewed the current literature surrounding the application of low-cost 3D remote sensing (RS) technologies. We also surveyed forestry professionals to determine what inventory metrics were considered important and/or difficult to capture using conventional methods. The current research focus regarding inventory metrics captured by low-cost sensors aligns with the metrics identified as important by survey respondents. Based on the literature review and survey, a suite of research directions are proposed to democratise the access to and development of low-cost 3D for forestry: (1) the development of methods for integrating standalone colour and depth (RGB-D) sensors into handheld or wearable devices; (2) the development of a sensor-agnostic method for determining the optimal capture procedures with low-cost RS technologies in forestry settings; (3) the development of simultaneous localisation and mapping (SLAM) algorithms designed for forestry environments; and (4) the exploration of plot-scale forestry captures that utilise low-cost devices at both terrestrial and airborne scales.

Novel methods for noisy 3D point cloud based object recognition

3D point cloud based object recognition becomes increasingly important in the last few years, as the widely use of point cloud over the low-cost 3D sensors have developed rapidly. However, the obtained 3D point cloud is inevitably contaminated with noise due to physical and environmental factors, which has a negative impact on recognition task. To address this problem, a complete object recognition framework for 3D noisy point cloud is presented into which a pre-processing step of filtering is integrated for the first time. In the filtering phase, our two proposed approaches, named Guided 3D Point Cloud Filter (G3DF) and Iterative Guidance Normal Filter (IGNF), are taken into account to produce high-quality point cloud model. Then, on the basis of advantages of local-based and global-based descriptors, a new type of feature descriptor, called Local-to-Global Histogram (LGH), is proposed, which contains Local Viewpoint Feature Histogram (LVFH) and Local Ensemble of Shape Function (LESF). Experimental results show that the comprehensive classification performance yielded by using proposed filters and descriptors is competitive compared to other state-of-the-art combinations. In particularly, the composition of G3DF and LVFH is more suited for real-time applications.

LOW-COST 3D DEVICES AND LASER SCANNERS COMPARISON FOR THE APPLICATION IN ORTHOPEDIC CENTRES

Abstract. Low-cost 3D sensors are nowadays widely diffused and many different solutions are available on the market. Some of these devices were developed for entertaining purposes, but are used also for acquisition and processing of different 3D data with the aim of documentation, research and study. Given the fact that these sensors were not developed for this purpose, it is necessary to evaluate their use in the capturing process. This paper shows a preliminary research comparing the Kinect 1 and 2 by Microsoft, the Structure Sensor by Occipital and the O&P Scan by Rodin4D in a medical scenario (i.e. human body scans). In particular, these sensors were compared to Minolta Vivid 9i, chosen as reference because of its higher accuracy. Different test objects were analysed: a calibrated flat plane, for the evaluation of the systematic distance error for each device, and three different parts of a mannequin, used as samples of human body parts. The results showed that the use of a certified flat plane is a good starting point in characterizing the sensors, but a complete analysis with objects similar to the ones of the real context of application is required. For example, the Kinect 2 presented the best results among the low-cost sensors on the flat plane, while the Structure Sensor was more reliable on the mannequin parts.

Multi-scale CNNs for 3D model retrieval

Recent advancements in low-cost 3D sensors and mobile devices of virtual 3D models have additionally facilitated the accessibility of 3D data. 3D model retrieval is becoming an indispensable function for modern search engines. An effective retrieval model is at the core of computer vision. With the continuous improvement of 3D data, there are large number of methods to solve this problem. Existing works proposed numerous works to deal with feature extraction and object matching. Most of them are unable to fully exploit the information of 3D representations. To address this problem, we propose a novel multi-layer deep network in this paper. First, multiple rendered images are extracted from a 3D object, and combined into one representative view, which is the actual input of the network. Then, the novel multi-layer network structure is trained and tested on these representative views, generating the feature leaning model, which owns the local and global information of a 3D object. Finally, simple Euclidean metric is used to compute the similarity between two different 3D models to complete the retrieval problem. Extensive experiments and corresponding experimental results have demonstrated the superiority of our approach.

Iterative guidance normal filter for point cloud

3D point clouds have become increasingly popular in recent year due to the rapid development of low-cost 3D sensors. One of the most interesting challenges is to filter point cloud, which undoubtedly becomes a crucial part of the point cloud processing pipeline. Based on normal information, this paper proposes a simple but effective point cloud filter framework. In this framework, a kd-tree structure is constructed for representing point cloud to search neighborhood and estimate normal for each point at first. Then, iteratively performing the processing that a bilateral filter is applied to the normal field obtained from the previous iteration, using the same normal field as the guidance; afterward, adjusting point positions is performed depending on the filtered normals. Experimental results indicate the effectiveness of our algorithms.

Complete End-to-End Low Cost Solution to a 3D Scanning System with Integrated Turntable

3D reconstruction is a technique used in computer vision which has a wide range of applications in areas like object recognition, city modelling, virtual reality, physical simulations, video games and special effects. Previously, to perform a 3D reconstruction, specialized hardwares were required. Such systems were often very expensive and was only available for industrial or research purpose. With the rise of the availability of high-quality low cost 3D sensors, it is now possible to design inexpensive complete 3D scanning systems. The objective of this work was to design an acquisition and processing system that can perform 3D scanning and reconstruction of objects seamlessly. In addition, the goal of this work also included making the 3D scanning process fully automated by building and integrating a turntable alongside the software. This means the user can perform a full 3D scan only by a press of a few buttons from our dedicated graphical user interface. Three main steps were followed to go from acquisition of point clouds to the finished reconstructed 3D model. First, our system acquires point cloud data of a person/object using inexpensive camera sensor. Second, align and convert the acquired point cloud data into a watertight mesh of good quality. Third, export the reconstructed model to a 3D printer to obtain a proper 3D print of the model.

Complete End-to-End Low Cost Solution to a 3D Scanning System with Integrated Turntable

3D reconstruction is a technique used in computer vision which has a wide range of applications in areas like object recognition, city modelling, virtual reality, physical simulations, video games and special effects. Previously, to perform a 3D reconstruction, specialized hardwares were required. Such systems were often very expensive and was only available for industrial or research purpose. With the rise of the availability of high-quality low cost 3D sensors, it is now possible to design inexpensive complete 3D scanning systems. The objective of this work was to design an acquisition and processing system that can perform 3D scanning and reconstruction of objects seamlessly. In addition, the goal of this work also included making the 3D scanning process fully automated by building and integrating a turntable alongside the software. This means the user can perform a full 3D scan only by a press of a few buttons from our dedicated graphical user interface. Three main steps were followed to go from acquisition of point clouds to the finished reconstructed 3D model. First, our system acquires point cloud data of a person/object using inexpensive camera sensor. Second, align and convert the acquired point cloud data into a watertight mesh of good quality. Third, export the reconstructed model to a 3D printer to obtain a proper 3D print of the model.

Application of the Kinect sensor for three dimensional characterization of vine canopy

Monitoring grapevine canopy size and evolution during time is of great interest for the management of the vineyard. An interesting and cost effective solution for 3D characterization is provided by the Kinect sensor. To assess its practical applicability, field experiments were carried out on two different grapevines varieties (Glera and Merlot) for a three months period. The results from 3D digital imaging were compared with those achieved by direct hand-made measurements. Estimated volume was then effectively correlated to the number of leaves and to the leaf area index. The experiments demonstrated how a low cost 3D sensor can be applied for fast and repeatable reconstruction of vine vegetation, opening up for new potential improvements in variable rate application or pruning

Color Reduction in an Authenticate Live 3D Point Cloud Video Streaming System

In this paper, an authenticate live 3D point cloud video streaming system is presented, using a low cost 3D sensor camera, the Microsoft Kinect. The proposed system is implemented on a client-server network infrastructure. The live 3D video is captured from the Kinect RGB-D sensor, then a 3D point cloud is generated and processed. Filtering and compression are used to handle the spatial and temporal redundancies. A color histogram based conditional filter is designed to reduce the color information for each frame based on the mean and standard deviation. In addition to the designed filter, a statistical outlier removal filter is used. A certificate-based authentication is used where the client will verify the identity of the server during the handshake process. The processed 3D point cloud video is live streamed over a TCP/IP protocol to the client. The system is evaluated in terms of: compression ratio, total bytes per points, peak signal to noise ratio (PSNR), and Structural Similarity (SSIM) index. The experimental results demonstrate that the proposed video streaming system have a best case with SSIM 0.859, PSNR of 26.6 dB and with average compression ratio of 8.42 while the best average compression ratio case is about 15.43 with PSNR 18.5128 dB of and SSIM 0.7936.

COMBINATION OF TLS POINT CLOUDS AND 3D DATA FROM KINECT V2 SENSOR TO COMPLETE INDOOR MODELS

The combination of data coming from multiple sensors is more and more applied for remote sensing issues (multi-sensor imagery) but also in cultural heritage or robotics, since it often results in increased robustness and accuracy of the final data. In this paper, the reconstruction of building elements such as window frames or door jambs scanned thanks to a low cost 3D sensor (Kinect v2) is presented. Their combination within a global point cloud of an indoor scene acquired with a terrestrial laser scanner (TLS) is considered. If the added elements acquired with the Kinect sensor enable to reach a better level of detail of the final model, an adapted acquisition protocol may also provide several benefits as for example time gain. The paper aims at analyzing whether the two measurement techniques can be complementary in this context. The limitations encountered during the acquisition and reconstruction steps are also investigated.

COMBINATION OF TLS POINT CLOUDS AND 3D DATA FROM KINECT V2 SENSOR TO COMPLETE INDOOR MODELS

Abstract. The combination of data coming from multiple sensors is more and more applied for remote sensing issues (multi-sensor imagery) but also in cultural heritage or robotics, since it often results in increased robustness and accuracy of the final data. In this paper, the reconstruction of building elements such as window frames or door jambs scanned thanks to a low cost 3D sensor (Kinect v2) is presented. Their combination within a global point cloud of an indoor scene acquired with a terrestrial laser scanner (TLS) is considered. If the added elements acquired with the Kinect sensor enable to reach a better level of detail of the final model, an adapted acquisition protocol may also provide several benefits as for example time gain. The paper aims at analyzing whether the two measurement techniques can be complementary in this context. The limitations encountered during the acquisition and reconstruction steps are also investigated.

Interactive 3D object recognition pipeline on mobile GPGPU computing platforms using low-cost RGB-D sensors

In this work, we propose the implementation of a 3D object recognition system which will be optimized to operate under demanding time constraints. The system must be robust so that objects can be recognized properly in poor light conditions and cluttered scenes with significant levels of occlusion. An important requirement must be met: The system must exhibit a reasonable performance running on a low power consumption mobile GPU computing platform (NVIDIA Jetson TK1) so that it can be integrated in mobile robotics systems, ambient intelligence or ambient-assisted living applications. The acquisition system is based on the use of color and depth (RGB-D) data streams provided by low-cost 3D sensors like Microsoft Kinect or PrimeSense Carmine. The resulting system is able to recognize objects in a scene in less than 7 seconds, offering an interactive frame rate and thus allowing its deployment on a mobile robotic platform. Because of that, the system has many possible applications, ranging from mobile robot navigation and semantic scene labeling to human–computer interaction systems based on visual information. A video showing the proposed system while performing online object recognition in various scenes is available on our project website ( http://www.dtic.ua.es/~agarcia/3dobjrecog-jetsontk1/ ).

3D Surface Reconstruction of Noisy Point Clouds Using Growing Neural Gas: 3D Object/Scene Reconstruction

With the advent of low-cost 3D sensors and 3D printers, scene and object 3D surface reconstruction has become an important research topic in the last years. In this work, we propose an automatic (unsupervised) method for 3D surface reconstruction from raw unorganized point clouds acquired using low-cost 3D sensors. We have modified the growing neural gas network, which is a suitable model because of its flexibility, rapid adaptation and excellent quality of representation, to perform 3D surface reconstruction of different real-world objects and scenes. Some improvements have been made on the original algorithm considering colour and surface normal information of input data during the learning stage and creating complete triangular meshes instead of basic wire-frame representations. The proposed method is able to successfully create 3D faces online, whereas existing 3D reconstruction methods based on self-organizing maps required post-processing steps to close gaps and holes produced during the 3D reconstruction process. A set of quantitative and qualitative experiments were carried out to validate the proposed method. The method has been implemented and tested on real data, and has been found to be effective at reconstructing noisy point clouds obtained using low-cost 3D sensors.

Face recognition based on Kinect

In this paper, we present a new algorithm that utilizes low-quality red, green, blue and depth (RGB-D) data from the Kinect sensor for face recognition under challenging conditions. This algorithm extracts multiple features and fuses them at the feature level. A Finer Feature Fusion technique is developed that removes redundant information and retains only the meaningful features for possible maximum class separability. We also introduce a new 3D face database acquired with the Kinect sensor which has released to the research community. This database contains over 5,000 facial images (RGB-D) of 52 individuals under varying pose, expression, illumination and occlusions. Under the first three variations and using only the noisy depth data, the proposed algorithm can achieve 72.5 % recognition rate which is significantly higher than the 41.9 % achieved by the baseline LDA method. Combined with the texture information, 91.3 % recognition rate has achieved under illumination, pose and expression variations. These results suggest the feasibility of low-cost 3D sensors for real-time face recognition.

Kinect 3D Point Cloud Live Video Streaming

We present a live video streaming system using a low cost 3D sensor camera like the Microsoft Kinect. The huge amount of raw point data that Kinect creates has to be stored and transmitted by efficient compact means. Noise and redundancy, however, make the process more difficult to achieve. To overcome these difficulties we propose a live streaming system that streams a 3D video to an Android Mobile phone and to Linux Desktop systems. For Android mobile phone client, the 3D video is filtered before streaming. Filtering stage contains 3 types of filters, Voxel Grid, Statistical Outlier Removal and Histogram-based conditional filters. The video is captured by the Kinect and a 3D point cloud is created. Voxel Grid filter is used since the generated 3D video will have millions of points; a downsampling procedure is applied to minimize the number of points. To reduce outliers and color information, statistical outlier removal and histogram-based conditional filters are used respectively. Conditional filter is customized by the scene histogram for each channel of RGB color to preserve the dominant color information for each scene. For Linux desktop client, the video is filtered by the histogram-based conditional filter and is compressed using an Octree structure that reduces spatial redundancies across the streamed video.

A multi-resolution approach for an automated fusion of different low-cost 3D sensors.

The 3D acquisition of object structures has become a common technique in many fields of work, e.g., industrial quality management, cultural heritage or crime scene documentation. The requirements on the measuring devices are versatile, because spacious scenes have to be imaged with a high level of detail for selected objects. Thus, the used measuring systems are expensive and require an experienced operator. With the rise of low-cost 3D imaging systems, their integration into the digital documentation process is possible. However, common low-cost sensors have the limitation of a trade-off between range and accuracy, providing either a low resolution of single objects or a limited imaging field. Therefore, the use of multiple sensors is desirable. We show the combined use of two low-cost sensors, the Microsoft Kinect and the David laserscanning system, to achieve low-resolved scans of the whole scene and a high level of detail for selected objects, respectively. Afterwards, the high-resolved David objects are automatically assigned to their corresponding Kinect object by the use of surface feature histograms and SVM-classification. The corresponding objects are fitted using an ICP-implementation to produce a multi-resolution map. The applicability is shown for a fictional crime scene and the reconstruction of a ballistic trajectory.