Patch-based Multi-view Stereo Research Articles

3D Reconstruction and Measurement Analysis of a Dense Point Cloud Fused with a Depth Image

To solve the problems of holes, noise, and texture information missing in the traditional incremental reconstruction of complex surface objects, a 3D reconstruction method of depth image fusion surface dense point clouds is proposed, and texture feature creation is combined to obtain a 3D reconstruction model that takes into account the main body and details of the reconstructed object. First, the mechanism of surface dense reconstruction based on the patch-based multiview stereo (PMVS) algorithm is analyzed. Combined with the principle of view angle selection of stereo images, surface point cloud density reconstruction is performed. Then, the depth value is optimized by the region growing method, and the optimization model is established. The depth image is fused into a dense surface, and the reconstructed part is supplemented by the depth information. Finally, the Markov random field (MRF) is introduced to describe the richness of image details, and combined with the calculating method of the area coordinate, the texture coordinates are accurately calculated to reproduce the texture details of the 3D reconstruction model. 3D reconstruction experiments are performed on multiple indoor and outdoor model surfaces, and the experimental results show that the proposed method can achieve complete and accurate reconstruction of complex surface objects. Our method provides technical support for complex surface topography detection and has industrial practical significance.

基于三维模型的谷子表型特征分析研究

As one of the ancient cultivated crops in China, millet has the characteristics of high nutritional value, drought resistance and barrenness. It also plays an important role in ensuring the supply of food in our country. At present, most of the millet breeding work uses manual extraction of phenotypic information, which is laborintensive and inefficient. Therefore, the development of an automated, efficient and accurate millet phenotype detection method has practical significance for the extraction of the millet genome. In this study, a combination of sparse reconstruction based on Structure from Motion (SfM) and Patch-based Multi-View Stereo (PMVS) was used to select three different varieties of millet. A total of 81 samples of 9 samples in each period were reconstructed to obtain a 3D model of millet. The combination of conditional filtering and statistical filtering is used to remove the noise points generated during the photographing process, and finally the obtained point cloud data is used to measure the agronomic traits of millet such as plant height and leaf area. The results show that the interval angle of 5° is the best reconstruction angle of millet. The coefficient of determination R2 of point cloud measurement results and manual measurement data regression analysis is higher than 0.94, indicating that the method used for 3D reconstruction has high applicability to different millet in different periods and high-throughput measurement of millet by the method in this paper is feasible. This study provides a theoretical basis for a millet phenotypic information measurement device

Research on 3D Phenotypic Reconstruction and Micro-Defect Detection of Green Plum Based on Multi-View Images

Rain spots on green plum are superficial micro-defects. Defect detection based on a two-dimensional image is easily influenced by factors such as placement position and light and is prone to misjudgment and omission, which are the main problems affecting the accuracy of defect screening of green plum. In this paper, using computer vision technology, an improved structure from motion (SFM) and patch-based multi-view stereo (PMVS) algorithm based on similar graph clustering and graph matching is proposed to perform three-dimensional sparse and dense reconstruction of green plums. The results show that, compared with the traditional algorithm, the running time of this algorithm is lower, at only 26.55 s, and the mean values of camera optical center error and pose error are 0.019 and 0.631, respectively. This method obtains a higher reconstruction accuracy to meet the subsequent plum micro-defect detection requirements. Aiming at the dense point cloud model of green plums, through point cloud preprocessing, the improved adaptive segmentation algorithm based on the Lab color space realizes the effective segmentation of the point cloud of green plum micro-defects. The experimental results show that the average running time of the improved adaptive segmentation algorithm is 2.56 s, showing a faster segmentation speed and better effect than the traditional K-means and K-means++ algorithms. After clustering the micro-defect point cloud, the micro-defect information of green plums was extracted on the basis of random sample consensus (RANSAC) plane fitting, which provides a theoretical model for further improving the accuracy of sorting the appearance quality of green plums.

Analysis of the Style Characteristics of National Dance Based on 3D Reconstruction.

National dance is an important symbol of national spiritual culture, as it embodies each nation's unique history, living habits, ideology, and culture. Many Chinese ethnic groups have developed their own dance forms and styles, each with their own set of charms. Strengthening our understanding of its style and characteristics is critical to our understanding of the national dance. To begin with, this paper proposes a patch-based multiview stereo reconstruction scheme that can accurately reconstruct 3D character models. The movement capture system then collects national dance movement data and uses them to drive a three-dimensional character model that generates dance visual animation. In the human skeleton model, a spatial-temporal HMM (hidden Markov model) is proposed that not only studies the temporal information of human movements but also the spatial information of adjacent connection points. The recognition rate of this algorithm increases to 95.68 percent when the number of setting points is increased to ten and to 98.28 percent when the number of setting points is increased to twenty, according to the research. The experimental results show that the 3D reconstruction model of ethnic dance developed in this paper is capable of meeting the goal of ethnic dance digitalization while being simple to use, being low in cost, and with a good display effect.

APPLICATION OF RGB-D SLAM IN 3D TUNNEL RECONSTRUCTION BASED ON SUPERPIXEL AIDED FEATURE TRACKING

Abstract. In large-scale projects such as hydropower and transportation, the real-time acquisition and generation of the 3D tunnel model can provide an important basis for the analysis and evaluation of the tunnel stability. The Simultaneous Localization And Mapping (SLAM) technology has the advantages of low cost and strong real-time, which can greatly improve the data acquisition efficiency during tunnel excavation. Feature tracking and matching are critical processes of traditional 3D reconstruction technologies such as Structure from Motion (SfM) and SLAM. However, the complicated rock mass structures on the tunnel surface and the limited lighting environment make feature tracking and matching difficult. Manhattan SLAM is a technology integrating superpixels and Manhattan world assumptions, in which both line features and planar features can be better extracted. Rock mass discontinuities including traces and structural planes are distributed on the inner surface of tunnels, which can be extracted for feature tracking and matching. Therefore, this paper proposes a 3D reconstruction pipeline for tunnels, in which, the Manhattan SLAM algorithm is applied for camera pose parameters estimation and the sparse point cloud generation, and the Patch-based Multi-View Stereo (PMVS) is adopted for dense reconstruction. In this paper, the Azure Kinect DK sensor is used for data acquisition. Experiments are proceeded and the results show that the proposed pipeline based on Manhattan SLAM and PMVS performs good robustness and feasibility for tunnels 3D reconstruction.

DENSE RECONSTRUCTION FOR TUNNELS BASED ON THE INTEGRATION OF DOUBLE-LINE PARALLEL PHOTOGRAPHY AND DEEP LEARNING

Abstract. In scenes of tunnels and underground engineering projects, the 3D modeling based on the traditional horizontal baseline photography method is difficult to make a trade-off between the modeling efficiency and image overlap under narrow space and close photography constraint conditions. Parallel photography provides a better alternative. Furthermore, the undulating tunnel surfaces make occlusion unavoidable, and the pixel scales vary evidently in parallel photography, both of which make it difficult to obtain expected models neither by Semi-Global Matching nor by Patch-based Multi-view Stereo techniques. Comparatively, more accurate 3D reconstruction results can be achieved by using learning techniques, which consider the global semantic information such as specular prior and reflective prior to making the matching more robust. Besides, it is convenient for geologists to acquire photos with smartphones or cameras by a single-line parallel photography method at the tunnel sites. But enough image overlap for reconstruction is still difficult for the above photography way. Therefore, this paper proposes a dense reconstruction method for tunnels using deep learning and double-line parallel photography techniques, and respectively makes comparisons with single-line parallel photography and traditional modeling methods. Experimental results show the feasibility and robustness of the proposed method, especially for tunnel surfaces with complicated textures and occlusions. Moreover, both the completeness and accuracy of the tunnel model with double-line photography are better than with single-line photography.

Three-Dimensional Reconstruction of Tunnel Face Based on Multiple Images

The current geological sketch in tunnel engineering is mainly based on sketches of workers. However, geological sketch drawn by workers always offers fundamental data purely due to its drawing mode. A novel drawing method for geological sketch has been introduced using multiview photos in this process. The images of tunnel faces are taken from multiple angles, and every two pictures have overlaps. By measuring the distance between the camera and the tunnel face using a laser range finder, the photographic scale of each photo can be confirmed. SpeededUp Robust Features (SURF) is a good practice for detecting feature points, and the sparse point cloud is reconstructed from multiview photos by structure from motion (SFM). However, the sparse point cloud is not suitable for analysis for structural planes due to its sparsity. Therefore, patch-based multiview stereo (PMVS) is used to reconstruct dense point cloud from the sparse point cloud. After 3D reconstruction, the details of the tunnel face are recorded. The proposed technique was applied to multiview photos acquired in the Xiaosanxia railway tunnel and Fengjie tunnel in Chongqing, China. In order to record the geological conditions of the tunnel face quickly and accurately, Chengdu Tianyou Tunnelkey has developed a set of software and hardware integration system called CameraPad. Besides, CameraPad was used to collect the multiview photos of the tunnel face in the No. 1 Xinan railway tunnel in Jilin, China. By comparing with traditional and existing methods, the proposed method offers a more reductive model for geological conditions of the tunnel face.

A novel objective wrinkle evaluation method for printed fabrics based on multi-view stereo algorithm

Traditionally, fabric wrinkle assessment is based on human eyes, which is subjective with the disadvantages of low efficiency and accuracy. In this paper, an objective fabric wrinkle evaluation method was proposed to solve the previous problems due to the subjective evaluation. A self-developed multiple image acquisition system was established for the image capturing of wrinkled fabrics from different direction. Three-dimensional (3D) surface profile of wrinkled fabric was reconstructed by an improved patch-based multi-view stereo vision algorithm. A two-dimensional depth image could be generated directly from the 3D point cloud model, after that, four texture feature parameters were extracted from the depth image using gray level co-occurrence matrix. Finally, these four feature parameters were selected as the input vector and wrinkle grade as the output to form a support vector machine for the objective assessment of fabric wrinkle appearance. Our experimental results indicated that the recognition accuracy of the proposed method and system was more than 90%. The originality of our research is that the wrinkle features based on 3D surface profile reconstructing can avoid interference caused by fabric color and texture and further improve the recognition accuracy of objective evaluation.

Objective evaluation of fabric pilling based on multi-view stereo vision

Fabric pilling evaluation is very important for the quality control of textile industry. Traditional image analysis based methods have the disadvantages of 2 D imaging and color sensitivity, this paper presents a new method based on stereo vision to solve the problem of 3 D imaging of fabric pilling. One set of self-developed mobile camera system is established to capture a group of images for the 3 D reconstruction of the fabric surface, the point cloud model of the fabric surface is generated by the self-developed stereo vision algorithm, including structure from motion (SFM) and patch-based multi-view stereo (PMVS) algorithm. One 2 D gray-scale image is obtained from the 3 D point cloud model by mapping to the 2 D image plane, which contains the depth information of fabric pilling. The segmentation of fabric pilling could be done by accurate positioning of edge detection, adaptive thresholding and morphological analysis. Four feature parameters including pilling number, pilling area, pilling density and coverage ratio are extracted for the determination of fabric pilling grade objectively. Experimental results show that the new developed system and method is effective and reliable for the fabric pilling evaluation, which is consistent with the subjective pilling evaluation. It is workable for the color printed or yarn dyed fabrics, the proposed imaging system could be a good solution for the digital intelligent quality control of textile products.

An Improved Building Reconstruction Algorithm Based on Manhattan World Assumption and Line-Restricted Hypothetical Plane Fitting

An improved patch-based multiview stereo (PMVS) algorithm based on Manhattan world assumption and the line-restricted hypothetical plane fitting method according to buildings’ spatial characteristics is proposed. Different from the original PMVS algorithm, our approach generates seed points purely from 3D line segments instead of using those feature points. First, 3D line segments are extracted using the existing Line3D++ algorithm, and the 3D line segment clustering criterion of buildings is established based on Manhattan world assumption. Next, by using the normal direction obtained using the result of 3D line segment clustering, we propose a multihypothetical plane fitting algorithm based on the mean shift method. Then, through subdividing on the triangle mesh constructed based on the building hypothetical plane model, semidense point cloud can be quickly obtained, and it is used as seed points of the PMVS pipeline instead of the sparse and noisy seed points generated by PMVS itself. After that, dense point cloud can be obtained through the existing PMVS expansion pipeline. Finally, unit and integration experiments are designed; the test results show that the proposed algorithm is 15%∼23% faster than the original PMWS in running time, and at the same time, the reconstruction quality of buildings is improved as well by successfully removing many noise points in the buildings.

Investigation of 3D surface profile reconstruction technology for automatic evaluation of fabric smoothness appearance

Objective evaluation of smoothness appearance of fabrics after repeated home laundering is not only the key point, but also a difficulty in the field of textile inspection and clothing industry. This paper reports a 3D reconstruction system and method for objective assessment of fabric smoothness appearance based on Multi-view Stereo Vision (MVS) and Bag-of-Features (BOF). An image acquisition system for texture-less fabric was designed with a specific illumination module. Three-dimensional surface profile of wrinkled fabric was reconstructed by Structure from Motion (SFM) and Patch-based Multi-view Stereo (PMVS) techniques. To facilitate feature extraction, the generated 3D point cloud was mapped to a 2D image plane. The Bag-of-Features (BOF) model was utilized to characterize the fabric wrinkles and the K-Nearest Neighbor (KNN) algorithm was used to achieve automatic evaluation of fabric smoothness appearance. The experimental results with a high recognition of 93.33% indicated the effectiveness and stability of the proposed method.

Folding patch correspondence for multiview stereo

AbstractIn this article, we propose the novel folding patch model which can replace the traditional patch model utilized in patch‐based multiview stereo (MVS) methods to significantly improve the reconstruction results. The patch model is applied as an approximation of the scene surface differential in the geometric estimation procedure. By minimizing the photometric discrepancy of the projection of the patch model on multiple source images, patch‐based MVS algorithms optimize the position and normal values for the 3D hypothesis of the target pixel. The optimization is based on the assumption that the patch model can fit the target scene surface perfectly. However, when it comes to complex scenes crowded with sharp edges, splintery surfaces, or round surfaces, the patch model is inherently not suitable since even from the microscopic perspective these surfaces are not entirely flat. We construct the folding patch model by folding the traditional patch model from the middle line. By adjusting the folding angle and direction, the folding patch model can fit complex surfaces more flexibly. We apply our folding patch model to the representative open‐source patch based multiview stereo (PMVS) and COLMAP, and validate the effectiveness on ETH3D benchmark and data sets captured in nature. The results demonstrate that utilizing the folding patch model can significantly improve the behavior of PMVS and COLMAP, especially on data sets mainly consist of complex surfaces from plants.

DEVELOPMENT OF MULTI-VIEW STEREO CONSIDERING ACCURACY OF EXTERIOR ORIENTATION ELEMENTS

Abstract. Structure from motion (SfM) has been widely used to achieve automatic 3D reconstructions. However, as the 3D point clouds obtained via SfM are sparse, multi-view stereo (MVS) was developed to compensate for this sparseness. The accuracy of the 3D surface depends on the accuracy of the orientation elements based on the SfM. Additionally, in the case of an unmanned aerial vehicle (UAV), SfM exhibits a decrease in the accuracy of the orientation elements during complex camera movements. This paper proposes a patch-based MVS (PMVS) method considering the accuracy of the orientation elements. The proposed method involves applying the global SfM, estimating accuracy of exterior orientation (EO) elements, and introducing the accuracy of EO elements to PMVS. The PMVS approximates an object surface by using small rectangular patches, namely local tangent plane approximation. The patches are optimized by minimizing the sum of the photometric discrepancy scores. The accuracy of the EO elements is introduced to the patch optimization as weighting function. This accuracy is defined using the variances of the estimated parameters in the bundle adjustment. We also investigate the types of weighting functions. The results indicate that the proposed method is capable of considering geometric conditions during patch estimation. The proposed method was applied to the three types of image datasets, i.e., images captured using an SLR camera at ground level, images captured using a UAV equipped with a SLR camera, and images captured using an airplane equipped with an oblique camera. Through the experimental results, the improved accuracy and the effectiveness of the proposed method were confirmed.

Automated Progress Controlling and Monitoring Using Daily Site Images and Building Information Modelling

This research presents a novel method for automated construction progress monitoring. Using the proposed method, an accurate and complete 3D point cloud is generated for automatic outdoor and indoor progress monitoring throughout the project duration. In this method, Structured-from-Motion (SFM) and Multi-View-Stereo (MVS) algorithms coupled with photogrammetric principles for the coded targets’ detection are exploited to generate as-built 3D point clouds. The coded targets are utilized to automatically resolve the scale and increase the accuracy of the point cloud generated using SFM and MVS methods. Having generated the point cloud, the CAD model is generated from the as-built point cloud and compared with the as-planned model. Finally, the quantity of the performed work is determined in two real case study projects. The proposed method is compared to the Structured-from-Motion (SFM)/Clustering Multi-Views Stereo (CMVS)/Patch-based Multi-View Stereo (PMVS) algorithm, as a common method for generating 3D point cloud models. The proposed photogrammetric Multi-View Stereo method reveals an accuracy of around 99 percent and the generated noises are less compared to the SFM/CMVS/PMVS algorithm. It is observed that the proposed method has extensively improved the accuracy of generated points cloud compared to the SFM/CMVS/PMVS algorithm. It is believed that the proposed method may present a novel and robust tool for automated progress monitoring in construction projects.

Optimal Multi-View Surface Normal Estimation Using Affine Correspondences.

An optimal, in the least squares sense, method is proposed to estimate surface normals in both stereo and multi-view cases. The proposed algorithm exploits exclusively photometric information via affine correspondences and estimates the normal for each correspondence independently. The normal is obtained as a root of a quartic polynomial. Therefore, the processing time is negligible. Eliminating the outliers, we propose a robust extension of the algorithm that combines maximum likelihood estimation and iteratively re-weighted least squares. The method has been validated on both synthetic and publicly available real-world datasets. It is superior to the state of the art in terms of accuracy and processing time. Besides, we demonstrate two possible applications: 1) using our algorithm as the seed-point generation step of patch-based multi-view stereo method, the obtained reconstruction is more accurate, and the error of the 3D points is reduced by 30% on average and 2) multi-plane fitting becomes more accurate applied to the resulting oriented point cloud.

3D measurements of micro-objects based on monocular wide-field optical microscopy with extended depth of field.

Monocular rotational microscopes suffer from shallow depth of field and reduced field of view. Consequently, we propose monocular wide-field optical microscopy with extended depth of field to enable accurate 3D measurements of micro-objects. We use a liquid lens and a 4f system to extend the system's depth of field and two additional rotational reflectors to capture multi-view image sequences. In addition, we increase the number of expansion directions in the patch-based multi-view stereo algorithm and optimize the expansion radius. Our approach outperforms that based on the patch-based multi-view stereo method and achieves a high relative measurement accuracy of 1.9%. RESEARCH HIGHLIGHTS: This study proposes 3D measurements of micro-objects via monocular rotational microscopy. The depth of field is extended by the use of a liquid lens and a 4f system. Our approach outperforms that based on the patch-based multi-view stereo method and achieves a high relative measurement accuracy of 1.9%.

Research on three-dimensional model reconstruction of slope erosion based on sequence images

To solve the problems of incomplete 2D display soil erosion, difficult visual expression, poor detail simulation of eroding slope, complex modeling technology, and high cost, a three-dimensional model reconstruction method for slope erosion based on sequence images is proposed. The method mainly uses structure from motion (SFM) and multi-view stereo (MVS) technology. First, the image sequence is feature matched using the scale-invariant feature transform algorithm for images captured by ordinary cameras. The random sampling and consensus algorithm is used to remove the false matching point generated by the nearest neighbor. Next, using the SFM method, the camera matrix and three-dimensional point coordinates are solved iteratively, and the nonlinear optimization is realized by bundle adjustment to distribute the errors evenly and improve the accuracy of the model. Finally, based on the patch-based multi-view stereo algorithm, the sparse point cloud generated by SFM is used as the seed patch to diffuse under local photometric uniformity and global visibility constraints to reconstruct the dense point cloud and establish the 3D model. To verify the reliability of the experimental methods, this paper takes the red soil erosion slope in Chenggong District of Kunming, Yunnan, China as the experimental object and realizes the rapid reconstruction of the three-dimensional model of the erosion gully on the slope. Multiple groups of adjacent eroded gully images are taken to verify the model, which involves method and accuracy of model. The results show that the modeling method has a good effect on soil modeling. The error from the control points is small and the relative accuracy is as high as 94.20%, 98.01%, etc. As a result, the control points can provide a reference for corrosion monitoring research for its reliable results.

3D Reconstruction of Space Objects from Multi-Views by a Visible Sensor.

In this paper, a novel 3D reconstruction framework is proposed to recover the 3D structural model of a space object from its multi-view images captured by a visible sensor. Given an image sequence, this framework first estimates the relative camera poses and recovers the depths of the surface points by the structure from motion (SFM) method, then the patch-based multi-view stereo (PMVS) algorithm is utilized to generate a dense 3D point cloud. To resolve the wrong matches arising from the symmetric structure and repeated textures of space objects, a new strategy is introduced, in which images are added to SFM in imaging order. Meanwhile, a refining process exploiting the structural prior knowledge that most sub-components of artificial space objects are composed of basic geometric shapes is proposed and applied to the recovered point cloud. The proposed reconstruction framework is tested on both simulated image datasets and real image datasets. Experimental results illustrate that the recovered point cloud models of space objects are accurate and have a complete coverage of the surface. Moreover, outliers and points with severe noise are effectively filtered out by the refinement, resulting in an distinct improvement of the structure and visualization of the recovered points.

A system for classifying vegetative structures on coffee branches based on videos recorded in the field by a mobile device

An expert system development to classify six coffee vegetative structures.A non-destructive system based on a smartphone camera and computer vision techniques.A system robust to illumination and occlusion changes using 2D and 3D features. As a drink, coffee is one of the most in demand products worldwide; as an agricultural product, it requires non-destructive tools for its monitoring and control. In order to create a non-destructive method which can be used in the field, a system was developed to find and classify six types of vegetative structures on coffee branches: leaves, stems, flowers, unripe fruits, semi-ripe fruits, and ripe fruits. Videos were obtained from 12 coffee branches in field conditions, using the rear camera of a mobile device. Approximately 90 frames, those which had the most information from the scene, were selected from each video. Next, a three-dimensional (3D) reconstruction was generated using the Structure from Motion (SfM) and Patch-based Multi-view Stereo (PMVS) techniques for each branch. All acquired images were manually recorded, and a Ground Truth point cloud was generated for each branch. The generated point clouds were filtered using a statistical outliers filter, in order to eliminate noise generated in the 3D reconstruction process. The points that were located in the deepest part were considered to be the scene background, and were removed using a band-pass filter. Point clouds were sub-sampled using a VoxelGrid filter, to reduce the number of points to 50% and therefore reduce computation time of the processes that followed. Various two-dimensional (2D) and 3D features were taken from the point clouds: 11 based on RGB, Lab, Luv, YCbCr, and HSV color space, four based on curvatures, and the remaining two based on shape and curvedness indexes. A Support Vector Machine (SVM) was trained with the previously mentioned features by using eight branches for the training stage, and four branches for the validation stage. Experimental results showed a precision of 0.82 and a recall of 0.79, when classifying said vegetative structures. The proposed system is economical, as only a mobile device is needed to obtain information. Remaining system processes were performed offline. Additionally, the system developed was not affected by changes in lighting conditions, when recording videos on a coffee plantation.

Adaptive segmentation for multi‐view stereo

This study presents an adaptive segmentation method for pre‐processing input data to the patch‐based multi‐view stereo algorithm. A specially developed greyscale transformation is applied to the input image data, thus redefining the intensity histogram. The Nelder–Mead simplex method is used to adaptively locate an optimised segmentation threshold point in the modified histogram. The transformed input image is then segmented using the acquired threshold value, into foreground and background data. The segmentation information acquired is applied to the initial feature extraction and the cyclic patch‐expansion procedure to constrain the reconstruction to a three‐dimensional visibility space that excludes background artefacts. The method is targeted at segmenting out potentially disruptive data and is able to realise a reduction in cumulative error of the reconstruction process and thus improve the final reconstruction. With this method, the authors obtain results that are relatively similar to the original patch‐based method, but with reduced time and space complexity.