Multi-view Stereo Vision Research Articles

Flexible technique for global calibration of multi-view stereo vision with a non-overlapping FOV using linear structured light projector

A flexible global calibration method for multi-view stereo vision with non-overlapping field of view (FOV) using linear structured light projector is presented in this paper. At least three groups of linear structured light plane equations are utilized to calculate the rotation between any pair of cameras with non-overlapping FOV, and then the closed solution of the translation is determined based on the constraint that the normal vector is perpendicular to the lines on linear structured light plane. Once the rotation and translation between all pairs of cameras with non-overlapping FOV are obtained, the rotation and translation of each camera relative to the reference position is eventually computed through global optimization. The experimental data reflects that the calibration accuracy is improved by 35.2% after global optimization, and the global calibration results of the proposed method deviates less from that of Zhang’s method.

An Improved TransMVSNet Algorithm for Three-Dimensional Reconstruction in the Unmanned Aerial Vehicle Remote Sensing Domain.

It is important to achieve the 3D reconstruction of UAV remote sensing images in deep learning-based multi-view stereo (MVS) vision. The lack of obvious texture features and detailed edges in UAV remote sensing images leads to inaccurate feature point matching or depth estimation. To address this problem, this study improves the TransMVSNet algorithm in the field of 3D reconstruction by optimizing its feature extraction network and costumed body depth prediction network. The improvement is mainly achieved by extracting features with the Asymptotic Pyramidal Network (AFPN) and assigning weights to different levels of features through the ASFF module to increase the importance of key levels and also using the UNet structured network combined with an attention mechanism to predict the depth information, which also extracts the key area information. It aims to improve the performance and accuracy of the TransMVSNet algorithm's 3D reconstruction of UAV remote sensing images. In this work, we have performed comparative experiments and quantitative evaluation with other algorithms on the DTU dataset as well as on a large UAV remote sensing image dataset. After a large number of experimental studies, it is shown that our improved TransMVSNet algorithm has better performance and robustness, providing a valuable reference for research and application in the field of 3D reconstruction of UAV remote sensing images.

CGAN-Based Forest Scene 3D Reconstruction from a Single Image

Forest scene 3D reconstruction serves as the fundamental basis for crucial applications such as forest resource inventory, forestry 3D visualization, and the perceptual capabilities of intelligent forestry robots in operational environments. However, traditional 3D reconstruction methods like LiDAR present challenges primarily because of their lack of portability. Additionally, they encounter complexities related to feature point extraction and matching within multi-view stereo vision sensors. In this research, we propose a new method that not only reconstructs the forest environment but also performs a more detailed tree reconstruction in the scene using conditional generative adversarial networks (CGANs) based on a single RGB image. Firstly, we introduced a depth estimation network based on a CGAN. This network aims to reconstruct forest scenes from images and has demonstrated remarkable performance in accurately reconstructing intricate outdoor environments. Subsequently, we designed a new tree silhouette depth map to represent the tree’s shape as derived from the tree prediction network. This network aims to accomplish a detailed 3D reconstruction of individual trees masked by instance segmentation. Our approach underwent validation using the Cityscapes and Make3D outdoor datasets and exhibited exceptional performance compared with state-of-the-art methods, such as GCNDepth. It achieved a relative error as low as 8% (with an absolute error of 1.76 cm) in estimating diameter at breast height (DBH). Remarkably, our method outperforms existing approaches for single-image reconstruction. It stands as a cost-effective and user-friendly alternative to conventional forest survey methods like LiDAR and SFM techniques. The significance of our method lies in its contribution to technical support, enabling the efficient and detailed utilization of 3D forest scene reconstruction for various applications.

Research on 3D reconstruction method of cattle face based on image

As the basis of intelligent breeding management and animal husbandry insurance, the identification of individual cattle is important in animal husbandry management. Given the difficulty of data acquisition caused by the non-rigid and lacking cooperation of cattle, this study proposes a method for cattle face image acquisition and processing that can efficiently adapt to the harsh environment of cattle barns. When processing the non-rigid cow face, the method of approximating the cow face to a rigid body is used to establish the cow face image data set., and the cattle face image data set is established. The Three Dimensional(3D) reconstruction method of cattle face uses a 3D image reconstruction method based on multiple perspectives. First, the scale-invariant feature transform algorithm is used to extract the image feature points. The fast library for approximate nearest neighbors algorithm is used to match feature points. The matching results are selected via random sampling consensus. Second, the structure of the motion method is used for the sparse reconstruction of point clouds, and the dense point cloud is then generated using the three-dimensional multi-view stereo vision algorithm. Finally, the Poisson surface reconstruction method is used for surface reconstruction. The results indicate that this method can effectively realize the three-dimensional reconstruction of cattle faces; the reconstructed images have obvious color, clear texture, and complete shape features.

GEMVS: a novel approach for automatic 3D reconstruction from uncalibrated multi-view Google Earth images using multi-view stereo and projective to metric 3D homography transformation

ABSTRACT This paper proposes a novel approach for automatic 3D surface reconstruction from uncalibrated and multi-view Google Earth images by using a multi-view stereo method and 3D projective to metric transformation. Without the Rational Polynomial Coefficients, it is impossible to obtain the metric reconstruction of the 3D surface from multi-view satellite images. We solve the uncalibrated multi-view satellite image problem by employing a multi-view stereo vision technique followed by a projective to metric transformation. The virtual pose parameters of the satellite images are obtained by using COLMAP, and the virtual 3D projective reconstruction is done by using EnSoft3D. For projective to metric transformation, we propose to employ 3D homography transformation. Eight 3D correspondence pairs on the viewing frustums between the virtual reference camera and the ideal nadir camera are used to derive a 3D homography matrix. Using the 3D homography matrix, we finally obtain the metric reconstruction of 3D surface up to an unknown height scale in a reference coordinate system of the Google Earth desktop software. Experiments are done in several world locations on Google Earth including building and vegetation areas. Reconstruction error analysis with the Data Fusion Contest 19 dataset is also presented. The average of MAE and RMSE of five tile regions in the dataset are 1.596 m and 2.083 m, respectively.

Multi-View Stereo Vision Patchmatch Algorithm Based on Data Augmentation

In this paper, a multi-view stereo vision patchmatch algorithm based on data augmentation is proposed. Compared to other works, this algorithm can reduce runtime and save computational memory through efficient cascading of modules; therefore, it can process higher-resolution images. Compared with algorithms utilizing 3D cost volume regularization, this algorithm can be applied on resource-constrained platforms. This paper applies the data augmentation module to an end-to-end multi-scale patchmatch algorithm and adopts adaptive evaluation propagation, avoiding the substantial memory resource consumption characterizing traditional region matching algorithms. Extensive experiments on the DTU and Tanks and Temples datasets show that our algorithm is very competitive in completeness, speed and memory.

Physical Structure Expression for Dense Point Clouds of Magnetic Levitation Image Data.

The research and development of an intelligent magnetic levitation transportation system has become an important research branch of the current intelligent transportation system (ITS), which can provide technical support for state-of-the-art fields such as intelligent magnetic levitation digital twin. First, we applied unmanned aerial vehicle oblique photography technology to acquire the magnetic levitation track image data and preprocessed them. Then, we extracted the image features and matched them based on the incremental structure from motion (SFM) algorithm, recovered the camera pose parameters of the image data and the 3D scene structure information of key points, and optimized the bundle adjustment to output 3D magnetic levitation sparse point clouds. Then, we applied multiview stereo (MVS) vision technology to estimate the depth map and normal map information. Finally, we extracted the output of the dense point clouds that can precisely express the physical structure of the magnetic levitation track, such as turnout, turning, linear structures, etc. By comparing the dense point clouds model with the traditional building information model, experiments verified that the magnetic levitation image 3D reconstruction system based on the incremental SFM and MVS algorithm has strong robustness and accuracy and can express a variety of physical structures of magnetic levitation track with high accuracy.

Refinement of Inverse Depth Plane in Textureless and Occluded Regions in a Multiview Stereo Matching Scheme

In the multiview stereo (MVS) vision, it is difficult to estimate accurate depth in the textureless and occluded regions. To solve this problem, several MVS investigations employ the matching cost volume (MCV) approach to refine the cost in the textureless and occluded regions. Usually, the matching costs in the large textureless image regions are not reliable. In addition, if an occluded region is also textureless, the matching cost contains a significant error. The goal of the proposed MVS method is to reconstruct accurate depth maps in both large-textureless and occluded-textureless regions by iteratively updating the erroneous disparity. The erroneous disparity in the textureless region is updated by the 3D disparity plane of the region in the inverse-depth space. Then, the surface consensus is computed and used to run the two processes, the surface consensus refinement and the matching cost update. By the iterative update of the 3D inverse depth plane, surface consensus, and matching cost, the performance of the depth reconstruction in the large-textureless and occluded-textureless regions is greatly improved. The performance of the proposed method is analyzed using the Middlebury multiview stereo dataset. The depth reconstruction performance is also compared with several stereo vision methods.

Optimization Design of Camera Pose for Network Base on Multi-view Stereo Vision

When designing a multi-vision stereo vision network, the camera’s positional information has a critical impact on the measurement accuracy. In order to solve the problem of optimizing the camera pose during network design, in this paper, we split the multi-vision stereo vision system into binocular stereo vision system. Based on the mathematical model of binocular stereo vision system, the error function of the positional parameters is constructed. By simulation analyzing the function models the field of view angle, the angle between the optical axis and the baseline, the baseline length, the law of the influence of each parameter on the measurement accuracy and the optimal range of values are obtained. Then we compare the actual measurement results with the simulation analysis results. The results show that as the field of view angle, optical axis and baseline angle, baseline length increases, the measurement error first decreases and then increases, and the increase is gradually accelerated. When each parameter is located in the optimal range, the measurement error meets the precision engineering measurement accuracy requirement of 0.05mm, and the minimum error is reduced by 0.114mm, 0.120mm and 0.061mm compared to the maximum error. In the network design, the optimization of the pose parameters can effectively obtain better pose information of the camera and thus improve the accuracy of the measurement.

Three-Dimensional Morphology and Size Measurement of High-Temperature Metal Components Based on Machine Vision Technology: A Review.

The three-dimensional (3D) size and morphology of high-temperature metal components need to be measured in real time during manufacturing processes, such as forging and rolling. Since the surface temperature of a metal component is very high during the forming and manufacturing process, manually measuring the size of a metal component at a close distance is difficult; hence, a non-contact measurement technology is required to complete the measurement. Recently, machine vision technology has been developed, which is a non-contact measurement technology that only needs to capture multiple images of a measured object to obtain the 3D size and morphology information, and this technology can be used in some extreme conditions. Machine vision technology has been widely used in industrial, agricultural, military and other fields, especially fields involving various high-temperature metal components. This paper provides a comprehensive review of the application of machine vision technology in measuring the 3D size and morphology of high-temperature metal components. Furthermore, according to the principle and method of measuring equipment structures, this review highlights two aspects in detail: laser scanning measurement and multi-view stereo vision technology. Special attention is paid to each method through comparisons and analyses to provide essential technical references for subsequent researchers.

A novel objective wrinkle evaluation method for printed fabrics based on multi-view stereo algorithm

Traditionally, fabric wrinkle assessment is based on human eyes, which is subjective with the disadvantages of low efficiency and accuracy. In this paper, an objective fabric wrinkle evaluation method was proposed to solve the previous problems due to the subjective evaluation. A self-developed multiple image acquisition system was established for the image capturing of wrinkled fabrics from different direction. Three-dimensional (3D) surface profile of wrinkled fabric was reconstructed by an improved patch-based multi-view stereo vision algorithm. A two-dimensional depth image could be generated directly from the 3D point cloud model, after that, four texture feature parameters were extracted from the depth image using gray level co-occurrence matrix. Finally, these four feature parameters were selected as the input vector and wrinkle grade as the output to form a support vector machine for the objective assessment of fabric wrinkle appearance. Our experimental results indicated that the recognition accuracy of the proposed method and system was more than 90%. The originality of our research is that the wrinkle features based on 3D surface profile reconstructing can avoid interference caused by fabric color and texture and further improve the recognition accuracy of objective evaluation.

Objective evaluation of fabric pilling based on multi-view stereo vision

Fabric pilling evaluation is very important for the quality control of textile industry. Traditional image analysis based methods have the disadvantages of 2 D imaging and color sensitivity, this paper presents a new method based on stereo vision to solve the problem of 3 D imaging of fabric pilling. One set of self-developed mobile camera system is established to capture a group of images for the 3 D reconstruction of the fabric surface, the point cloud model of the fabric surface is generated by the self-developed stereo vision algorithm, including structure from motion (SFM) and patch-based multi-view stereo (PMVS) algorithm. One 2 D gray-scale image is obtained from the 3 D point cloud model by mapping to the 2 D image plane, which contains the depth information of fabric pilling. The segmentation of fabric pilling could be done by accurate positioning of edge detection, adaptive thresholding and morphological analysis. Four feature parameters including pilling number, pilling area, pilling density and coverage ratio are extracted for the determination of fabric pilling grade objectively. Experimental results show that the new developed system and method is effective and reliable for the fabric pilling evaluation, which is consistent with the subjective pilling evaluation. It is workable for the color printed or yarn dyed fabrics, the proposed imaging system could be a good solution for the digital intelligent quality control of textile products.

Flexible multicamera calibration method with a rotating calibration plate.

Camera calibration is necessary for accurate image measurements, particularly in multicamera systems. The calibration process involves corresponding the coordinates of 3D calibration points with a 2D image and requires the establishment of a reliable 3D world coordinate system. This paper presents a convenient multicamera calibration method that uses a rotating calibration plate and multi-view stereo vision to calculate 3D points and their relationship with the image coordinates. Despite simple implementation, the rotation of the calibration plate presents numerous calibration points from various planes, increasing the stability of the solution and the noise reduction. The relocation accuracy and reprojection error are experimentally verified.

Investigation of 3D surface profile reconstruction technology for automatic evaluation of fabric smoothness appearance

Objective evaluation of smoothness appearance of fabrics after repeated home laundering is not only the key point, but also a difficulty in the field of textile inspection and clothing industry. This paper reports a 3D reconstruction system and method for objective assessment of fabric smoothness appearance based on Multi-view Stereo Vision (MVS) and Bag-of-Features (BOF). An image acquisition system for texture-less fabric was designed with a specific illumination module. Three-dimensional surface profile of wrinkled fabric was reconstructed by Structure from Motion (SFM) and Patch-based Multi-view Stereo (PMVS) techniques. To facilitate feature extraction, the generated 3D point cloud was mapped to a 2D image plane. The Bag-of-Features (BOF) model was utilized to characterize the fabric wrinkles and the K-Nearest Neighbor (KNN) algorithm was used to achieve automatic evaluation of fabric smoothness appearance. The experimental results with a high recognition of 93.33% indicated the effectiveness and stability of the proposed method.

A comparative analysis between active structured light and multi-view stereo vision technique for 3D reconstruction of face model surface

In some application fields, such as reverse engineering or ancient cultural protection, to collect surface information from objects to generate 3D data is necessary. In this work, active structured light and multi-view stereo vision approach are compared for 3D face reconstruction. The comparison is performed by projecting coded structured light and taking multi-view images for face model. The final point cloud from each sensor is compared in terms of measurements in the scanned objects, to determine which sensor is best suited depending on the environment. For a fair comparison, a preliminary analysis of the performances of the two techniques is produced. Our results can be used as a reference for further comparisons in the analysis of other 3D techniques and algorithms.

Improved Sensors Based on Scheimpflug Conditions and Multi-Focal Constraints

Binocular vision sensor composed of two traditional camerasis widely utilized in many fields. However, there are still twodeficiencies, i.e. limited measurement area and weakconstraint for feature matching. In this paper, we firstintroduced measurement principle of the Scheimpflug camera and its calibrationmethod. Then an improved binocular vision sensor composed of two Scheimpflug cameras is constructed. Measurement area is maximized.Then, a trinocular vision sensor as an example of the multi-viewstereo vision sensor is analyzed and multi-focal constraint basedmeasurement model is given. Feature matching according to theconstraint of a trifocal tensor is strict, which is verified by relatedexperiments. Calibration procedures for the camera and improved visionsensor are detailed. Experiment results show that measurement area ofour improved vision sensor increases up to 86.2% (when tilted angleis 4°). Root mean square error of distance is0.107 0 / 00 with regard to the measurementarea.

Full-field displacement measurement of corneoscleral shells by combining multi-camera speckle interferometry with 3D shape reconstruction

Changes in the biomechanical properties of the connective tissue of the eye occur with age and underlie the development of several ocular diseases, such as glaucoma, myopia, and keratoconus. The biomechanical dynamics of ocular connective tissue are measured by ex vivo inflation testing, in which intraocular pressure (IOP) is varied and optical methods are used to produce maps of corneal and scleral displacement. Current optical methods are limited by acquisition rate, occlusions, poor spatial resolution, and insufficient 3D mapping. We developed an interferometric optical method integrates four-camera electronic speckle pattern interferometry (ESPI) and a novel three-dimensional (3D) shape reconstruction process to measure shape and full-field mechanical deformations of corneal and scleral shells during ex vivo inflation testing. Each camera provides accurate measurements of the laser beam phase related to deformations of the specimen surface; a multi-view stereovision method generates the shape of the specimen and a functional form that links every pixel of a given camera to 3D points on the specimen's visible surface. In this way, dynamic deformations of the specimen are localized, with quantification of the time-dependent 3D displacements of the specimen at nanometric accuracy. The ESPI-3D system is suitable for analyzing scleral deformation and morphological changes caused by time-varying IOP.

多目立体视觉测量网络节点的位姿优化方法

多目立体视觉测量网络节点的位姿优化方法

An automatic markerless registration method for neurosurgical robotics based on an optical camera.

Current markerless registration methods for neurosurgical robotics use the facial surface to match the robot space with the image space, and acquisition of the facial surface usually requires manual interaction and constrains the patient to a supine position. To overcome these drawbacks, we propose a registration method that is automatic and does not constrain patient position. An optical camera attached to the robot end effector captures images around the patient's head from multiple views. Then, high coverage of the head surface is reconstructed from the images through multi-view stereo vision. Since the acquired head surface point cloud contains color information, a specific mark that is manually drawn on the patient's head prior to the capture procedure can be extracted to automatically accomplish coarse registration rather than using facial anatomic landmarks. Then, fine registration is achieved by registering the high coverage of the head surface without relying solely on the facial region, thus eliminating patient position constraints. The head surface was acquired by the camera with a good repeatability accuracy. The average target registration error of 8 different patient positions measured with targets inside a head phantom was [Formula: see text], while the mean surface registration error was [Formula: see text]. The method proposed in this paper achieves automatic markerless registration in multiple patient positions and guarantees registration accuracy inside the head. This method provides a new approach for establishing the spatial relationship between the image space and the robot space.

Three-dimensional measurement of small inner surface profiles using feature-based 3-D panoramic registration.

Rapid development in the performance of sophisticated optical components, digital image sensors, and computer abilities along with decreasing costs has enabled three-dimensional (3-D) optical measurement to replace more traditional methods in manufacturing and quality control. The advantages of 3-D optical measurement, such as noncontact, high accuracy, rapid operation, and the ability for automation, are extremely valuable for inline manufacturing. However, most of the current optical approaches are eligible for exterior instead of internal surfaces of machined parts. A 3-D optical measurement approach is proposed based on machine vision for the 3-D profile measurement of tiny complex internal surfaces, such as internally threaded holes. To capture the full topographic extent (peak to valley) of threads, a side-view commercial rigid scope is used to collect images at known camera positions and orientations. A 3-D point cloud is generated with multiview stereo vision using linear motion of the test piece, which is repeated by a rotation to form additional point clouds. Registration of these point clouds into a complete reconstruction uses a proposed automated feature-based 3-D registration algorithm. The resulting 3-D reconstruction is compared with x-ray computed tomography to validate the feasibility of our proposed method for future robotically driven industrial 3-D inspection.