Multi-view Matching Research Articles

Digital image correlation calculation method for RGB-D camera multi-view matching using variable template

Multi-view matching is an indispensable method for digital image correlation 3D reconstruction. However, as the camera viewing angle changes, traditional DIC methods can lead to the accumulation of matching errors and an increase in mismatches, thereby affecting the 3D reconstruction quality. This paper proposes a digital image correlation method for multi-view RGB-D images based on variable template matching. Initial positioning is achieved by dynamically adjusting the size and sampling direction of the variable template. Sub-pixel registration is then performed using the inverse compositional algorithm, followed by multi-view point cloud fusion using the Iterative Closest Point (ICP) registration method. Experimental results show that the iteration speed of point cloud registration is increased by 30%, and the cumulative error is reduced by 62.5%. These improvements demonstrate that the proposed method is suitable for multi-viewpoint matching scenarios, achieving excellent results in multi-view feature point matching and multi-view point cloud fusion, significantly enhancing 3D reconstruction accuracy and efficiency.

3D reconstruction method for tree seedlings based on point cloud self-registration

The 3D reconstruction of tree seedlings can help to assess phenotypic architectures, conceive virtual urban landscapes and design computer games. The existing multicamera photograph technology already has the capability to accurately reconstruct 3D models for small scene plants, such as corn and vegetable seedlings. However, the existing plant 3D reconstruction system has several shortcomings, such as its high cost, complicated operation procedure, and unsuitability for seedling trees. Therefore, this paper proposes an autonomous alignment method for seedling point clouds that can realize the low-cost and fast 3D reconstruction of batch seedlings. In this study, we designed a system based on a low-cost Kinect camera and a precision turntable to construct 3D seedling models. A special turntable was adopted to achieve self-registration for the seedling point clouds. It was efficient for us to obtain several 3D seedlings models with only one registration. The system could capture images automatically from different viewpoints and submit these images to a graphic workstation for processing. In our work, we set three fixed views, V2, V3 and V4, to evaluate the cumulative errors caused by multiview matching. It needn’t touch any parts of the seedings to create 3D models at different view by the proposed method. Herein, the large proportions of 0 < mean absolute distance, MD ≤ 0.6 cm and 0 < standard deviation, SD ≤ 0.4 cm, between the reference and the reconstructed point cloud showed that the 3D reconstruction method was accurate, stable and flexible. Additionally, we validated the phenotypic structure measurement, and the height H was highly accurate (R2 > 0.985) when using the 3D reconstruction models of seedlings. Experiments demonstrate that the proposed method has the potential to obtain high-precision 3D reconstruction models and phenotypic parameters for seedlings via low-cost equipment with high-efficiency processing algorithms.

Enhanced Soft 3D Reconstruction Method with an Iterative Matching Cost Update Using Object Surface Consensus.

In this paper, we propose a multi-view stereo matching method, EnSoft3D (Enhanced Soft 3D Reconstruction) to obtain dense and high-quality depth images. Multi-view stereo is one of the high-interest research areas and has wide applications. Motivated by the Soft3D reconstruction method, we introduce a new multi-view stereo matching scheme. The original Soft3D method is introduced for novel view synthesis, while occlusion-aware depth is also reconstructed by integrating the matching costs of the Plane Sweep Stereo (PSS) and soft visibility volumes. However, the Soft3D method has an inherent limitation because the erroneous PSS matching costs are not updated. To overcome this limitation, the proposed scheme introduces an update process of the PSS matching costs. From the object surface consensus volume, an inverse consensus kernel is derived, and the PSS matching costs are iteratively updated using the kernel. The proposed EnSoft3D method reconstructs a highly accurate 3D depth image because both the multi-view matching cost and soft visibility are updated simultaneously. The performance of the proposed method is evaluated by using structured and unstructured benchmark datasets. Disparity error is measured to verify 3D reconstruction accuracy, and both PSNR and SSIM are measured to verify the simultaneous enhancement of view synthesis.

A discovery about the positional distribution pattern among candidate homologous pixels and its potential application in aerial multi-view image matching

In aerial multi-view photogrammetry, whether there is a special positional distribution pattern among candidate homologous pixels of a matching pixel in the multi-view images? If so, can this positional pattern be used to precisely confirm the real homologous pixels? These problems have not been studied at present. Therefore, the study of the positional distribution pattern among candidate homologous pixels based on the adjustment theory in surveying is investigated in this paper. Firstly, the definition and computing method of pixel’s pseudo object-space coordinates are given, which can transform the problem of multi-view matching for confirming real homologous pixels into the problem of surveying adjustment for computing the pseudo object-space coordinates of the matching pixel. Secondly, according to the surveying adjustment theory, the standardized residual of each candidate homologous pixel of the matching pixel is figured out, and the positional distribution pattern among these candidate pixels is theoretically inferred utilizing the quantitative index of standardized residual. Lastly, actual aerial images acquired by different sensors are used to carry out experimental verification of the theoretical inference. Experimental results prove not only that there is a specific positional distribution pattern among candidate homologous pixels, but also that this positional distribution pattern can be used to develop a new object-side multi-view image matching method. The proposed study has an important reference value on resolving the defects of existing image-side multi-view matching methods at the mechanism level.

Viewpoint adaptation learning with cross-view distance metric for robust vehicle re-identification

Many vehicle re-identification (Re-ID) problems require the robust recognition of vehicle instances across multiple viewpoints. Existing approaches for dealing with the vehicle re-ID problem are insufficiently robust because they cannot distinguish among vehicles of the same type nor recognize high-level representations in deep networks for identical vehicles with various views. To address these issues, this paper proposes a viewpoint adaptation network (VANet) with a cross-view distance metric for robust vehicle Re-ID. This method consists of two modules. The first module is the VANet with cross-view label smoothing regularization (CVLSR), which abstracts different levels of a vehicle’s visual patterns and subsequently integrates multi-level features. In particular, CVLSR based on color domains assigns a virtual label to the generated data to smooth image-image translation noise. Accordingly, this module supplies the viewing angle information of the training data and provides strong robust capability for vehicles across different viewpoints. The second module is the cross-view distance metric, which designs a cascaded cross-view matching approach to combine the original features with the generated ones, and thus, obtain additional supplementary viewpoint information for the multi-view matching of vehicles. Results of extensive experiments on two large scale vehicle Re-ID datasets, namely, VeRi-776 and VehiclelD demonstrate that the performance of the proposed method is robust and superior to other state-of-the-art Re-ID methods across multiple viewpoints.

Automatic 3D building reconstruction from multi-view aerial images with deep learning

The study presented in this paper introduced a new fully automatic three-dimensional building reconstruction method that can generate first level of detail (LoD 1) building models from multi-view aerial images without any assistance from other data. The accuracy and completeness of our reconstructed models have approached that of manually delineated models to a large extent. The presented method consists of three parts: (1) efficient dense matching and Earth surface reconstruction, (2) reliable building footprint extraction and polygon regularization, and (3) highly accurate height inference of building roofs and bases. First, our novel deep learning-based multi-view matching method, composed of a convolutional neural network, gated recurrent convolutions, and a multi-scale pyramid matching structure, is used to reconstruct the digital surface model (DSM) and digital orthophoto map (DOM) efficiently without generating epipolarly rectified images. Second, our three-stage 2D building extraction method is introduced to deliver reliable and accurate building contours. Deep-learning based segmentation, assisted with DSM, is used to segment buildings from backgrounds; and the generated building maps are fused with a terrain classification algorithm to reach better segmentation results. A polygon regularization algorithm and a level set algorithm are thereafter employed to transfer the binary segmentation maps to structured vector-form building polygons. Third, a novel method is introduced to infer the height of building roofs and bases using adaptive local terrain filtering and neighborhood buffer analysis. We tested our method on a large experimental area that covered 2284 aerial images and 782 various types of buildings. Our results as far as correctness and completeness exceeded the results of other similar methods in a between-method comparison by at least 15% for individual 3D building models with many of them comparable to manual delineation results.

PHOTOMATCH: AN OPEN-SOURCE MULTI-VIEW AND MULTI-MODAL FEATURE MATCHING TOOL FOR PHOTOGRAMMETRIC APPLICATIONS

Abstract. Automatic feature matching is a crucial step in Structure-from-Motion (SfM) applications for 3D reconstruction purposes. From an historical perspective we can say now that SIFT was the enabling technology that made SfM a successful and fully automated pipeline. SIFT was the ancestor of a wealth of detector/descriptor methods that are now available. Various research activities have tried to benchmark detector/descriptors operators, but a clear outcome is difficult to be drawn. This paper presents an ISPRS Scientific Initiative aimed at providing the community with an educational open-source tool (called PhotoMatch) for tie point extractions and image matching. Several enhancement and decolorization methods can be initially applied to an image dataset in order to improve the successive feature extraction steps. Then different detector/descriptor combinations are possible, coupled with different matching strategies and quality control metrics. Examples and results show the implemented functionality of PhotoMatch which has also a tutorial for shortly explaining the implemented methods.

A Texture Selection Approach for Cultural Artifact 3D Reconstruction Considering Both Geometry and Radiation Quality

3D reconstruction of culture artifacts has great potential in digital heritage documentation and protection. Choosing the proper images for texture mapping from multi-view images is a major challenge for high precision and high quality 3D reconstruction of culture artifacts. In this study, a texture selection approach, considering both the geometry and radiation quality for 3D reconstruction of cultural artifacts while using multi-view dense matching is proposed. First, a Markov random field (MRF) method is presented to select images from the best angle of view among texture image sets. Then, an image radiation quality evaluation model is proposed in the virtue of a multiscale Tenengrad definition and brightness detection to eliminate fuzzy and overexposed textures. Finally, the selected textures are mapped to the 3D model under the mapping parameters of the multi-view dense matching and a semi-automatic texture mapping is executed on the 3DMax MudBox platform. Experimental results with two typical cultural artifacts data sets (bronze wares and porcelain) show that the proposed method can reduce abnormal exposure or fuzzy images to yield high quality 3D model of cultural artifacts.

Social Link Inference via Multiview Matching Network From Spatiotemporal Trajectories.

In this article, we investigate the problem of social link inference in a target location-aware social network (LSN), which aims at predicting the unobserved links between users within the network. This problem is critical for downstream applications, including network completion and friend recommendation. In addition to the network structures commonly used in general link prediction, the studies tailored for social link inference in an LSN leverage user trajectories from the spatial aspect. However, the temporal factor lying in user trajectories is largely overlooked by most of the prior studies, limiting the capabilities of capturing the temporal relevance between users. Moreover, effective user matching by fusing different views, i.e., social, spatial, and temporal factors, remains unresolved, which hinders the potential improvement of link inference. To this end, this article devises a novel multiview matching network (MVMN) by regarding each of the three factors as one view of any target user pair. MVMN enjoys the flexibility and completeness of modeling each factor by developing its suitable matching module: 1) location matching module; 2) time-series matching module; and 3) relation matching module. Each module learns a view-specific representation for matching, and MVMN fuses them for final link inference. Extensive experiments on two real-world data sets demonstrate the superiority of our approach against several competitive baselines for link prediction and sequence matching, validating the contribution of its key components.

A spatio-temporal multi-scale binary descriptor.

Binary descriptors are widely used for multi-view matching and robotic navigation. However, their matching performance decreases considerably under severe scale and viewpoint changes in non-planar scenes. To overcome this problem, we propose to encode the varying appearance of selected 3D scene points tracked by a moving camera with compact spatio-temporal descriptors. To this end, we first track interest points and capture their temporal variations at multiple scales. Then, we validate feature tracks through 3D reconstruction and compress the temporal sequence of descriptors by encoding the most frequent and stable binary values. Finally, we determine multiscale correspondences across views with a matching strategy that handles severe scale differences. The proposed spatio-temporal multi-scale approach is generic and can be used with a variety of binary descriptors. We show the effectiveness of the joint multiscale extraction and temporal reduction through comparisons of different temporal reduction strategies and the application to several binary descriptors.

基于光场结构特性与多视点匹配的深度估计

基于光场结构特性与多视点匹配的深度估计

Robust Precise Dynamic Point Reconstruction From Multi-View

Reconstructing precise dynamic points with multiple camera systems (MCSs) is a pivotal work in many computer vision applications, such as motion capture. However, the deviation of 2-D position leads to frequent mismatch when searching for correspondence from multi-view. This paper puts forward a two-stage framework based on passive optical motion capture system to reconstruct precise dynamic points with MCSs. Our proposed method improves the performance of calibration and matching simultaneously. In the calibration stage, the extrinsic parameters of numerous cameras are calibrated synchronously via an L-shaped frame, where the position of four reference points is optimized with multiple geometric constraints. Bundle adjustment occurs after calibration. In the reconstruction stage, we propose a novel sparse multi-view matching method called cyclical voting, which includes multiple pairs of global voting and in-group voting. Point residual method is proposed to exclude outliers in matching groups further. The experiments show that our proposed method can decrease mismatching significantly and achieve commendable reconstruction results compared with Cortex (one of the most successful commercial motion analysis software).

An enhanced multi-view vertical line locus matching algorithm of object space ground primitives based on positioning consistency for aerial and space images

The traditional multi-view vertical line locus (TMVLL) matching method is an object-space-based method that is commonly used to directly acquire spatial 3D coordinates of ground objects in photogrammetry. However, the TMVLL method can only obtain one elevation and lacks an accurate means of validating the matching results. In this paper, we propose an enhanced multi-view vertical line locus (EMVLL) matching algorithm based on positioning consistency for aerial or space images. The algorithm involves three components: confirming candidate pixels of the ground primitive in the base image, multi-view image matching based on the object space constraints for all candidate pixels, and validating the consistency of the object space coordinates with the multi-view matching result. The proposed algorithm was tested using actual aerial images and space images. Experimental results show that the EMVLL method successfully solves the problems associated with the TMVLL method, and has greater reliability, accuracy and computing efficiency.

基于物方定位一致性约束的光学航空影像多视铅垂线轨迹匹配

基于物方定位一致性约束的光学航空影像多视铅垂线轨迹匹配

Development of dense models of the Earth’s surface based on the semiglobal matching method for the case of multiple overlapping images

A method is developed for constructing a three-dimensional digital surface model based on the use of aerial images with multiple overlapping. The specific feature of this study is the use of the multiview matching method instead of stereo matching. The method is based on adapting the energy aggregation algorithm, which was proposed in the semiglobal matching (SGM) method, to the object space, as well as using the one-to-many scheme of cost calculation. The reconstructed scene is represented as a voxel grid. A high-performance implementation of the digital surface model construction at all stages is proposed based on the massive parallelization of computations on a graphics processing unit.

Multiview Matching Algorithm for Processing Mobile Sequence Images

Multiview Matching Algorithm for Processing Mobile Sequence Images

GIFT: Towards Scalable 3D Shape Retrieval

Projective analysis is an important solution in three-dimensional (3D) shape retrieval, since human visual perceptions of 3D shapes rely on various 2D observations from different viewpoints. Although multiple informative and discriminative views are utilized, most projection-based retrieval systems suffer from heavy computational cost, and thus cannot satisfy the basic requirement of scalability for search engines. In the past three years, shape retrieval contest (SHREC) pays much attention to the scalability of 3D shape retrieval algorithms, and organizes several large scale tracks accordingly [1] – [3] . However, the experimental results indicate that conventional algorithms cannot be directly applied to large datasets. In this paper, we present a real-time 3D shape search engine based on the projective images of 3D shapes. The real-time property of our search engine results from the following aspects: 1) efficient projection and view feature extraction using GPU acceleration; 2) the first inverted file, called F-IF, is utilized to speed up the procedure of multiview matching; and 3) the second inverted file, which captures a local distribution of 3D shapes in the feature manifold, is adopted for efficient context-based reranking. As a result, for each query the retrieval task can be finished within one second despite the necessary cost of IO overhead. We name the proposed 3D shape search engine, which combines G PU acceleration and i nverted f ile ( t wice), as GIFT . Besides its high efficiency, GIFT also outperforms state-of-the-art methods significantly in retrieval accuracy on various shape benchmarks (ModelNet40 dataset, ModelNet10 dataset, PSB dataset, McGill dataset) and competitions (SHREC14LSGTB, ShapeNet Core55, WM-SHREC07).

Clustering and selecting vantage images in a low-cost system for 3D reconstruction of texture-less objects

Abstract Structure from Motion is one of the image-based methods for 3D reconstruction. This technique coupled with Dense Multi-View Matching can provide an accurate and dense point cloud from texture-full objects. Although the procedure in these techniques is now fully automatic, capturing images of the object in suitable position is difficult, especially when the object is texture-less and needs projectors to provide a pattern on the object. Even though the images can be captured, the processing time in Dense Matching software is directly related to the number of images. A package including hardware and software is presented in this paper for solving these issues. Regarding the hardware, a low-cost and portable system for 3D modeling of texture-less objects is proposed. This system includes a rotation table designed and developed by using a stepper motor and a light circular plate. The system also has eight laser light sources with very dense and strong beams which provide a relatively appropriate pattern on texture-less objects. Regarding the software, Imaging Network Designer (IND), recently developed by the first author, was adjusted to be exploited by the proposed system. IND can both design a complete imaging network and cluster and select vantages images in a dense imaging network. In this system, regarding to the step of stepper motor, images are semi automatically taken by a digital camera. The images were imported in Structure From Motion (SFM) procedures implemented in Agisoft Photoscan to align images and provide a rough model. IND then used this information, to cluster and select vantage images for dense image matching. The comparison between normal procedure and using IND in the proposed system showed the importance of IND in reducing the elapsed time by 30 percentages. To evaluate the accuracy of the proposed system, two dark objects were considered and the point clouds of these objects were obtained by both the proposed system and a structured light system called GOM ATOS Compact Scan 2M laser scanner with an accuracy of 0.02 mm. The comparison between point clouds in three different tests showed the ability of the proposed system in delivering a 3D model for the texture-less objects with an accuracy of 0.3 mm.

Colorful 3-D Imaging Using an Infrared Dammann Grating

3-D profilometry is one of the most important branches of measurement science and technology. In this paper, a new colorful 3-D imaging method using near-infrared (NIR) laser (850 nm) dot array generated by a Dammann grating is proposed. Two NIR cameras mounted with narrowband passed filters are adopted to capture laser patterns on objects. A color camera mounted with an infrared cutoff filter is adopted to capture color information. A simplified multiviews’ matching method for dot array, which is the key of this 3-D imaging system, is proposed. A new geometric constraint named ray constraint is applied to multiview matching with the help of precalibration of laser dot array. Point clouds with color information are obtained according to triangulation. We used the experimental apparatus to scan a colorful plane with colorful attachments on it, and its colorful point clouds (about 56 000 sample points) were obtained. Experiments demonstrate that this apparatus achieves a depth accuracy of 0.21 mm at the measuring distance of 300 mm. This new colorful 3-D imaging has advantages of low cost, low power consumption, flexible structure, and portability, which should be highly interesting for practical applications.

HIGH DENSITY AERIAL IMAGE MATCHING: STATE-OF-THE-ART AND FUTURE PROSPECTS

Ongoing innovations in matching algorithms are continuously improving the quality of geometric surface representations generated automatically from aerial images. This development motivated the launch of the joint ISPRS/EuroSDR project “Benchmark on High Density Aerial Image Matching”, which aims on the evaluation of photogrammetric 3D data capture in view of the current developments in dense multi-view stereo-image matching. Originally, the test aimed on image based DSM computation from conventional aerial image flights for different landuse and image block configurations. The second phase then put an additional focus on high quality, high resolution 3D geometric data capture in complex urban areas. This includes both the extension of the test scenario to oblique aerial image flights as well as the generation of filtered point clouds as additional output of the respective multi-view reconstruction. The paper uses the preliminary outcomes of the benchmark to demonstrate the state-of-the-art in airborne image matching with a special focus of high quality geometric data capture in urban scenarios.