Unordered Images Research Articles

EC-SfM: Efficient Covisibility-Based Structure-From-Motion for Both Sequential and Unordered Images

Structure-from-Motion is a technology used to obtain scene structure through image collection, which is a fundamental problem in computer vision. For unordered Internet images, SfM is very slow due to the lack of prior knowledge about image overlap. For sequential images, knowing the large overlap between adjacent frames, SfM can adopt a variety of acceleration strategies, which are only applicable to sequential data. To further improve the reconstruction efficiency and break the gap of strategies between these two kinds of data, this paper presents an efficient covisibility-based incremental SfM. Different from previous methods, we exploit covisibility and registration dependency to describe the image connection which is suitable to any kind of data. Based on this general image connection, we propose a unified framework to efficiently reconstruct sequential images, unordered images, and the mixture of these two. Experiments on the unordered images and mixed data verify the effectiveness of the proposed method, which is three times faster than the state-of-the-art on feature matching, and an order of magnitude faster on reconstruction without sacrificing the accuracy. The source code is publicly available at https://github.com/openxrlab/xrsfm.

Image quality using synthetic brain MRI: an age-stratified study.

Synthetic magnetic resonance imaging (MRI) might replace the conventional MR sequences in brain evaluation to shorten scan time and obtain multiple quantitative parameters. To evaluate the image quality of multiple-delay-multiple-echo (MDME) sequence-derived synthetic brain MR images compared to conventional images by considering a multi-age sample. Image sets of conventional and synthetic MRI of 200 participants were included. On the basis of the presence of intracranial lesions, the participants were divided into a normal group and a pathological group. Two neuroradiologists compared the anonymous and unordered images. Image quality, artifacts, and diagnostic performance were analyzed. In the quantitative analysis, comparing with conventional images, MDME sequence-derived synthetic MRI demonstrated an equal/greater signal-to-noise ratio and contrast-to-noise ratio (CNR) in all age groups. Specifically, for participants aged ≤2 years, synthetic T2-fluid-attenuated inversion recovery imaging showed a significantly higher cerebellum gray/white matter CNR (P < 0.05). In the qualitative and artifact analyses, except for the superior sagittal sinus and cranial nerves, synthetic MRI showed good imaging quality (≥3 points) in all brain structures. On synthetic T1-weighted imaging, high signal intensity within the superior sagittal sinus was found in most of our participants (107/118, 90.7%). No difference was observed between synthetic and conventional MRI in diagnosing the lesions. MDME sequence-derived synthetic MRI showed similar image quality and diagnostic performance with a shorter acquisition time than conventional MRI. However, the high signal intensity within the superior sagittal sinus on synthetic T1-weighted images requires consideration.

Unordered Image Stitching Based on the Registration Index Table

In order to solve the problem of large amount of calculation and time-consuming in the process of unordered image stitching, we propose an image stitching algorithm based on the image registration index table in this paper. Firstly, we construct the matching index sequence based on the similarity of the matching area. Then we search for image matching pairs according to the matching index sequence, carry out image registration and stitch directly, and generate the registration index table. For image stitching, we build a binary tree model assisted by the image registration index table, we need to search for matching pairs and carry out registration and stitching according to the registration index table generated by each layer. Compare to the current research on disordered image stitching, our method significantly reduces the times of feature extraction and matching by at least half, therefore, it greatly improves the efficiency of image stitching.

Evaluating Feature Extraction Methods with Synthetic Noise Patterns for Image-Based Modelling of Texture-Less Objects

Image-based three-dimensional (3D) reconstruction is a process of extracting 3D information from an object or entire scene while using low-cost vision sensors. A structure-from-motion coupled with multi-view stereo (SFM-MVS) pipeline is a widely used technique that allows 3D reconstruction from a collection of unordered images. The SFM-MVS pipeline typically comprises different processing steps, including feature extraction and feature matching, which provide the basis for automatic 3D reconstruction. However, surfaces with poor visual texture (repetitive, monotone, etc.) challenge the feature extraction and matching stage and affect the quality of reconstruction. The projection of image patterns while using a video projector during the image acquisition process is a well-known technique that has been shown to be successful for such surfaces. In this study, we evaluate the performance of different feature extraction methods on texture-less surfaces with the application of synthetically generated noise patterns (images). Seven state-of-the-art feature extraction methods (HARRIS, Shi-Tomasi, MSER, SIFT, SURF, KAZE, and BRISK) are evaluated on problematic surfaces in two experimental phases. In the first phase, the 3D reconstruction of real and virtual planar surfaces evaluates image patterns while using all feature extraction methods, where the patterns with uniform histograms have the most suitable morphological features. The best performing pattern from Phase One is used in Phase Two experiments in order to recreate a polygonal model of a 3D printed object using all of the feature extraction methods. The KAZE algorithm achieved the lowest standard deviation and mean distance values of 0.0635 mm and −0.00921 mm, respectively.

An End-to-End Approach to Reconstructing 3D Model From Image Set

Large-scale 3D reconstruction from imagery has received much attention from the computer vision community. However, recovering 3D structures from 2D images is a notoriously complex process that requires expertise with often limited results. This paper presents an end-to-end 3D reconstruction system that can produce high-quality 3D models from a set of unordered image collections. Our workflow is a typical 3D reconstruction architecture that consists of structure from motion (SFM), multi-view stereo (MVS), and surface reconstruction, and can automatically recover desirable 3D models without any interactive operations. Finally, a comprehensive experiment is conducted on several benchmark datasets to assess the presented system. Experimental results show that the presented system achieves significant improvements in reconstruction accuracy and completeness over the existing state-of-the-art approach.

A Method of Hierarchical Image Retrieval for Real-Time Photogrammetry Based on Multiple Features

In near real-time photogrammetry, the first step in processing each new added image is determining the most relevant image in pre-sequence unordered images quickly and exactly, which is pivotal for accurate image matching and 3D reconstruction. This paper presents a hierarchical image retrieval algorithm based on multiple features and details the choice for representation of multiple features which is critical to the improvement of accuracy of this algorithm. First, we represent global features using AlexNet-FC7(fully connected layers) or ResNet101-Pool5(pooling layers) and local features using SIFT (scale-invariant feature transform) in two parallel threads with support of GPU (Graphics Processing Unit). Next, we obtain candidates based on cosine similarities between global features of each pre-sequence image and new added image. Finally, we determine the most relevant image from those candidates according to feature matching results for each candidate and new added image. The experimental results confirm that the second step is rather fast and the third step is necessary to tackle the problem that global features cannot distinguish objects from the same class. The total time our algorithm takes is about 83.6ms for determining the most relevant image in 5063 pre-sequence unordered images of size $1024\times 768$ , which outperforms exhaustive pairwise matching, Bag of Words and multi-vocabulary trees. Accuracy of our algorithm also perform better than the state-of-the-art methods on three benchmark datasets. SIFT matching results obtained in the third step after eliminating mismatches with RANSAC (Random Sample Consensus) can also be used for high-precision incremental SFM (Structure from Motion) reconstruction.

An Unordered Image Stitching Method Based on Binary Tree and Estimated Overlapping Area.

Aiming at the complex computation and time-consuming problem during unordered image stitching, we present a method based on the binary tree and the estimated overlapping areas to stitch images without order in this paper. For image registration, the overlapping areas between input images are estimated, so that the extraction and matching of feature points are only performed in these areas. For image stitching, we build a model of the binary tree to stitch each two matched images without sorting. Compared to traditional methods, our method significantly reduces the computational time of matching irrelevant image pairs and improves the efficiency of image registration and stitching. Moreover, the stitching model of the binary tree proposed in this paper further reduces the distortion of the panorama. Experimental results show that the number of extracted feature points in the estimated overlapping area is approximately 0.3∼0.6 times of that in the entire image by using the same method, which greatly reduces the computational time of feature extraction and matching. Compared to the exhaustive image matching method, our approach only takes about 1/3 of the time to find all matching images.

OPEN-SOURCE IMAGE-BASED 3D RECONSTRUCTION PIPELINES: REVIEW, COMPARISON AND EVALUATION

Abstract. State-of-the-art automated image orientation (Structure from Motion) and dense image matching (Multiple View Stereo) methods commonly used to produce 3D information from 2D images can generate 3D results – such as point cloud or meshes – of varying geometric and visual quality. Pipelines are generally robust and reliable enough, mostly capable to process even large sets of unordered images, yet the final results often lack completeness and accuracy, especially while dealing with real-world cases where objects are typically characterized by complex geometries and textureless surfaces and obstacles or occluded areas may also occur. In this study we investigate three of the available commonly used open-source solutions, namely COLMAP, OpenMVG+OpenMVS and AliceVision, evaluating their results under diverse large scale scenarios. Comparisons and critical evaluation on the image orientation and dense point cloud generation algorithms is performed with respect to the corresponding ground truth data. The presented FBK-3DOM datasets are available for research purposes.

Dependency-Aware Attention Control for Image Set-Based Face Recognition

This paper considers the problem of image set-based face verification and identification. Unlike traditional single sample (an image or a video) setting, this situation assumes the availability of a set of heterogeneous collection of orderless images and videos. The samples can be taken at different check points, different identity documents $etc$ . The importance of each image is usually considered either equal or based on a quality assessment of that image independent of other images and/or videos in that image set. How to model the relationship of orderless images within a set remains a challenge. We address this problem by formulating it as a Markov Decision Process (MDP) in a latent space. Specifically, we first propose a dependency-aware attention control (DAC) network, which uses actor-critic reinforcement learning for attention decision of each image to exploit the correlations among the unordered images. An off-policy experience replay is introduced to speed up the learning process. Moreover, the DAC is combined with a temporal model for videos using divide and conquer strategies. We also introduce a pose-guided representation (PGR) scheme that can further boost the performance at extreme poses. We propose a parameter-free PGR without the need for training as well as a novel metric learning-based PGR for pose alignment without the need for pose detection in testing stage. Extensive evaluations on IJB-A/B/C, YTF, Celebrity-1000 datasets demonstrate that our method outperforms many state-of-art approaches on the set-based as well as video-based face recognition databases.

A Secret Image Restoring Scheme Using Threshold Pairs of Unordered Image Shares

Secret image sharing has been widely used in numerous areas such as remote sensing and information security. The goal is to share and recover one or more images using several image shares. However, most schemes only consider ordered shares. In addition, the dependency between the images and the share is determined, that is, which image the share belongs to is determined. In this paper, by using the generalized Chinese remainder theorem method, we propose a secret image sharing scheme using threshold pairs of unordered image shares, where $t$ out of $n$ share pairs are needed to recover the original secret images, even if the affiliation between the shares and the images is disturbed. Compared with existing schemes, the proposed sharing algorithm is more secure and the recovery algorithm is more effective. Moreover, the conditions for the success or failure of recovering two images from their unordered shares are obtained. Simulations are also presented to show the efficiency of the proposed algorithms.

360-Degree Panoramic Image Stitching for Un-ordered Images Based on Harris Corner Detection

Objectives: This work aims to provide a system to arrange images automatically and produce the High-resolution panoramic images of a 360-degree wide range. Methods: This work consists of many stages which start by arranging the set of unordered input images. The Harris corner detector is used in the process of detecting key-points. Euclidean distance measure is used to compute the distance between points and determine matches. The stage of stitching consists of several steps which are detecting key-points, matching, estimating the shifting matrix between images by using Random Sample Consensus (RANSAC) algorithm, aligning, and blending. Findings: Experiments show that this method is able to arrange different sets of images, where samples taken are between 15–18 images. Depending on the threshold values in the verification process applied to the matched points, which range from (0.4 to 0.6), the number of extracted features and the degree of accuracy can be controlled. Also, the process of blending images based on the taking of a different percentage of the overlapping areas according to the proximity to the contact line gave a distinctive blending result. Application: Production of panoramic images are used in real estate photography, the creation of an attractive experience on viewing all location details, and lots of other uses. Keywords: Blending, Harris Corner Detector, Image Stitching, Panorama

Low-Cost and Efficient Indoor 3D Reconstruction through Annotated Hierarchical Structure-from-Motion

With the widespread application of location-based services, the appropriate representation of indoor spaces and efficient indoor 3D reconstruction have become essential tasks. Due to the complexity and closeness of indoor spaces, it is difficult to develop a versatile solution for large-scale indoor 3D scene reconstruction. In this paper, an annotated hierarchical Structure-from-Motion (SfM) method is proposed for low-cost and efficient indoor 3D reconstruction using unordered images collected with widely available smartphone or consumer-level cameras. Although the reconstruction of indoor models is often compromised by the indoor complexity, we make use of the availability of complex semantic objects to classify the scenes and construct a hierarchical scene tree to recover the indoor space. Starting with the semantic annotation of the images, images that share the same object were detected and classified utilizing visual words and the support vector machine (SVM) algorithm. The SfM method was then applied to hierarchically recover the atomic 3D point cloud model of each object, with the semantic information from the images attached. Finally, an improved random sample consensus (RANSAC) generalized Procrustes analysis (RGPA) method was employed to register and optimize the partial models into a complete indoor scene. The proposed approach incorporates image classification in the hierarchical SfM based indoor reconstruction task, which explores the semantic propagation from images to points. It also reduces the computational complexity of the traditional SfM by avoiding exhausting pair-wise image matching. The applicability and accuracy of the proposed method was verified on two different image datasets collected with smartphone and consumer cameras. The results demonstrate that the proposed method is able to efficiently and robustly produce semantically and geometrically correct indoor 3D point models.

DeepImSeq: Deep image sequencing for unsynchronized cameras

Consider a set of n images of a dynamic scene captured using multiple hand-held devices. The order in which these images are captured is unknown. For n images, there can be n! possible arrangements, which makes this problem extremely challenging. In this work, we address the problem of sequencing such a set of unordered images in its temporal order. We propose an LSTM-based deep neural network which addresses this problem in an end-to-end manner. The network takes the set of images as input and outputs their order of capture. We formulate the problem as a sequence-to-sequence mapping task, in which each image is mapped to its position in the ordered sequence. We do not provide any other information to the network apart from the input images. We show that the proposed approach obtains the state-of-the-art results on the standard dataset. Further, we show through experimental results that the network learns better when the target sequence is reversed.

Structure from motion for ordered and unordered image sets based on random k-d forests and global pose estimation

In this paper, we present a new fast and robust method for structure from motion (SfM) for data sets potentially comprising thousands of ordered or unordered images. Our work focuses on the two most time-consuming procedures: (a) image matching and (b) pose estimation. For image matching, a new method employing a random k-d forest is proposed to quickly obtain pairs of overlapping images from an unordered set. After that, image matching and the estimation of relative orientation parameters are performed only for pairs found to be very likely to overlap. For pose estimation, we use a two-stage global approach, separating the determination of rotation matrices and translation parameters; the latter are computed simultaneously using a new method. In order to cope with outliers in the relative orientations, which global approaches are particularly sensitive to, we present a new constraint based on triplet loop closure errors of rotation and translation. Finally, a robust bundle adjustment is carried out to refine the image orientation parameters.We demonstrate the potential and limitations of our pipeline using various real-world datasets including ordered image data acquired from UAV (unmanned aerial vehicle) and other platforms as well as unordered data from the internet. The experiments show that our work performs better than comparable state-of-the-art SfM systems in terms of run time, while we achieve a similar accuracy and robustness.

Fine-Tuning CNN Image Retrieval with No Human Annotation.

Image descriptors based on activations of Convolutional Neural Networks (CNNs) have become dominant in image retrieval due to their discriminative power, compactness of representation, and search efficiency. Training of CNNs, either from scratch or fine-tuning, requires a large amount of annotated data, where a high quality of annotation is often crucial. In this work, we propose to fine-tune CNNs for image retrieval on a large collection of unordered images in a fully automated manner. Reconstructed 3D models obtained by the state-of-the-art retrieval and structure-from-motion methods guide the selection of the training data. We show that both hard-positive and hard-negative examples, selected by exploiting the geometry and the camera positions available from the 3D models, enhance the performance of particular-object retrieval. CNN descriptor whitening discriminatively learned from the same training data outperforms commonly used PCA whitening. We propose a novel trainable Generalized-Mean (GeM) pooling layer that generalizes max and average pooling and show that it boosts retrieval performance. Applying the proposed method to the VGG network achieves state-of-the-art performance on the standard benchmarks: Oxford Buildings, Paris, and Holidays datasets.

ROBUST IMAGE ORIENTATION BASED ON RELATIVE ROTATIONS AND TIE POINTS

Abstract. In this paper we present a novel approach for image orientation by combining relative rotations and tie points. First, we choose an initial image pair with enough correspondences and large triangulation angle, and we then iteratively add clusters of new images. The rotation of these newly added images is estimated from relative rotations by single rotation averaging. In the next step, a linear equation system is set up for each new image to solve the translation parameters with triangulated tie points which can be viewed in that new image, followed by a resection for refinement. Finally, we optimize the cluster of reconstructed images by local bundle adjustment. We show results of our approach on different benchmark datasets. Furthermore, we orient several larger datasets incl. unordered image datasets to demonstrate the robustness and performance of our approach.

Integrating geometric models, site images and GIS based on Google Earth and Keyhole Markup Language

Abstract Technologies for information management and visualization are instrumental in enhancing human perceptions and interpretations of complicated project information. 3D/4D modeling, Virtual Reality (VR), Building Information Models (BIM) and Geographical Information Systems (GIS) have been increasingly used for data management and analytics in construction. Apart from virtual models, it is essential to represent the ever-changing site reality by integrating images captured with drones, mobile devices, and digital cameras. To improve the cognitive perception of the site environment from fragmented datasets, this paper proposes a framework to integrate unordered images, geometric models and surrounding environment in Google Earth using Keyhole Markup Language (KML). A ground-control-free methodology to geo-reference sequential aerial imageries and ground imageries is proposed in order to place unordered images into the physical coordinate system of Google Earth. To combine geometric models, site images and panoramic images with the site surrounding environment in 3D GIS, a KML and cloud storage based data management system is conceptualized to handle large scale datasets. The research provides construction engineers with a low-cost and low-technology-barrier solution to represent a dynamic construction site through information management, integration and visualization.

Optimization of incremental structure from motion combining a random k-d forest and pHash for unordered images in a complex scene

On the basis of today’s popular virtual reality and scientific visualization, three-dimensional (3-D) reconstruction is widely used in disaster relief, virtual shopping, reconstruction of cultural relics, etc. In the traditional incremental structure from motion (incremental SFM) method, the time cost of the matching is one of the main factors restricting the popularization of this method. To make the whole matching process more efficient, we propose a preprocessing method before the matching process: (1) we first construct a random k-d forest with the large-scale scale-invariant feature transform features in the images and combine this with the pHash method to obtain a value of relatedness, (2) we then construct a connected weighted graph based on the relatedness value, and (3) we finally obtain a planned sequence of adding images according to the principle of the minimum spanning tree. On this basis, we attempt to thin the minimum spanning tree to reduce the number of matchings and ensure that the images are well distributed. The experimental results show a great reduction in the number of matchings with enough object points, with only a small influence on the inner stability, which proves that this method can quickly and reliably improve the efficiency of the SFM method with unordered multiview images in complex scenes.

On Structure-From-Motion Application Challenges

Numerous software solutions that implement the Structure-from-Motion/Multi-View Stereo (SfM/MVS) 3D reconstruction approach have been made available during the last decade. These allow the production of high quality in terms of geometry and colour information 3D models with the use of unordered image collections that depict a static scene or object from arbitrary viewpoints. Nowadays SfM/MVS-based 3D reconstruction approaches constitute a popular solution in a variety of applications within the cultural heritage domain. As with all 3D reconstruction approaches it has its limitations and application challenges. In this work, the authors attempt to compose a set of guidelines that are based on the important outcomes of important published works that propose solutions that overcome the various challenges introduced by non-friendly surface types.

Application Research of SfM in Consolidation Grouting Material Performance

The main purpose of this paper is to develop a new evaluation method based on structure from motion (SfM) for consolidation grouting material performance. An artificial sand object, which have been injected by consolidation grouting materials with different proportion, was built in an experimental room. Based on SfM technique, the model of the artificial sand object could be built with only some unordered images, correspondingly, the volume could be calculated and compared in order to evaluate the performance of the consolidation grouting material with different proportion.