SfM Algorithm Research Articles

3D RECONSTRUCTION WITH A COLLABORATIVE APPROACH BASED ON SMARTPHONES AND A CLOUD-BASED SERVER

Abstract. The paper presents a collaborative image-based 3D reconstruction pipeline to perform image acquisition with a smartphone and geometric 3D reconstruction on a server during concurrent or disjoint acquisition sessions. Images are selected from the video feed of the smartphone’s camera based on their quality and novelty. The smartphone’s app provides on-the-fly reconstruction feedback to users co-involved in the acquisitions. The server is composed of an incremental SfM algorithm that processes the received images by seamlessly merging them into a single sparse point cloud using bundle adjustment. Dense image matching algorithm can be lunched to derive denser point clouds. The reconstruction details, experiments and performance evaluation are presented and discussed.

Indoor Manhattan spatial layout recovery from monocular videos via line matching

We present an end-to-end system for structure and motion computation in a Manhattan layout from monocular videos. Unlike most SFM algorithms that rely on point feature matching, only line matches are considered in this work. This may be convenient in indoor environment characterized by extended textureless walls, where point features may be scarce. Our system relies on the notion of “characteristic lines”, which are invariants of two views of the same parallel line pairs on a surface of known orientation. Experiments with indoor video sequences demonstrate the robustness of the proposed system.

INVESTIGATIONS ON THE POSSIBILITIES OF MONITORING COASTAL CHANGES INCLUDING SHALLOW UNDER WATER AREAS WITH UAS PHOTO BATHMETRY

UAS become a very valuable tool for coastal morphology. Not only for mapping but also for change detection and a better understanding of processes along and across the shore. This contribution investigates the possibilities of UAS to determine the water depth in clear shallow waters by means of the so called "photo bathymetry". From the results of several test flights it became clear that three factors influence the ability and the accuracy of bathymetric sea floor measurements. Firstly, weather conditions. Sunny weather is not always good. Due to the high image resolution the sunlight gets focussed even in very small waves causing moving patterns on shallow grounds with high reflection properties, such as sand. This effect invisible under overcast weather conditions. Waves, may also introduce problems and mismatches. Secondly the quality and the accuracy of the georeferencing with SFM algorithms. As multi image key point matching will not work over water, the proposed approach will only work for projects closely to the coastline with enough control on the land. Thirdly the software used and the intensity of post processing and filtering. Refraction correction and the final interpolation of the point cloud into a DTM are the last steps. If everything is done appropriately, accuracies in the bathymetry in the range of 10 – 50 cm, depending on the water depth are possible.

INVESTIGATIONS ON THE POSSIBILITIES OF MONITORING COASTAL CHANGES INCLUDING SHALLOW UNDER WATER AREAS WITH UAS PHOTO BATHMETRY

Abstract. UAS become a very valuable tool for coastal morphology. Not only for mapping but also for change detection and a better understanding of processes along and across the shore. This contribution investigates the possibilities of UAS to determine the water depth in clear shallow waters by means of the so called "photo bathymetry". From the results of several test flights it became clear that three factors influence the ability and the accuracy of bathymetric sea floor measurements. Firstly, weather conditions. Sunny weather is not always good. Due to the high image resolution the sunlight gets focussed even in very small waves causing moving patterns on shallow grounds with high reflection properties, such as sand. This effect invisible under overcast weather conditions. Waves, may also introduce problems and mismatches. Secondly the quality and the accuracy of the georeferencing with SFM algorithms. As multi image key point matching will not work over water, the proposed approach will only work for projects closely to the coastline with enough control on the land. Thirdly the software used and the intensity of post processing and filtering. Refraction correction and the final interpolation of the point cloud into a DTM are the last steps. If everything is done appropriately, accuracies in the bathymetry in the range of 10 – 50 cm, depending on the water depth are possible.

무인항공사진측량 방법에 의한 수치지도 갱신

Currently, Korea’s digital maps are being produced through traditional aerial photogrammetry methods. Aerial photogrammetry is the most economical way to produce a map of a wide area. However, timely survey is not allowed depends on weather condition and it is inefficient for small area surveying in economic point of view. Therefore, it costs too much and needs long time to produce a map for various small areas where are terrestrial changes for updating the map. In contrast, UAV photogrammetry is possible to work even in cloudy weather because of shooting at low altitude below the clouds. It also has excellent mobility and shoot quickly and well suited for small-scale mapping in several places by low cost. In this study, we produced an ortho-photo and digital map with the UAV photogrammetry method using SIFT and SfM algorithm and verified its accuracy to evaluate the applicability for future digital map updates. The accuracy was verified by comparing the results of the ground survey for check points selected on the digital map. Test results show small errors at ±2.6cm in X coordinates, ±2.8cm in Y coordinates and ±5.8cm in height and we could find a possibility that UAV photogrammetry would be fully applicable for digital map updating.

Crowdsourcing Lost Cultural Heritage

Abstract. Cultural Heritage all over the world is at high risk. Natural and human activities endanger the current state of monuments and sites, whereas many of them have already been destroyed especially during the last years. Preventive actions are of utmost importance for the protection of human memory and the prevention of irreplaceable. These actions may be carried out either in situ or virtually. Very often in situ preventive, or protective or restoration actions are difficult or even impossible, as e.g. in cases of earthquakes, fires or war activity. Digital preservation of cultural heritage is a challenging task within photogrammetry and computer vision communities, as efforts are taken to collect digital data, especially of the monuments that are at high risk. Visit to the field and data acquisition is not always feasible. To overcome the missing data problem, crowdsourced imagery is used to create a visual representation of lost cultural heritage objects. Such digital representations may be 2D or 3D and definitely help preserve the memory and history of the lost heritage. Sometimes they also assist studies for their reconstruction. An initiative to collect imagery data from the public and create a visual 3D representation of a recently destroyed stone bridge almost 150 years old is being discussed in this study. To this end, a crowdsourcing platform has been designed and the first images collected have been processed with the use of SfM algorithms.

TOWARDS AN EASIER ORIENTATION FOR SPHERICAL PHOTOGRAMMETRY

Abstract. For architectural metric documentation, Spherical Photogrammetry (SP) has demonstrated its validity and efficiency in many projects already. The speed of surveying is high, the accuracy and completeness of the plotting are satisfactory. However, there are still many problems to be solved. The weakest point is the orientation procedure, which is rather difficult to perform, in the sense that only very experienced people can run it, and few people only make use of it. The old orientation steps are 1) model formation (limited to binocular panoramas couples); 2) link of all the models in a block adjustment with independent model triangulation; 3) block bundle adjustment with 4 parameters/pano (3 coord.+1 orientation bearing); 4) block bundle adjustment with 6 parameters/pano, say the previous 4 + 2 correction angles around the horizontal axes. The panoramas must be spherical and quasi-horizontal. In order to make easier the orientation, enabling more people to use SP, an improved approach has been set up. It consists in the combination of any possible model formed either by three and two panoramas. The trinocular vision, say the combination of three different panoramas to form a unique model, has the advantage to be much more robust in comparison to binocular vision in the sense that the trinocular model is likely to be more error-free than any of the three composing binocular models. It contains less model deformation, the model coordinates are validated by the mutual comparison of the three intersecting binocular models. In addition, the number of possible trinocular models is normally much larger than the one of binocular models. The steps for a semi-automatic orientation of a block of panoramas proceed as follows: - Form any possible trinocular models by combination of the panoramas; - in case that no trinocular model has been formed, form any possible binocular model; - run a block adjustment with the algorithm of independent model, to link together the models in a unique reference system; - run a block bundle adjustment with collinearity equations with 4 unknown parameters per panorama; - run a block bundle adjustment with collinearity equations with 6 unknown parameters per panorama. The control requirements are limited to one fixed control point and two perpendicular constraints. Given a project with n panoramas, the trinocular possible models are the combination of n, 3 by 3, say (n*(n-1)*(n-2))/3!. As an example, in a project with 10 panoramas the trinocular possible models are 120, while the possible binocular models are (n*(n-1))/2= 45. Obviously not all the panoramas will be formed, because a minimum of four common tie points is required. It is possible to select and filter out the best models by fixing a critical value on sigma naught for testing. The advantages are as follows: - no need of approximate value for the unknown coordinates and parameters; - easiness to find gross errors in the gradual block formation. The epipolar geometry has been investigated, the expression of the curve described and assessed: it can be useful in the trinocular model formation. For any try of the model the operator, thus causing troubles to him, must approximately choose formation, up to now the starting value of the orientation bearing for any panorama. However, a new procedure, not requiring any approximate value of the orientation bearing, has been set-up and tested: the orientation bearing can vary stepwise by a given selected interval, until the model is successfully formed. The computation time increases significantly, but it is still negligible. Up to now the control and tie points selection and observation is manual, making the orientation and the plotting to be slow and time consuming. Nevertheless, on the other end, the procedure is fully controlled and supervised by the operator, enabling the observation of the residuals and allowing the improvement of the orientation, and to perform some statistical quality analysis on the adjustment, which are not possible with automatic procedures based on Structure from Motion (SfM). Another weak point of the SP is the speed of plotting. Commercial programs are now available, based on SFM algorithm, producing point clouds of good quality in an unknown scale and orientation. The correct scale and orientation can be therefore supplied by SP orientation with high accuracy. Examples are shown taken from two real projects.

3D models mapping optimization through an integrated parameterization approach: cases studies from Ravenna

Abstract. Image-based modelling tools based on SfM algorithms gained great popularity since several software houses provided applications able to achieve 3D textured models easily and automatically. The aim of this paper is to point out the importance of controlling models parameterization process, considering that automatic solutions included in these modelling tools can produce poor results in terms of texture utilization. In order to achieve a better quality of textured models from image-based modelling applications, this research presents a series of practical strategies aimed at providing a better balance between geometric resolution of models from passive sensors and their corresponding (u,v) map reference systems. This aspect is essential for the achievement of a high-quality 3D representation, since "apparent colour" is a fundamental aspect in the field of Cultural Heritage documentation. Complex meshes without native parameterization have to be "flatten" or "unwrapped" in the (u,v) parameter space, with the main objective to be mapped with a single image. This result can be obtained by using two different strategies: the former automatic and faster, while the latter manual and time-consuming. Reverse modelling applications provide automatic solutions based on splitting the models by means of different algorithms, that produce a sort of "atlas" of the original model in the parameter space, in many instances not adequate and negatively affecting the overall quality of representation. Using in synergy different solutions, ranging from semantic aware modelling techniques to quad-dominant meshes achieved using retopology tools, it is possible to obtain a complete control of the parameterization process.

A medical diagnostic tool based on radial basis function classifiers and evolutionary simulated annealing

ObjectiveThe profusion of data accumulating in the form of medical records could be of great help for developing medical decision support systems. The objective of this paper is to present a methodology for designing data-driven medical diagnostic tools, based on neural network classifiers. MethodsThe proposed approach adopts the radial basis function (RBF) neural network architecture and the non-symmetric fuzzy means (NSFM) training algorithm, which presents certain advantages including better approximation capabilities and shorter computational times. The novelty in this work consists of adapting the NSFM algorithm to train RBF classifiers, and suitably tailoring the evolutionary simulated annealing (ESA) technique to optimize the produced RBF models. The integration of ESA is critical as it helps the optimization procedure to escape from local minima, which could arise from the application of the traditional simulated annealing algorithm, and thus discover improved solutions. The resulting method is evaluated in nine different medical benchmark datasets, where the common objective is to train a suitable classifier. The evaluation includes a comparison with two different schemes for training classifiers, including a standard RBF training technique and support vector machines (SVMs). Accuracy% and the Matthews Correlation Coefficient (MCC) are used for comparing the performance of the three classifiers. ResultsResults show that the use of ESA helps to greatly improve the performance of the NSFM algorithm and provide satisfactory classification accuracy. In almost all benchmark datasets, the best solution found by the ESA-NSFM algorithm outperforms the results produced by the SFM algorithm and SVMs, considering either the accuracy% or the MCC criterion. Furthermore, in the majority of datasets, the average solution of the ESA-NSFM population is statistically significantly higher in terms of accuracy% and MCC at the 95% confidence level, compared to the global optimum solution that its rivals could achieve. As far as computational times are concerned, the proposed approach was found to be faster compared to SVMs. ConclusionsThe results of this study suggest that the ESA-NSFM algorithm can form the basis of a generic method for knowledge extraction from data originating from different kinds of medical records. Testing the proposed approach on a number of benchmark datasets, indicates that it provides increased diagnostic accuracy in comparison with two different classifier training methods.

Topographic structure from motion: a new development in photogrammetric measurement

ABSTRACTThe production of topographic datasets is of increasing interest and application throughout the geomorphic sciences, and river science is no exception. Consequently, a wide range of topographic measurement methods have evolved. Despite the range of available methods, the production of high resolution, high quality digital elevation models (DEMs) requires a significant investment in personnel time, hardware and/or software. However, image‐based methods such as digital photogrammetry have been decreasing in costs. Developed for the purpose of rapid, inexpensive and easy three‐dimensional surveys of buildings or small objects, the ‘structure from motion’ photogrammetric approach (SfM) is an image‐based method which could deliver a methodological leap if transferred to geomorphic applications, requires little training and is extremely inexpensive. Using an online SfM program, we created high‐resolution digital elevation models of a river environment from ordinary photographs produced from a workflow that takes advantage of free and open source software. This process reconstructs real world scenes from SfM algorithms based on the derived positions of the photographs in three‐dimensional space. The basic product of the SfM process is a point cloud of identifiable features present in the input photographs. This point cloud can be georeferenced from a small number of ground control points collected in the field or from measurements of camera positions at the time of image acquisition. The georeferenced point cloud can then be used to create a variety of digital elevation products. We examine the applicability of SfM in the Pedernales River in Texas (USA), where several hundred images taken from a hand‐held helikite are used to produce DEMs of the fluvial topographic environment. This test shows that SfM and low‐altitude platforms can produce point clouds with point densities comparable with airborne LiDAR, with horizontal and vertical precision in the centimeter range, and with very low capital and labor costs and low expertise levels. Copyright © 2012 John Wiley & Sons, Ltd.

A Fast Bilinear Structure from Motion Algorithm Using a Video Sequence and Inertial Sensors

In this paper, we study the benefits of the availability of a specific form of additional information—the vertical direction (gravity) and the height of the camera, both of which can be conveniently measured using inertial sensors and a monocular video sequence for 3D urban modeling. We show that in the presence of this information, the SfM equations can be rewritten in a bilinear form. This allows us to derive a fast, robust, and scalable SfM algorithm for large scale applications. The SfM algorithm developed in this paper is experimentally demonstrated to have favorable properties compared to the sparse bundle adjustment algorithm. We provide experimental evidence indicating that the proposed algorithm converges in many cases to solutions with lower error than state-of-art implementations of bundle adjustment. We also demonstrate that for the case of large reconstruction problems, the proposed algorithm takes lesser time to reach its solution compared to bundle adjustment. We also present SfM results using our algorithm on the Google StreetView research data set.

Taking computer vision aloft – archaeological three‐dimensional reconstructions from aerial photographs with photoscan

AbstractStructure from motion (SFM) algorithms are known for their ability to reconstruct a sparse point cloud of scenes that were imaged by a series of overlapping photographs. When complemented by stereo‐matching algorithms, detailed three‐dimensional models can be built from such photograph collections in a fully automated way. Since 2010, a computer vision software package called PhotoScan has been available from the Russian manufacturer AgiSoft LLC. Even though the programme has a straightforward and simple interface, state‐of‐the‐art SFM and other computer vision algorithms are implemented that enable PhotoScan to generate very dense and accurate three‐dimensional meshes of various (archaeological) settings captured in two‐dimensional imagery. Using a mixture of archaeological case studies, it will be shown that this low‐cost application produces excellent results from aerial imagery in a minimum of time with almost no effort needed. Because of its solid and uncomplicated operation, it looks like this application could become an essential part of the aerial archaeologist's toolkit. Copyright © 2011 John Wiley & Sons, Ltd.

Constrained planar motion analysis by decomposition

General SFM methods give poor results for images captured by constrained motions such as planar motion. In this paper, we propose new SFM algorithms for images captured under a common but constrained planar motion: the image plane is perpendicular to the motion plane. We show that a 2D image captured under such constrained planar motion can be decoupled into two 1D images: one 1D projective and one 1D affine. We then introduce the 1D affine camera model for completing 1D camera models. Next, we describe new subspace reconstruction methods, and apply these methods to the images captured by concentric mosaics, which undergo a special case of constrained planar motion. Finally, we demonstrate both in theory and experiments the advantage of the decomposition method over the general SFM methods by incorporating the constrained motion into the earliest stage of motion analysis.

Understanding the Behavior of SFM Algorithms: A Geometric Approach

We put forth in this paper a geometrically motivated motion error analysis which is capable of supporting investigation of global effect such as inherent ambiguities. This is in contrast with the usual statistical kinds of motion error analyses which can only deal with local effect such as noise perturbations, and where much of the results regarding global ambiguities are empirical in nature. The error expression that we derive allows us to predict the exact conditions likely to cause ambiguities and how these ambiguities vary with motion types such as lateral or forward motion. Given the erroneous 3-D motion estimates caused by the inherent ambiguities, it is also important to study the behavior of the resultant distortion in depth recovered under different motion-scene configurations. Such an investigation may alert us to the occurrence of ambiguities under different conditions and be more careful in picking the solution. Our formulation, though geometrically motivated, was also put to use in modeling the effect of noise and in revealing the strong influence of feature distribution. Experiments on both synthetic and real image sequences were conducted to verify the various theoretical predictions.