Photogrammetric Computer Vision Research Articles

Application of Photogrammetric Computer Vision and Deep Learning in High-Resolution Underwater Mapping: A Case Study of Shallow-Water Coral Reefs

Abstract. Underwater mapping is vital for engineering applications and scientific research in ocean environments, with coral reefs being a primary focus. Unlike more uniform and predictable terrestrial environments, coral reefs present a unique challenge for 3D reconstruction due to their intricate and irregular structures. Traditional 3D reconstruction methods struggle to accurately capture the nuances of coral reefs. This is primarily because coral reefs exhibit a high degree of spatial heterogeneity, featuring diverse shapes, sizes, and textures. Additionally, the dynamic nature of underwater conditions, such as varying light, water clarity, and movement, further complicates the accurate geometrical estimation of these ecosystems. With the rapid advancement of photogrammetric computer vision and deep learning technologies, there are emerging methods that have potential to enhance the quality of its 3D reconstruction. In this context, this study formulates a coral reef reconstruction workflow that incorporates these cutting-edge technologies. This workflow is divided into two core stages: sparse reconstruction and dense reconstruction. We conduct individual summaries of the relevant research efforts in these stages and outline the available methods. To assess the specific capabilities of these methods, we apply them to real-world coral reef images and conduct a comprehensive evaluation. Additionally, we analyze the strengths and weaknesses of different methods and identify areas for improvement. We believe this study offers valuable references for future research in underwater mapping.

UAV Photogrammetry as a Photogrammetric Computer Vision Technique

Geleneksel fotogrametriden farklı olarak, düşük maliyetli metrik olmayan dijital kameralarla farklı yüksekliklerden ve farklı açılardan çekilmiş görüntü verilerinin toplanmasına olanak sağlayan İHA-fotogrametrisi, bilgisayarlı görü ve fotogrametrinin kombinasyonunu içeren yöntemler ve iş akış sürecine sahiptir. Bu kapsamda kullanılmakta olan Hareket Tabanlı Yapısal Algılama (SfM) tekniği, İHA- tabanlı ortofoto ve 3B arazi modeli üretiminin standart tekniği haline gelmiştir. Bu nedenle literatürde SFM fotogrametrisi terminolojisi de kullanılmaya başlamıştır. Bu çalışmada birbirine yakın terminolojiye sahip ve iç içe geçmiş yöntem ve algoritmalara sahip bu teknikler (fotogrametri, İHA-fotogrametrisi, SfM fotogrametrisi) arasındaki ayrım ve benzerlikler kavramsal olarak ve bir uygulama pratiği açısından analiz edilecektir.

Fine-Grained 3D Modeling and Semantic Mapping of Coral Reefs Using Photogrammetric Computer Vision and Machine Learning.

Corals play a crucial role as the primary habitat-building organisms within reef ecosystems, forming expansive structures that extend over vast distances, akin to the way tall buildings define a city's skyline. However, coral reefs are vulnerable to damage and destruction due to their inherent fragility and exposure to various threats, including the impacts of climate change. Similar to successful city management, the utilization of advanced underwater videography, photogrammetric computer vision, and machine learning can facilitate precise 3D modeling and the semantic mapping of coral reefs, aiding in their careful management and conservation to ensure their survival. This study focuses on generating detailed 3D mesh models, digital surface models, and orthomosaics of coral habitats by utilizing underwater coral images and control points. Furthermore, an innovative multi-modal deep neural network is designed to perform the pixel-wise semantic segmentation of orthomosaics, enabling the projection of resulting semantic maps onto a 3D space. Notably, this study achieves a significant milestone by accomplishing semantic fine-grained 3D modeling and rugosity evaluation of coral reefs with millimeter-level accuracy, providing a potent means to understand coral reef variations under climate change with high spatial and temporal resolution.

Learning to Extract Building Footprints From Off-Nadir Aerial Images.

Extracting building footprints from aerial images is essential for precise urban mapping with photogrammetric computer vision technologies. Existing approaches mainly assume that the roof and footprint of a building are well overlapped, which may not hold in off-nadir aerial images as there is often a big offset between them. In this paper, we propose an offset vector learning scheme, which turns the building footprint extraction problem in off-nadir images into an instance-level joint prediction problem of the building roof and its corresponding "roof to footprint" offset vector. Thus the footprint can be estimated by translating the predicted roof mask according to the predicted offset vector. We further propose a simple but effective feature-level offset augmentation module, which can significantly refine the offset vector prediction by introducing little extra cost. Moreover, a new dataset, Buildings in Off-Nadir Aerial Images (BONAI), is created and released in this paper. It contains 268,958 building instances across 3,300 aerial images with fully annotated instance-level roof, footprint, and corresponding offset vector for each building. Experiments on the BONAI dataset demonstrate that our method achieves the state-of-the-art, outperforming other competitors by 3.37 to 7.39 points in F1-score. The codes, datasets, and trained models are available at https://github.com/jwwangchn/BONAI.git.

The influence of flight height and overlap on UAV imagery over featureless surfaces and constructing formulas predicting the geometrical accuracy

ABSTRACT The improvement of unmanned aerial system and photogrammetric computer vision (CV) algorithms has presented an aerial imaging technique for high accuracy and low-cost alternatives for mapping and topographic applications. Structure from motion (SFM) is an automation photogrammetric CV algorithm used for generating 3D coloured point clouds and 3D models from overlapping images. One of the biggest problems preventing the automation extraction and matching key points in the aligning aerial images is the non-texture of the covered area surface. This paper assessed the effect of flight altitude and overlap degree on 3D point clouds’ geometric accuracy and models produced by unmanned aerial vehicle (UAV) images captured over non-textured sandy areas. Four flight altitudes (140, 160, 180 and 200 m) related to spatial resolution (3.41, 3.9, 4.39 and 4.68 cm/pix ground sample distance (GSD)), respectively, and three overlap levels (60%, 70% and 80%) were assessed using RGB images captured by UX5 UAV over a non-textured sandy area in Jahra, Kuwait. The results showed that altitude increment might reduce flight time, processing time and cost, keeping with the acceptable and suitable geometric accuracy. Generally, favourable results are obtained for the four altitudes and overlap degrees of 80% at least. Multivariate nonlinear regression analysis was used to fit the relation between geometric accuracy, image overlap and GSD cm/pixel for the seven missions determining two formulas that predict the geometrical accuracy of the UAV point cloud with a precision of 92.76% and 91.91% for both formulas.

Keypose synthesis from 3D motion capture data by using evolutionary clustering

SummaryMotion Capture datasets are captured as high‐frequency discrete samples using photogrammetric computer vision or various industrial geodetic measurement methods over the relevant model. Because Motion Capture data are inherently large‐datasets, expressing Motion Capture data without employing a motion data abstraction approach such as keypose is challenging. Keypose synthesis is a serious problem in many applications. Unfortunately, the local statistical features of Motion Capture data over time periods vary often, making it difficult to determine a key point summing the motion activation involved. Conventional clustering methods, such as Fuzzy C‐Means (FCM), that are sensitive to initial conditions, can be fitted to a local solution rather than producing a highly reliable keypose. Data clustering can be performed using evolutionary search methods without being limited to relatively local solutions. In this paper, the Motion Capture data were clustered using evolutionary search algorithms, and related keyposes were synthesized using the “minimum‐distance to cluster‐centers” principle. In Experiments section of this paper, motion data containing sportive movements were clustered by using evolutionary algorithms (i.e., Particle Swarm Optimization, Artificial Bee Colony, and Differential Search Algorithm) and classical clustering algorithms (i.e., Self‐Organizing Neural Network, FCM) to obtain related keyposes. The computed statistics exposed that evolutionary methods were more successful in obtaining keypose than classical methods.

Pavement Macrotexture Determination Using Multi-View Smartphone Images

Pavement macrotexture contributes greatly to road surface friction, which in turn plays a vital role in reducing road accidents. Conventional methods for macrotexture measurement are either expensive, time-consuming, or of poor repeatability. Based on multi-view smartphone images collected in situ, this paper develops and evaluates an affordable and convenient alternative approach for pavement macrotexture measurement. Photogrammetric computer vision techniques are first applied to create high resolution point clouds of the pavement. Analytics are then developed to determine the macrotexture metric: mean profile depth by using the image-based point clouds. Experiments are carried out with 790 images over 25 spots on three state routes and six spots at an Indiana Department of Transportation test site. We demonstrate multi-view smartphone images can yield results comparable to the ones from the conventional laser texture scanner. It is expected that the developed approach can be adopted for large scale operational uses.

A Comparative Study of Weighting Methods for Local Reference Frame

In the field of photogrammetric engineering, computer vision, and graphics, local shape description is an active research area. A wide variety of local shape descriptors (LSDs) have been designed for different applications, such as shape retrieval, object recognition, and 3D registration. The local reference frame (LRF) is an important component of the LSD. Its repeatability and robustness directly influence the descriptiveness and robustness of the LSD. Several weighting methods have been proposed to improve the repeatability and robustness of the LRF. However, no comprehensive comparison has been implemented to evaluate their performance under different data modalities and nuisances. In this paper, we focus on the comparison of weighting methods by using six datasets with different data modalities and application contexts. We evaluate the repeatability of the LRF under different nuisances, including occlusion, clutter, partial overlap, varying support radii, Gaussian noise, shot noise, point density variation, and keypoint localization error. Through the experiments, the traits, advantages, and disadvantages of weighting methods are summarized.

A new method to determine multi-angular reflectance factor from lightweight multispectral cameras with sky sensor in a target-less workflow applicable to UAV

A new physically based method to estimate hemispheric-directional reflectance factor (HDRF) from lightweight multispectral cameras that have a downwelling irradiance sensor is presented. It combines radiometry with photogrammetric computer vision to derive geometrically and radiometrically accurate data purely from the images, without requiring reflectance targets or any other additional information apart from the imagery. The sky sensor orientation is initially computed using photogrammetric computer vision and revised with a non-linear regression comprising radiometric and photogrammetry-derived information. It works for both clear sky and overcast conditions. A ground-based test acquisition of a Spectralon target observed from different viewing directions and with different sun positions using a typical multispectral sensor configuration for clear sky and overcast showed that both the overall value and the directionality of the reflectance factor as reported in the literature were well retrieved. An RMSE of 3% for clear sky and up to 5 for overcast sky was observed.

Photogrammetric Computer Vision Workshop 2019 (PCV 19) and EarthVision 2019 Workshop

The Photogrammetric RecordVolume 34, Issue 165 p. 117-118 Notes Photogrammetric Computer Vision Workshop 2019 (PCV 19) and EarthVision 2019 Workshop First published: 26 March 2019 https://doi.org/10.1111/phor.8_12274Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat No abstract is available for this article. Volume34, Issue165March 2019Pages 117-118 RelatedInformation

MODELLING BUILDING COSTS FROM 3D BUILDING MODELS – ESTIMATING THE CONSTRUCTION EFFORT FROM IMAGE-BASED SURFACE MODELS OF DRY-STONE SHEPHERD SHELTERS (KRAS, SLOVENIA)

Abstract. In the second half of the 19th and early 20th century, sheep shepherds have built dry-stone shelters all over the Slovene Kras (or Karst) region. Despite being made out of stones that are interlocked without the use of any binding material, many of these vernacular constructions survived – even though sometimes only partially – the ravages of time. The fact that over one hundred fifty shepherd shelters are currently known is mainly due to the craftsmanship of their builders and thanks to (and even despite) their present location. A majority of these stone constructions can be found in areas that are nowadays forested, thus shielding them from weather-related or anthropogenic damage (because they are difficult to spot). This paper reports on the geometric documentation of those shelters using a photogrammetric computer vision pipeline, thereby mainly focussing on the difficulties that were encountered during this process. However, such image-based modelling approaches merely yield digital three-dimensional (3D) approximations of the shelters’ surface geometry (along with some sub-optimal colour data). Although these 3D surface models might be suitable to digitally preserve vulnerable vernacular buildings to some extent, they do not magically advance our understanding of them. The second part of this article focuses, therefore, on the extraction of archaeological information from these digital 3D constructions. More specifically, the total amount of stones, the total building time and the building cost regarding caloric energy expenditure are estimated for each of the digitised shelters. Although this assessment of architectural energetics provided useful insight into the building efforts and nutrient uptake of the shepherds, it also revealed many assumptions and shortcomings that often characterise archaeological information extraction from digital 3D models of buildings.

FOCUSING ON OUT-OF-FOCUS: ASSESSING DEFOCUS ESTIMATION ALGORITHMS FOR THE BENEFIT OF AUTOMATED IMAGE MASKING

Abstract. Acquiring photographs as input for an image-based modelling pipeline is less trivial than often assumed. Photographs should be correctly exposed, cover the subject sufficiently from all possible angles, have the required spatial resolution, be devoid of any motion blur, exhibit accurate focus and feature an adequate depth of field. The last four characteristics all determine the “sharpness” of an image and the photogrammetric, computer vision and hybrid photogrammetric computer vision communities all assume that the object to be modelled is depicted “acceptably” sharp throughout the whole image collection. Although none of these three fields has ever properly quantified “acceptably sharp”, it is more or less standard practice to mask those image portions that appear to be unsharp due to the limited depth of field around the plane of focus (whether this means blurry object parts or completely out-of-focus backgrounds). This paper will assess how well- or ill-suited defocus estimating algorithms are for automatically masking a series of photographs, since this could speed up modelling pipelines with many hundreds or thousands of photographs. To that end, the paper uses five different real-world datasets and compares the output of three state-of-the-art edge-based defocus estimators. Afterwards, critical comments and plans for the future finalise this paper.

Recent trends in cultural heritage 3D survey: The photogrammetric computer vision approach

Abstract The techniques of measuring and 3D modelling based on images, as is typical in photogrammetry, grew in interest again in recent years, since a new generation of software tools has spread. These ones implement in different measure the algorithms developed by computer vision, increasing the automation of the standard photogrammetric process. This made the use of image-based approaches for 3D models reconstruction enormously increase, which is an essential part of the Cultural Heritage documentation and analysis processes. Starting from these assumptions, the aim of the paper is to evaluate what and where it is possible nowadays to find the main differences between photogrammetry and computer vision approaches and how these have to be considered in the choice of the processing technique. The analysis has been performed starting from a theoretical point of view in order to trace the main characteristics of the two methods. Moreover, in order to complete the investigation, an experimental part is reported on two particular cases study, considered as representative of two types of usually surveyed objects. The results allow to enlighten some differences between the two image processing approaches, in terms of accuracy and achieved products.

Photogrammetric Computer Vision: Statistics, Geometry, Orientation and Reconstruction

Photogrammetric Computer Vision: Statistics, Geometry, Orientation and Reconstruction

AUTOMATED INSPECTION OF POWER LINE CORRIDORS TO MEASURE VEGETATION UNDERCUT USING UAV-BASED IMAGES

Abstract. Power line corridor inspection is a time consuming task that is performed mostly manually. As the development of UAVs made huge progress in recent years, and photogrammetric computer vision systems became well established, it is time to further automate inspection tasks. In this paper we present an automated processing pipeline to inspect vegetation undercuts of power line corridors. For this, the area of inspection is reconstructed, geo-referenced, semantically segmented and inter class distance measurements are calculated. The presented pipeline performs an automated selection of the proper 3D reconstruction method for on the one hand wiry (power line), and on the other hand solid objects (surrounding). The automated selection is realized by performing pixel-wise semantic segmentation of the input images using a Fully Convolutional Neural Network. Due to the geo-referenced semantic 3D reconstructions a documentation of areas where maintenance work has to be performed is inherently included in the distance measurements and can be extracted easily. We evaluate the influence of the semantic segmentation according to the 3D reconstruction and show that the automated semantic separation in wiry and dense objects of the 3D reconstruction routine improves the quality of the vegetation undercut inspection. We show the generalization of the semantic segmentation to datasets acquired using different acquisition routines and to varied seasons in time.

Method for assessing beam column joints in RC structures using photogrammetric computer vision

The response of beam column joints is fundamental for structural analysis of reinforcement concrete frames, particularly, when subjected to dynamic actions. Monitoring the key parameters of those joints is essential to perform a detailed analysis of the structure. These are often difficult to obtain, and only a few sections in critical regions are assessed. High-end technologies can bring huge advantages in such measurements. Methods based on computer vision have been applied to monitoring reinforced concrete structures with excellent results, considering the accuracy achieved and the large quantity of information recorded. In this paper, a new method, named “Photo-Node,” based on photogrammetry and image processing is presented. Photo-Node is a noncontact method used to directly monitoring concrete surfaces, developed to measure the relevant physical parameters of reinforcement concrete beam column joints during experimental tests. The method addresses several postprocessing features to evaluate joint rotation, joint distortion, principal directions and stresses, cracks pattern, and load path. Additionally, the critical sections of the beam, that is, the region adjacent to the joint where the formation of the plastic hinge is expected, can be also evaluated. Photo-Node provides a large amount of data, which can be processed to properly assess the joints behaviour. The degree of discretisation achieved enables to identify behaviours impossible to detect using traditional and less discretised approaches. The application of the method showed promising and accurate results, proving to be an efficient and a wide-ranging tool to be used in experimental programs.

Implicit Regularization for Reconstructing 3D Building Rooftop Models Using Airborne LiDAR Data.

With rapid urbanization, highly accurate and semantically rich virtualization of building assets in 3D become more critical for supporting various applications, including urban planning, emergency response and location-based services. Many research efforts have been conducted to automatically reconstruct building models at city-scale from remotely sensed data. However, developing a fully-automated photogrammetric computer vision system enabling the massive generation of highly accurate building models still remains a challenging task. One the most challenging task for 3D building model reconstruction is to regularize the noises introduced in the boundary of building object retrieved from a raw data with lack of knowledge on its true shape. This paper proposes a data-driven modeling approach to reconstruct 3D rooftop models at city-scale from airborne laser scanning (ALS) data. The focus of the proposed method is to implicitly derive the shape regularity of 3D building rooftops from given noisy information of building boundary in a progressive manner. This study covers a full chain of 3D building modeling from low level processing to realistic 3D building rooftop modeling. In the element clustering step, building-labeled point clouds are clustered into homogeneous groups by applying height similarity and plane similarity. Based on segmented clusters, linear modeling cues including outer boundaries, intersection lines, and step lines are extracted. Topology elements among the modeling cues are recovered by the Binary Space Partitioning (BSP) technique. The regularity of the building rooftop model is achieved by an implicit regularization process in the framework of Minimum Description Length (MDL) combined with Hypothesize and Test (HAT). The parameters governing the MDL optimization are automatically estimated based on Min-Max optimization and Entropy-based weighting method. The performance of the proposed method is tested over the International Society for Photogrammetry and Remote Sensing (ISPRS) benchmark datasets. The results show that the proposed method can robustly produce accurate regularized 3D building rooftop models.

Evaluations on multi-scale camera networks for precise and geo-accurate reconstructions from aerial and terrestrial images with user guidance

During the last decades photogrammetric computer vision systems have been well established in scientific and commercial applications. Recent developments in image-based 3D reconstruction systems have resulted in an easy way of creating realistic, visually appealing and accurate 3D models. We present a fully automated processing pipeline for metric and geo-accurate 3D reconstructions of complex geometries supported by an online feedback method for user guidance during image acquisition. Our approach is suited for seamlessly matching and integrating images with different scales, from different view points (aerial and terrestrial), and with different cameras into one single reconstruction. We evaluate our approach based on different datasets for applications in mining, archaeology and urban environments and thus demonstrate the flexibility and high accuracy of our approach. Our evaluation includes accuracy related analyses investigating camera self-calibration, georegistration and camera network configuration.

ON THE USE OF UAVS IN MINING AND ARCHAEOLOGY - GEO-ACCURATE 3D RECONSTRUCTIONS USING VARIOUS PLATFORMS AND TERRESTRIAL VIEWS

Abstract. During the last decades photogrammetric computer vision systems have been well established in scientific and commercial applications. Especially the increasing affordability of unmanned aerial vehicles (UAVs) in conjunction with automated multi-view processing pipelines have resulted in an easy way of acquiring spatial data and creating realistic and accurate 3D models. With the use of multicopter UAVs, it is possible to record highly overlapping images from almost terrestrial camera positions to oblique and nadir aerial images due to the ability to navigate slowly, hover and capture images at nearly any possible position. Multi-copter UAVs thus are bridging the gap between terrestrial and traditional aerial image acquisition and are therefore ideally suited to enable easy and safe data collection and inspection tasks in complex or hazardous environments. In this paper we present a fully automated processing pipeline for precise, metric and geo-accurate 3D reconstructions of complex geometries using various imaging platforms. Our workflow allows for georeferencing of UAV imagery based on GPS-measurements of camera stations from an on-board GPS receiver as well as tie and control point information. Ground control points (GCPs) are integrated directly in the bundle adjustment to refine the georegistration and correct for systematic distortions of the image block. We discuss our approach based on three different case studies for applications in mining and archaeology and present several accuracy related analyses investigating georegistration, camera network configuration and ground sampling distance. Our approach is furthermore suited for seamlessly matching and integrating images from different view points and cameras (aerial and terrestrial as well as inside views) into one single reconstruction. Together with aerial images from a UAV, we are able to enrich 3D models by combining terrestrial images as well inside views of an object by joint image processing to generate highly detailed, accurate and complete reconstructions.

Fusion of range-based data and image-based datasets for efficient documentation of cultural heritage objects and sites

Abstract. Nowadays it is possible to measure accurately dense point clouds either with aerial/terrestrial laser scanning systems or with imagebased solutions (namely based on photogrammetric computer vision algorithms such as structure-from-motion (SfM)), from which highly detailed 3D models can be achieved. Besides, direct tools in the form of simple devices such as rulers, compass and plumblines are usually required in simple metric surveys, as well as high-end surveying and geodetic instruments such as robotized imagebased total stations and GNSS (probably to a lesser degree but still required) to set the archaeological/architectural recording project in a global reference frame. With all this gamut of image-based and range-based sensors and datasets (in the form of coordinates, point clouds or 3D models), in different coordinate systems (most of the times local for each device), lack of uniform scale, orientation and levelling, the fusion of data tends to be cumbersome. This paper presents an efficient way to fuse and merge different datasets in the form of point clouds/3D models and geodetic/UTM coordinates. The new developed 3DVEM – Register GEO software is able to handle datasets coming from both direct and indirect methods in order to provide unified and precise deliverables.