Close-range Imagery Research Articles

Comparative analysis of traditional and transfer learning algorithms for coal spoil classification via close-range imagery

ABSTRACT The characterisation of materials is a prerequisite for evaluating and predicting the stability of mining waste dumps. Over the past three decades, the BHP Mitsubishi Alliance Coal framework has been a cornerstone in Australian coal mines for characterising waste dump materials. However, its reliance on subjective human observations has introduced potential inaccuracies and subjectivity into the process. In response to these limitations, this study proposes an innovative approach to classify coal spoil attributes by remotely acquiring images through phones/tablets. Automated image-based classification relies on feature extraction and a substantial amount of data. Nevertheless, the inherent complexity of geological factors contributing to the formation of both rare and dominant materials leads to imbalanced data. Recognising the need for classification mechanisms to overcome these challenges in spoil classification, the study explores and compares the use of convolutional neural networks, hybrid deep learning, and traditional techniques. Among the sixteen models evaluated in this study, the ResNet18-support vector machine model emerges as a powerful tool in geotechnical characterisation. However, it is essential to address issues of interpretability and adaptability to diverse datasets. As this study evolves, the field of geotechnical characterisation of spoil can anticipate the development of more robust methods in the future.

GAN-assisted Road Segmentation from Satellite Imagery

Geo-information extraction from satellite imagery has become crucial to carry out large-scale ground surveys in a short amount of time. With the increasing number of commercial satellites launched into orbit in recent years, high-resolution RGB color remote sensing imagery has attracted a lot of attention. However, because of the high cost of image acquisition and even more complicated annotation procedures, there are limited high-resolution satellite datasets available. Compared to close-range imagery datasets, existing satellite datasets have a much lower number of images and cover only a few scenarios (cities, background environments, etc. ). They may not be sufficient for training robust learning models that fit all environmental conditions or be representative enough for training regional models that optimize for local scenarios. Instead of collecting and annotating more data, using synthetic images could be another solution to boost the performance of a model. This study proposes a GAN-assisted training scheme for road segmentation from high-resolution RGB color satellite images, which includes three critical components: a) synthetic training sample generation, b) synthetic training sample selection, and c) assisted training strategy. Apart from the GeoPalette and cSinGAN image generators introduced in our prior work, this paper in detail explains how to generate new training pairs using OpenStreetMap (OSM) and introduces a new set of evaluation metrics for selecting synthetic training pairs from a pool of generated samples. We conduct extensive quantitative and qualitative experiments to compare different image generators and training strategies. Our experiments on the downstream road segmentation task show that 1) our proposed metrics are more aligned with the trained model performance compared to commonly used GAN evaluation metrics such as the Fréchet inception distance (FID); and 2) by using synthetic data with the best training strategy, the model performance, mean Intersection over Union (mean IoU), is improved from 60.92% to 64.44%, when 1,000 real training pairs are available for learning, which reaches a similar level of performance as a model that is standard-trained with 4,000 real images (64.59%), i.e. , enabling a 4-fold reduction in real dataset size.

Linking the Spectra of Decomposing Litter to Ecosystem Processes: Tandem Close-Range Hyperspectral Imagery and Decomposition Metrics

Efforts to monitor terrestrial decomposition dynamics at broad spatial scales are hampered by the lack of a cost-effective and scalable means to track the decomposition process. Recent advances in remote sensing have enabled the simulation of litter spectra throughout decomposition for grasses in general, yet unique decomposition pathways are hypothesized to create subtly different litter spectral signatures with unique ecosystem functional significance. The objectives of this study were to improve spectra–decomposition linkages and thereby enable the more comprehensive monitoring of ecosystem processes such as nutrient and carbon cycles. Using close-range hyperspectral imaging, litter spectra and multiple decomposition metrics were concurrently monitored in four classes of naturally decayed litter under four decomposition treatments. The first principal component accounted for approximately 94% of spectral variation in the close-range imagery and was attributed to the progression of decomposition. Decomposition-induced spectral changes were moderately correlated with the leaf carbon to nitrogen ratio (R2 = 0.52) and sodium hydroxide extractables (R2 = 0.45) but had no correlation with carbon dioxide flux. Temperature and humidity strongly influenced the decomposition process but did not influence spectral variability or the patterns of surface decomposition. The outcome of the study is that litter spectra are linked to important metrics of decomposition and thus remote sensing could be utilized to assess decomposition dynamics and the implications for nutrient recycling at broad spatial scales. A secondary study outcome is the need to resolve methodological challenges related to inducing unique decomposition pathways in a lab environment. Improving decomposition treatments that mimic real-world conditions of temperature, humidity, insolation, and the decomposer community will enable an improved understanding of the impacts of climatic change, which are expected to strongly affect microbially mediated decomposition.

As-Textured As-Built BIM Using Sensor Fusion, Zee Ain Historical Village as a Case Study

The work described in the paper emphasizes the importance of integrating imagery and laser scanner techniques (TLS) to optimize the geometry and visual quality of Heritage BIM. The fusion-based workflow was approached during the recording of Zee Ain Historical Village in Saudi Arabia. The village is a unique example of traditional human settlements, and represents a complex natural and cultural heritage site. The proposed workflow divides data integration into two levels. At the basic level, UAV photogrammetry with enhanced mobility and visibility is used to map the ragged terrain and supplement TLS point data in upper and unaccusable building zones where shadow data originated. The merging of point clouds ensures that the building’s overall geometry is correctly rebuilt and that data interpretation is improved during HBIM digitization. In addition to the correct geometry, texture mapping is particularly important in the area of cultural heritage. Constructing a realistic texture remains a challenge in HBIM; because the standard texture and materials provided in BIM libraries do not allow for reliable representation of heritage structures, mapping and sharing information are not always truthful. Thereby, at the second level, the workflow proposed true orthophoto texturing method for HBIM models by combining close-range imagery and laser data. True orthophotos have uniform scale that depicts all objects in their respective planimetric positions, providing reliable and realistic mapping. The process begins with the development of a Digital Surface Model (DSM) by sampling TLS 3D points in a regular grid, with each cell uniquely associated with a model point. Then each DSM cell is projected in the corresponding perspective imagery in order to map the relevant spectral information. The methods allow for flexible data fusion and image capture using either a TLS-installed camera or a separate camera at the optimal time and viewpoint for radiometric data. The developed workflows demonstrated adequate results in terms of complete and realistic textured HBIM, allowing for a better understanding of the complex heritage structures.

Development of a Fully Convolutional Neural Network to Derive Surf-Zone Bathymetry from Close-Range Imagery of Waves in Duck, NC

Timely observations of nearshore water depths are important for a variety of coastal research and management topics, yet this information is expensive to collect using in situ survey methods. Remote methods to estimate bathymetry from imagery include using either ratios of multi-spectral reflectance bands or inversions from wave processes. Multi-spectral methods work best in waters with low turbidity, and wave-speed-based methods work best when wave breaking is minimal. In this work, we build on the wave-based inversion approaches, by exploring the use of a fully convolutional neural network (FCNN) to infer nearshore bathymetry from imagery of the sea surface and local wave statistics. We apply transfer learning to adapt a CNN originally trained on synthetic imagery generated from a Boussinesq numerical wave model to utilize tower-based imagery collected in Duck, North Carolina, at the U.S. Army Engineer Research and Development Center’s Field Research Facility. We train the model on sea-surface imagery, wave conditions, and associated surveyed bathymetry using three years of observations, including times with significant wave breaking in the surf zone. This is the first time, to the authors’ knowledge, an FCNN has been successfully applied to infer bathymetry from surf-zone sea-surface imagery. Model results from a separate one-year test period generally show good agreement with survey-derived bathymetry (0.37 m root-mean-squared error, with a max depth of 6.7 m) under diverse wave conditions with wave heights up to 3.5 m. Bathymetry results quantify nearshore bathymetric evolution including bar migration and transitions between single- and double-barred morphologies. We observe that bathymetry estimates are most accurate when time-averaged input images feature visible wave breaking and/or individual images display wave crests. An investigation of activation maps, which show neuron activity on a layer-by-layer basis, suggests that the model is responsive to visible coherent wave structures in the input images.

Extracting three-dimensional (3D) spatial information from sequential oblique unmanned aerial system (UAS) imagery for digital surface modeling

ABSTRACT The advent of unmanned aerial system (UAS) has prompted close-range imagery a prevalent source to diversify spatial applications. In addition to nadir scenes, UAS is able to take oblique imagery, which increases the opportunity to acquire sophisticated spatial information from different viewing angles. These images provide more possibilities to reconstruct the land surfaces more completely in three-dimensions (3D). However, dealing with UAS imagery for 3D modelling has been a challenging task for years due to unstable and/or unknown exterior orientation parameters (EOPs) measured by direct georeferencing. With sequential oblique UAS imagery with inadequate or missing EOPs, this paper attempts to extract 3D spatial information from these types of images to achieve digital surface reconstruction. A modular workflow integrating the recovery of camera EOPs and 3D reconstruction in a relative space is presented. These images are spatially related by a feature-based incremental structure-from-motion (fi-SfM) for localization, stereo pairs selection and modification. Digital surface reconstruction, thenceforth, is addressed through dense matching and space intersection upon the outcomes of fi-SfM. The experimental results show that the designed schema is coherent in estimating the camera EOPs and modifying the inappropriate image pairs for improved 3D reconstruction. Furthermore, the surface model generated by discrete stereo pairs can be merged automatically to present a complete digital surface model (DSM). The completeness assessment has verified that the majority of the land surface can be successfully obtained by more than 90%, and the accuracy less than 1 (m) indicates that the implemented workflow can be used to achieve 3D modelling effectively.

A NOVEL TIE POINT BASED STRATEGY FOR POINT CLOUD AND IMAGERY DATA FINE REGISTRATION

Abstract. Laser scanner generated point cloud and photogrammetric imagery are complimentary data for many applications and services. Misalignment between imagery and point cloud data is a common problem, which causes to inaccurate products and procedures. In this paper, a novel strategy is proposed for coarse to fine registration between close-range imagery and terrestrial laser scanner point cloud data. In such a case, tie points are extracted and matched from photogrammetric imagery and preprocessing is applied on generated tie points to eliminate non-robust ones. At that time, for every tie point, two neighbor pixels are selected and matched in all overlapped images. After that, coarse interior orientation parameters (IOPs) and exterior orientation parameters (EOPs) of the images are employed to reconstruct object space points of the tie point and its two neighbor pixels. Then, corresponding nearest points to the object space photogrammetric points are estimated in the point cloud data. Estimated three point cloud points are used to calculate a plane and its normal vector. Theoretically, every object space tie point should be located on the aforementioned plane, which is used as conditional equation alongside the collinearity equation to fine register the photogrammetric imagery network. Attained root mean square error (RMSE) results on check points, have been shown less than 2.3 pixels, which shows the accuracy, completeness and robustness of the proposed method.

Utilizing the Road Mark Training Set from Ground-Based Mapping System to Airborne Imagery in Deep Learning Framework

Abstract. Deep Learning is a kind of Machine Learning technology which utilizing the deep neural network to learn a promising model from a large training data set. Convolutional Neural Network (CNN) has been successfully applied in image segmentation and classification with high accuracy results. The CNN applies multiple kernels (also called filters) to extract image features via image convolution. It is able to determine multiscale features through the multiple layers of convolution and pooling processes. The variety of training data plays an important role to determine a reliable CNN model. The benchmarking training data for road mark extraction is mainly focused on close-range imagery because it is easier to obtain a close-range image rather than an airborne image. For example, KITTI Vision Benchmark Suite. This study aims to transfer the road mark training data from mobile lidar system to aerial orthoimage in Fully Convolutional Networks (FCN). The transformation of the training data from ground-based system to airborne system may reduce the effort of producing a large training data set.This study uses FCN technology and aerial orthoimage to localize road marks on the road regions. The road regions are first extracted from 2-D large-scale vector map. The input aerial orthoimage is 10 cm spatial resolution and the non-road regions are masked out before the road mark localization. The training data are road mark’s polygons, which are originally digitized from ground-based mobile lidar and prepared for the road mark extraction using mobile mapping system. This study reuses these training data and applies them for the road mark extraction using aerial orthoimage. The digitized training road marks are then transformed to road polygon based on mapping coordinates. As the detail of ground-based lidar is much better than the airborne system, the partially occulted parking lot in aerial orthoimage can also be obtained from the ground-based system. The labels (also called annotations) for FCN include road region, non-regions and road mark. The size of a training batch is 500 pixel by 500 pixel (50 m by 50 m on the ground), and the total number of training batches for training is 75 batches. After the FCN training stage, an independent aerial orthoimage (Figure 1a) is applied to predict the road marks. The results of FCN provide initial regions for road marks (Figure 1b). Usually, road marks show higher reflectance than road asphalts. Therefore, this study uses this characteristic to refine the road marks (Figure 1c) by a binary classification inside the initial road mark’s region.To compare the automatically extracted road marks (Figure 1c) and manually digitized road marks (Figure 1d), most road marks can be extracted using the training set from ground-based system. This study also selects an area of 600 m × 200 m in quantitative analysis. Among the 371 reference road marks, 332 can be extracted from proposed scheme, and the completeness reached 89%. The preliminary experiment demonstrated that most road marks can be successfully extracted by the proposed scheme. Therefore, the training data from the ground-based mapping system can be utilized in airborne orthoimage in similar spatial resolution.

INVESTIGATION OF PARALLAX ISSUES FOR MULTI-LENS MULTISPECTRAL CAMERA BAND CO-REGISTRATION

Abstract. The multi-lens multispectral cameras (MSCs), such as Micasense Rededge and Parrot Sequoia, can record multispectral information by each separated lenses. With their lightweight and small size, which making they are more suitable for mounting on an Unmanned Aerial System (UAS) to collect high spatial images for vegetation investigation. However, due to the multi-sensor geometry of multi-lens structure induces significant band misregistration effects in original image, performing band co-registration is necessary in order to obtain accurate spectral information. A robust and adaptive band-to-band image transform (RABBIT) is proposed to perform band co-registration of multi-lens MSCs. First is to obtain the camera rig information from camera system calibration, and utilizes the calibrated results for performing image transformation and lens distortion correction. Since the calibration uncertainty leads to different amount of systematic errors, the last step is to optimize the results in order to acquire a better co-registration accuracy. Due to the potential issues of parallax that will cause significant band misregistration effects when images are closer to the targets, four datasets thus acquired from Rededge and Sequoia were applied to evaluate the performance of RABBIT, including aerial and close-range imagery. From the results of aerial images, it shows that RABBIT can achieve sub-pixel accuracy level that is suitable for the band co-registration purpose of any multi-lens MSC. In addition, the results of close-range images also has same performance, if we focus on the band co-registration on specific target for 3D modelling, or when the target has equal distance to the camera.

LASER SCANNER AND CAMERA-EQUIPPED UAV ARCHITECTURAL SURVEYS

Abstract. The paper reports the workflow of 3D modelling and photorealistic texture mapping based on close range imagery acquired through a terrestrial laser scan and a camera mounted on a mini Unmanned Aerial Vehicle (UAV). The 3D models of the same architectural object – a watchtower named “Torre Pelosa” (Bari, Italy) - were compared in order to evaluate strengths and weaknesses of the used instruments and methods. The evaluated parameters where geo-referencing capability, manageability, manoeuvrability, operational constraint, survey completeness and costs. Despite a lower accuracy of the results, the use of a UAV mounted camera is essential to obtain a whole representation of the tower.

Design and implementation of a computational processing system for off-line digital close-range photogrammetry

Through the adoption of recent innovations in automation in vision metrology, it can be demonstrated that rigorous, yet user-friendly digital photogrammetric processes of calibration and orientation/triangulation can be incorporated into a computational scheme which is on the one hand capable of meeting the demands of high metric quality, while on the other offering the facilities necessary to support wider application by non-specialist users. The software system Australis, developed for image mensuration and restitution of off-line close-range photogrammetric networks, is featured to illustrate these processes and procedures. By describing the structure and components of Australis, the authors aim to demonstrate that many processes which have on occasion been viewed to be the exclusive province of automated, high-precision vision metrology are indeed suited to more general application across a broad range of fields which involve 3D object recording via close-range imagery.

Effective application of automated digital photogrammetry for geomorphological research

Developments in digital photogrammetry have provided geomorphologists with an automated tool to generate digital elevation models (DEMs) at exceedingly high densities. Although such software tools are available at low cost and run on far cheaper hardware than previous generations of photogrammetric instrumentation, some expertise is still required to derive accurate data. Various recommendations are provided that should enable the inexperienced user to make effective use of digital photogrammetry. Key issues discussed include the role of photo-control, the significance of checkpoints in the object space, and the importance of camera calibration data. An overview of self-calibration methods is provided, which is valuable in situations when a non-photogrammetric camera has been used. Accurate camera modelling also affects DEM quality and this issue is examined through assessing the impact of DEM inaccuracies upon derived data used for geomorphological enquiry. Although the automated software packages are designed primarily for use with satellite imagery or vertical aerial photography, it is explained how such software can be used for both close range and oblique imagery. The procedure requires rotating the object coordinate system defined by the photo-control points and the mathematics required to achieve this is provided. The procedure is used to derive the morphology of a soil surface using oblique, close-range imagery and demonstrates the effectiveness of this particular approach. Copyright © 1999 John Wiley & Sons, Ltd.

Digital Photogrammetric Techniques for Transportation Data Acquisition and Management

State-of-the-art digital photogrammetric techniques for the acquisition of transportation-related information from digital are presented. Photogrammetric data collection can be performed using either aerial or closerange sensors. The difference between the two lies on the nature of data that can be acquired, in direct analogy to the resolution differences. The current state of technology for digital imagery acquisition permits the direct capture of digital close-range imagery, whereas digital aerial imagery is produced by digitizing analog diapositives. Thus, the availability of metric quality imagery in digital format is ensured, either directly or indirectly. Advancements in digital photogrammetric research are making feasible the transition from analytical to digital photogrammetric methods in transportation applications. This transition is characterized by automation and improved performance of traditional tasks. Furthermore, the use of imagery in digital format is offering the potential to radically modify tasks, products, and procedures. Digital photogrammetric techniques can be embedded within a transportation data-management environment, supported by additional information, and supporting decision making.