Imaging Geometry Research Articles

Deep Shape-from-Template: Single-image quasi-isometric deformable registration and reconstruction

Shape-from-Template (SfT) solves 3D vision from a single image and a deformable 3D object model, called a template. Concretely, SfT computes registration (the correspondence between the template and the image) and reconstruction (the depth in camera frame). It constrains the object deformation to quasi-isometry. Real-time and automatic SfT represents an open problem for complex objects and imaging conditions. We present four contributions to address core unmet challenges to realise SfT with a Deep Neural Network (DNN). First, we propose a novel DNN called DeepSfT, which encodes the template in its weights and hence copes with highly complex templates. Second, we propose a semi-supervised training procedure to exploit real data. This is a practical solution to overcome the render gap that occurs when training only with simulated data. Third, we propose a geometry adaptation module to deal with different cameras at training and inference. Fourth, we combine statistical learning with physics-based reasoning. DeepSfT runs automatically and in real-time and we show with numerous experiments and an ablation study that it consistently achieves a lower 3D error than previous work. It outperforms in generalisation and achieves great performance in terms of reconstruction and registration error with wide-baseline, occlusions, illumination changes, weak texture and blur. • DeepSfT is a DNN fully-convolutional based on residual encoder-decoder for SfT. • We propose a semi-supervised end-to-end training procedure with synthetic and real data. • DeepSfT is a solution to cope with multiple imaging geometries, at training and inference. • We propose to combine DeepSfT with a physics-based estimation procedure.

A new phase unwrapping method for cross-track interferometric synthetic aperture radar systems

One of the essential applications of interferometric synthetic aperture radar (InSAR) systems is to estimate the ground surface height. To this end, the altitude of each pixel in the region is estimated according to the phase difference of the two received signals by two sensors. However, the sensor has an ambiguity in the relative terrain altitude due to the 2π cyclic nature of the interferometric phase, which leads to a significant error in altitude estimation. In this paper, by focusing on the phase unwrapping issue in InSAR processing, an efficient iterative approach to minimize the phase difference error is presented. For this purpose, first, according to the InSAR imaging geometry, the interferometric phase relationships are described. Second, based on the least-squares approach, the phase unwrapping problem is expressed as an optimization issue with a specific cost function. Third, the cost function is converted to a submodular convex Markov random field (MRF) function. Finally, the resulting optimization problem is solved by using a fast and efficient approach to optimize submodular convex MRF functions. The proposed method's performance is examined using several numerical studies compared to its relevant competitors. The mean square error (MSE) of elevation estimation is considered the performance metric. Numerical simulations have confirmed the efficiency of the proposed algorithm in terms of MSE of elevation estimation. As seen, the outperformance of the proposed method with respect to its best competitor is at least 2 times (3 dB). The operation times of various methods are also compared as a measure of their convergence rate. Based on simulation results, the convergence rate of the proposed algorithm is more than 50 times better than that of its closest competitor.

Gravitational lensing effects of supermassive black holes in cluster environments

ABSTRACT This study explores the gravitational lensing effects of supermassive black holes (SMBHs) in galaxy clusters. While the presence of central SMBHs in galaxies is firmly established, recent work from high-resolution simulations predict the existence of an additional population of wandering SMBHs. Though the masses of these SMBHs are a minor perturbation on the larger scale and individual galaxy scale dark matter components in the cluster, they can impact statistical lensing properties and individual lensed image configurations. Probing for these potentially observable signatures, we find that SMBHs imprint detectable signatures in rare, higher order strong lensing image configurations although they do not manifest any statistically significant detectable evidence in either the magnification distribution or the integrated shear profile. Investigating specific lensed image geometries, we report that a massive, near point-like, potential of an SMBH causes the following detectable effects: (i) image splitting leading to the generation of extra images; (ii) positional and magnification asymmetries in multiply imaged systems; and (iii) the apparent disappearance of a lensed counter image. Of these, image splitting inside the cluster tangential critical curve, is the most prevalent notable observational signature. We demonstrate these possibilities in two cases of observed giant arcs in SGAS J003341.5+024217 and RX J1347.5−1145, wherein specific image configurations seen can be reproduced with SMBHs. Future observations with high-resolution instrumentation (e.g. MAVIS-Very Large Telescope, MICADO-Extremely Large Telescope, and the upgraded ngVLA, along with data from the Euclid and Nancy Grace Roman Space Telescopes and the Rubin LSST Observatory are likely to allow us to probe these unique yet rare SMBHs lensing signatures.

Online geometry calibration for retrofit computed tomography from a mouse rotation system and a small-animal imager.

Computed tomography (CT) generates a three-dimensional rendering that can be used to interrogate a given region or desired structure from any orientation. However, in preclinical research, its deployment remains limited due to relatively high upfront costs. Existing integrated imaging systems that provide merged planar X-ray also dwarfs CT popularity in small laboratories due to their addedversatility. In this paper, we sought to generate CT-like data using an existing small-animal X-ray imager with a specialized specimen rotation system, or MiSpinner. This setup conforms to the cone-beam CT (CBCT) geometry, which demands high spatial calibration accuracy. Therefore, a simple but robust geometry calibration algorithm is necessary to ensure that the entire imaging system works properly andaccurately. Because the rotation system is not permanently affixed, we propose a structure tensor-based two-step online (ST-TSO) geometry calibration algorithm. Specifically, two datasets are needed, namely, calibration and actual measurements. A calibration measurement detects the background of the system forward X-ray projections. A study on the background image reveals the characteristics of the X-ray photon distribution, and thus, provides a reliable estimate of the imaging geometry origin. Actual measurements consisted of an X-ray of the intended object, including possible geometry errors. A comprehensive image processing technique helps to detect spatial misalignment information. Accordingly, the first processing step employs a modified projection matrix-based calibration algorithm to estimate the relevant geometric parameters. Predicted parameters are then fine-tuned in a second processing step by an iterative strategy based on the symmetry property of the sum of projections. Virtual projections calculated from the parameters after two-step processing compensate for the scanning errors and are used for CT reconstruction. Experiments on phantom and mouse imaging data were performed to validate the calibrationalgorithm. Once system correction was conducted, CBCT of a CT bar phantom and a cohort of euthanized mice were analyzed. No obvious structure error or spatial artifacts were observed, validating the accuracy of the proposed geometry calibration method. Digital phantom simulation indicated that compared withthe preset spatial values, errors in the final estimated parameters could be reduced to 0.05° difference in dominant angle and 0.5-pixel difference in dominant axis bias. The in-plane resolution view of the CT-bar phantom revealed that the resolution approaches 150 m. A constrained two-step online geometry calibration algorithm has been developed to calibrate an integrated X-ray imaging system, defined by a first-step analytical estimation and a second-step iterative fine-tuning. Test results have validated its accuracy in system correction, thus demonstrating the potential of the described system to be modified and adapted for preclinicalresearch.

TIGRE-VarianCBCT for on-board cone-beam computed tomography, an open-source toolkit for imaging, dosimetry and clinical research

We presented TIGRE-VarianCBCT, an open-source toolkit Matlab-GPU for Varian on-board cone-beam CT with particular emphasis to address challenges in raw data preprocessing, artifacts correction, tomographic reconstruction and image post-processing. The aim of this project is to provide not only a tool to bridge the gap between clinical usage of CBCT scan data and research algorithms but also a framework that breaks down the imaging chain into individual processes so that research effort can be focused on a specific part. The entire imaging chain, module-based architecture, data flow and techniques used in the creation of the toolkit are presented. Raw scan data are first decoded to extract X-ray fluoro image series and set up the imaging geometry. Data conditioning operations including scatter correction, normalization, beam-hardening correction, ring removal are performed sequentially. Reconstruction is supported by TIGRE with FDK as well as a variety of iterative algorithms. Pixel-to-HU mapping is calibrated by a CatphanTM 504 phantom. Imaging dose in CTDIw is calculated in an empirical formula. The performance was validated on real patient scans with good agreement with respect to vendor-designed program. Case studies in scan protocol optimization, low dose imaging and iterative algorithm comparison demonstrated its substantial potential in performing scan data based clinical studies. The toolkit is released under the BSD license, imposing minimal restrictions on its use and distribution. The toolkit is accessible as a module at https://github.com/CERN/TIGRE.

Training generative adversarial networks for optical property mapping using synthetic image data.

We demonstrate the training of a generative adversarial network (GAN) for the prediction of optical property maps (scattering and absorption) using spatial frequency domain imaging (SFDI) image data sets that are generated synthetically with a free open-source 3D modelling and rendering software, Blender. The flexibility of Blender is exploited to simulate 5 models with real-life relevance to clinical SFDI of diseased tissue: flat samples containing a single material, flat samples containing 2 materials, flat samples containing 3 materials, flat samples with spheroidal tumours and cylindrical samples with spheroidal tumours. The last case is particularly relevant as it represents wide-field imaging inside a tubular organ e.g. the gastro-intestinal tract. In all 5 scenarios we show the GAN provides an accurate reconstruction of the optical properties from single SFDI images with a mean normalised error ranging from 1.0-1.2% for absorption and 1.1%-1.2% for scattering, resulting in visually improved contrast for tumour spheroid structures. This compares favourably with the ∼10% absorption error and ∼10% scattering error achieved using GANs on experimental SFDI data. Next, we perform a bi-directional cross-validation of our synthetically-trained GAN, retrained with 90% synthetic and 10% experimental data to encourage domain transfer, with a GAN trained fully on experimental data and observe visually accurate results with an error of 6.3%-10.3% for absorption and 6.6%-11.9% for scattering. Our synthetically trained GAN is therefore highly relevant to real experimental samples but provides the significant added benefits of large training datasets, perfect ground-truths and the ability to test realistic imaging geometries, e.g. inside cylinders, for which no conventional single-shot demodulation algorithms exist. In the future, we expect that the application of techniques such as domain adaptation or training on hybrid real-synthetic datasets will create a powerful tool for fast, accurate production of optical property maps for real clinical imaging systems.

Machine-Learning-Inspired Workflow for Camera Calibration.

The performance of modern digital cameras approaches physical limits and enables high-precision measurements in optical metrology and in computer vision. All camera-assisted geometrical measurements are fundamentally limited by the quality of camera calibration. Unfortunately, this procedure is often effectively considered a nuisance: calibration data are collected in a non-systematic way and lack quality specifications; imaging models are selected in an ad hoc fashion without proper justification; and calibration results are evaluated, interpreted, and reported inconsistently. We outline an (arguably more) systematic and metrologically sound approach to calibrating cameras and characterizing the calibration outcomes that is inspired by typical machine learning workflows and practical requirements of camera-based measurements. Combining standard calibration tools and the technique of active targets with phase-shifted cosine patterns, we demonstrate that the imaging geometry of a typical industrial camera can be characterized with sub-mm uncertainty up to distances of a few meters even with simple parametric models, while the quality of data and resulting parameters can be known and controlled at all stages.

A novel QA phantom based on scintillating fiber ribbons with implementation of 2D dose tomography for small-field radiotherapy.

To develop a novel instrument for real-time quality assurance (QA) procedures in radiotherapy. The system implements a scintillation-based phantom and associated signal acquisition and processing modules and aims to monitor two-dimensional (2D) dose distributions of small fields. For the proposed phantom, we have designed and realized a prototype implementing six high-resolution tissue-equivalent scintillating fiber ribbons stacked with in-plane 30° rotated orientations from each other. Each ribbon output is coupled to a silicon photodiode linear array (with an element pitch of 400μm) to detect scintillating signal, which represents the projected irradiation profile perpendicular to the ribbon's orientation. For the system providing six acquired projected dose profiles at different orientations, we have developed a two-step signal processing method to perform 2D dose reconstruction. The first step is to determine irradiation field geometry parameters using a tomographic geometry approach, and the second one is to perform specific penumbra estimation. The QA system prototype has been tested on a Novalis TrueBeam STX with a 6-MV photon beam for small elliptic fields defined by 5- and 10-mm cone collimators and for 10×10- and 20×10-mm2 rectangular fields defined by the micro-multileaf collimator. Gamma index analysis using EBT3 films as reference has been carried out with tight 2%-dose-difference (DD)/700-μm-distance-to-agreement (DTA) as well as 1%-DD/1-mm-DTA criteria for evaluating the system performances. The testing also includes an evaluation of the proposed two-step field reconstruction method in comparison with two conventional methods: filtered back projection (FBP) and simultaneous iterative reconstruction technique (SIRT). The reconstructed 2D dose distributions have gamma index pass rates higher than 95% for all the tested configurations as compared with EBT3 film measurements with both 2%-DD/700-μm-DTA and 1%-DD/1-mm criteria. 2D global gamma analysis shows that the two-step and FBP radiation field reconstruction methods systematically outperform the SIRT approach. Moreover, higher gamma index success rates are obtained with the two-step method than with FBP in the case of the fields defined with the stereotactic cones. The proposed small-field QA system makes a use of six water-equivalent scintillating detectors (fiber ribbons) to acquire dose distribution. The developed two-step signal processing method performs tomographic 2D dose reconstruction. A system prototype has been built and tested using hospital facilities with small rectangular and elliptic fields. Testing results show 2D reconstructed dose distributions with high accuracy and resolution. Such a system could potentially be an alternative approach to film dosimetry for small-field QA, which is still widely used as reference in clinical practice.

Improving image quality in diffuse optical tomography

Diffuse optical tomography (DOT) modality uses diffusion equation solution of low power laser distribution in the imaging tissue. DOT method has three main running mode and three geometric subbranches. Run modes are: Continuous Wave (CW), Time Resolved (TR) and Frequency Domain (FD) modes. These run modes might have transmission-through, back-reflected or ring geometry depend on the source and detector placements on tissue surface. In this work, we tested our novel imaging method on the frequency-domain back-reflected DOT simulation model. Our novelty is: selecting a great number of multi-frequency data in the wide frequency spectrum which was not tested before in the literature. Most of the literature works consist of narrow frequency band with the limited number of frequencies such as maximum 12 multi-frequency-data which cover from 100 up to 700 MHz. In our work, we tested and compared reconstructed inclusion images generated by our 500 wide spectrum multi-frequency data versus reconstructed inclusion images generated by 20 narrow band multi-frequency data which was generally used in FD DOT studies. We observed that our novel imaging method which used wide spectrum multi-frequency data is superior to the traditionally used narrow-band multi-frequency method based on the reconstructed inclusion image localization errors. Since most of the homogenous geometry consists of fat tissue, we selected background absorption coefficient µa = 0.2 cm−1 and tissue scattering coefficient $${\upmu }$$ s = 80 cm−1 for 800 nm laser source wavelength. Three-dimensional cubic imaging simulation media has 75-mm grid sizes in each direction. Two different imaging scenarios were tested. In the first scenario, one inclusion with absorption coefficient µa = 0.7 cm−1 was put inside the three-dimensional imaging geometry at 27 mm depth. In the second scenario one inclusion with absorption coefficient µa = 0.7 cm−1 was embedded in 48 mm depth location. Inclusions are cubic and they have 6 mm xyz length at each direction.

Geometry image super-resolution with AnisoCBConvNet architecture for efficient cloth modeling.

We propose an anisotropic constrained-boundary convolutional neural networks (hereafter, AnisoCBConvNet) that can stably express high-quality meshes without oscillation by applying super-resolution operations to low-resolution cloth meshes. As a training set for the neural network, we use a pair between simulation data of low resolution (LR) cloth and data obtained by applying the same simulation to high resolution (HR) cloth with increased quad mesh resolution of LR cloth. The actual data used for training are 2D geometry images converted from 3D meshes. The proposed AnisoCBConvNet is used to train an image synthesizer that converts LR geometry images to HR geometry images. In particular, by controlling the weights anisotropically near the boundary, the problem of surface wrinkling caused by oscillation is alleviated. When the HR geometry image obtained through AnisoCBConvNet is converted back to the HR cloth mesh, details including wrinkles are expressed better than the input cloth mesh. In addition, our results improved the noise problem in the existing geometry image approach. We tested AnisoCBConvNet-based super-resolution in various simulation scenarios, and confirmed stable and efficient performance in most of the results. By using our method, it will be possible to effectively produce CG VFX created using high-quality cloth simulation in games and movies.

Three-Dimensional Coordinate Extraction Based on Radargrammetry for Single-Channel Curvilinear SAR System

With the rapid development of high-resolution synthetic aperture radar (SAR) systems, the technique that utilizes multiple two-dimensional (2-D) SAR images with different view angles to extract three-dimensional (3-D) coordinates of targets has gained wide concern in recent years. Unlike the traditional multi-channel SAR utilized for 3-D coordinate extraction, the single-channel curvilinear SAR (CLSAR) has the advantages of large variation of view angle, requiring fewer acquisition data, and lower device cost. However, due to the complex aerodynamic configuration and flight characteristics, important issues should be considered, including the mathematical model establishment, imaging geometry analysis, and high-precision extraction model design. In this paper, to address these challenges, a 3-D vector model of CLSAR was presented and the imaging geometries under different view angles were analyzed. Then, a novel 3-D coordinate extraction approach based on radargrammetry was proposed, in which the unique property of the SAR system, called cylindrical symmetry, was utilized to establish a novel extraction model. Compared with the conventional approach, the proposed one has fewer constraints on the trajectory of radar platform, requires fewer model parameters, and can obtain higher extraction accuracy without the assistance of extra ground control points (GCPs). Numerical results using simulated data demonstrated the effectiveness of the proposed approach.

DetecDiv, a generalist deep-learning platform for automated cell division tracking and survival analysis.

Automating the extraction of meaningful temporal information from sequences of microscopy images represents a major challenge to characterize dynamical biological processes. So far, strong limitations in the ability to quantitatively analyze single-cell trajectories have prevented large-scale investigations to assess the dynamics of entry into replicative senescence in yeast. Here, we have developed DetecDiv, a microfluidic-based image acquisition platform combined with deep learning-based software for high-throughput single-cell division tracking. We show that DetecDiv can automatically reconstruct cellular replicative lifespans with high accuracy and performs similarly with various imaging platforms and geometries of microfluidic traps. In addition, this methodology provides comprehensive temporal cellular metrics using time-series classification and image semantic segmentation. Last, we show that this method can be further applied to automatically quantify the dynamics of cellular adaptation and real-time cell survival upon exposure to environmental stress. Hence, this methodology provides an all-in-one toolbox for high-throughput phenotyping for cell cycle, stress response, and replicative lifespan assays.

Defocusing μPTV With Increased Measurement Depth Using Shadowgraphy

In this study we can show that shadowgraphy is a feasible alternative to fluorescence for microscopic particle imaging. Furthermore, using the particle shadow image geometry (diameter for defocusing methods or axis lengths for astigmatism methods), the particle location along the optical axis can be determined, enabling a three-dimensional flow measurement with a single camera. However, the rate of change of the particle shadow image geometry as a function of the distance to the focal plane is one order of magnitude smaller as compared to fluorescence/scattering imaging approaches, such that the measurement depths is effectively increased. This increase is due to the fact that the signal to noise ratio of the relatively small particle shadow images is large, even if the particle is located far away from the focal plane, as the intensity is distributed over a small sensor area. In addition, due to their smaller dimensions, more particle shadow images can be captured on the camera sensor, yielding a larger information density for microscopic particle imaging. A measurement of a micro-channel (channel height: 650 µm) flow showed a good agreement with the ideal parabolic flow profile, proving the viability of this microscopic particle shadow imaging approach as a suitable addition to the existing fluorescence-based methods.

Dense Oil Tank Detection and Classification via YOLOX-TR Network in Large-Scale SAR Images

Oil storage tank detection and classification in synthetic aperture radar (SAR) images play a vital role in monitoring energy distribution and consumption. Due to the SAR side-looking imaging geometry and multibouncing scattering mechanism, dense oil tank detection and classification tasks have faced more challenges, such as overlapping, blurred contours, and geometric distortion, especially for small-sized tanks. To address the above issues, this paper proposes YOLOX-TR, an improved YOLOX based on the Transformer encoder and structural reparameterized VGG-like (RepVGG) blocks, to achieve end-to-end oil tank detection and classification in densely arranged areas of large-scale SAR images. Based on YOLOX, the Transformer encoder, a self-attention-based architecture, is integrated to enhance the representation of feature maps and capture the region of interest of oil tanks in densely distributed scenarios. Furthermore, RepVGG blocks are employed to reparameterize the backbone with multibranch typologies to strengthen the distinguishable feature extraction of multi-scale oil tanks without increasing computation in inference time. Eventually, comprehensive experiments based on a Gaofen-3 1 m oil tank dataset (OTD) demonstrated the effectiveness of the Transformer encoder and RepVGG blocks, as well as the performance superiority of YOLOX-TR with a mAP and mAP0.5 of 60.8% and 94.8%, respectively.

PCGAN: Prediction-Compensation Generative Adversarial Network for Meshes

Unlike natural images, the topology similarity among meshes can hardly be handled with classical deep learning because of their irregular structures. Parameterization provides a way to represent meshes in the form of geometry and normal images, which reflects the correlation between neighboring sample locations. Generative Adversarial Networks (GANs) can efficiently generate images without explicitly computing probability densities of the underlying distribution. However, existing GANs such as Coupled Generative Adversarial Network (CoGAN) generally have two drawbacks: (1) Inability to process unnatural images. (2) Insufficient exploration of the inherent relation between normal and the corresponding geometry image. To address these issues, this paper proposes an efficient method named Prediction-Compensation Generative Adversarial Network (PCGAN) to learn a joint distribution of both geometry and normal images, which aims for generating meshes with two GANs. The consistency of two GANs for the geometry and the normal is guaranteed by utilizing a sequence of prediction-compensation pairs. The sequence can estimate the normal image from the geometry image and compensate the geometry from normal progressively. Particularly, the prediction has a closed-form expression, which provides high estimation accuracy and reduces training complexity. Extensive experimental results on facial mesh generation indicate that our PCGAN outperforms CoGAN and other architectures in retaining the geometry of the faces and in generating realistic face meshes with rich facial attributes such as facial expression and morphology. Moreover, quantitative evaluations demonstrate our superior performance compared with the methods mentioned above.

An Adjusted Frequency-Domain Algorithm for Arc Array Bistatic SAR Data with One-Moving Transmitter.

Arc array synthetic aperture radar (AA-SAR), which can observe the scene in all directions, breaks through the single view of traditional SAR. However, the concealment of AA-SAR is poor. To mitigate this, arc array bistatic SAR (AA-BiSAR) with the moving transmitter is proposed, it has the advantages of good concealment and can expand the imaging scene, and improve the flexibility of the system. The imaging geometry including the signal model is established, and a range frequency-domain algorithm based on keystone transform (KT) is proposed in this paper. In the first step, the slant range equation is approximated by Taylor series expansion to compensate for the residual phase caused by the transmitter motion. In the second step, the range cell migration between the range and azimuth is eliminated through the KT method in the range frequency-domain. In the third step, using the data after range cell migration correction in step 2, an azimuth pulse compression is performed to obtain the focused image. In addition, the spatial resolution of the AA-BiSAR system is analyzed in detail. Finally, three simulation results verify the effectiveness of the proposed algorithm and the change in the spatial resolution.

Graph Attention Network in Microwave Imaging for Anomaly Localization

So far, numerous learned models have been pressed to use in microwave imaging problems. These models however, are oblivious to the imaging geometry. It has always been hard to bake the physical setup of the imaging array into the structure of the network, resulting in a data-intensive models that are not practical. This work put forward a graph formulation of the microwave imaging array. The architectures proposed is made cognizant of the physical setup, allowing it to incorporate the symmetries, resulting in a less data requirements. Graph convolution and attention mechanism is deployed to handle the cases of fully-connected graphs corresponding to multi-static arrays. The model works with a modular fashion at node level to strike a trade-off between flexibility and efficient capture of mutual information present in measured signals. Additionally, the modular working fashion endows the model with immunity to overfitting. The graph-treatment of the problem is evaluated on experimental setup in context of anomaly localization with imaging array and has shown higher performance as compared to the popular radar technique. The thin model was realized with a feasibly procured reasonably-sized dataset in the order of few hundreds, thus eliminating the need to resort to simulations for augmentation.

Deep learning-based individual tree crown delineation in mangrove forests using very-high-resolution satellite imagery

Mangrove forests are vulnerable ecosystems that require broad-scale monitoring. Various solutions based on satellite imagery have emerged for this purpose but still suffer from the lack of methods to accurately delineate individual tree crowns (ITCs). Within-stand variability in crown size and shape, crown clumping and fragmentation, and understory vegetation hamper the delineation in these ecosystems. To cope with these factors, the proposed method combines a deep learning-based enhancement of ITCs with a marker-controlled watershed segmentation algorithm. The MT-EDv3 neural network is employed to compute the normalized Euclidean distance of crown pixels to treetops and a Laplacian of Gaussian filter is applied to the resulting image to enhance crown borders before segmentation. The method was applied to WorldView imagery over four mangrove sites worldwide and compared to previously published methods using standardized metrics. Accurate detection (Overall Accuracy ≥ 0.93 and Kappa ≥ 0.87) and area estimation (R2 ≥ 0.66, NRMSE ≤ 12%) of crowns was achieved for all sites using either the panchromatic band or a combination of the pan-sharpened visible-near-infrared bands. Based on Precision, Recall, and F1-score, the proposed method outperformed previous watershed segmentation and software-based algorithms of crown delineation, as well as the Mask R-CNN segmentation framework. The viewing geometry of images and the forest heterogeneity were identified as important contributors to the delineation accuracy. This study is the first to achieve accurate delineation of ITCs in mangrove forests across sites, opening perspectives of applications to satellite-based monitoring. The method shows promising transferability to other very-high-resolution satellite sensors as well as to aerial and unmanned aerial vehicle imagery and could be improved by including more spectral information and LiDAR-derived canopy height models.

SAR Image Matching Based on Local Feature Detection and Description Using Convolutional Neural Network

Feature detection is a vital step for the image registration process whose target is the misalignment correction among images to increase the convergency level. Deep learning (DL) in remote sensing has become a worldwide sensation. Despite its huge potential, DL has not reached its intended target concerning the applications of Synthetic Aperture Radar (SAR) images. In this study, we focus on matching SAR images using a Convolutional Neural Network. The big challenge in this study is how to modify a pretrained Visual Geometry Group model based on the multispectral dataset to act as a SAR image feature detector where it does not require any prior knowledge about the nature of the SAR feature. Since SAR images have different characteristics from optical images such as SAR dynamic range and imaging geometry, some problems arise and should be considered during the matching process. Despite all these difficulties, results demonstrate the robustness of the registration process where it can provide descriptors that preserve the localization data of features. Also, the proposed approach provides reasonable results compared to the state-of-the-art methods and outperforms the correlation approach and ORB descriptor under scaling. In addition, it may be considered an end-to-end image matching tool for SAR images, although the calculations of fine matching parameters are included.

EPIPOLAR GEOMETRY BETWEEN PHOTOGRAMMETRY AND COMPUTER VISION – A COMPUTATIONAL GUIDE

Abstract. Stereo image orientation is one of the major topics in computer vision, photogrammetry, and robotics. The stereo vision problem solution represents the basic element of the multi-view Structure from Motion SfM in computer vision and photogrammetry.A successfully reconstructed stereo image geometry is based on solving the epipolar constraint using the fundamental matrix which is based on the projective geometry in computer vision. However, in photogrammetry, the problem is well known as relative orientation and there is a different solution that is based on the euclidean geometry using collinearity or coplanarity equations.A lot of literature and discussions were found in the last decades to solve the epipolar geometry problem. However, there is still no clear description to compare between solutions introduced using both projective and euclidean solutions and which method of the relative image orientation is mostly preferred.To the best of our knowledge, computing and plotting the epipolar lines using photogrammetric collinearity and coplanarity equations is not shown before in the educational litrature. In this paper, a detailed mathematical solution of the epipolar geometry will be shown using both photogrammetric and computer vision techniques. This is aimed to remove any confusion for new learners in using the current methods in both scientific fields and show that using any technique should lead to comparable results with advantages and disadvantages.