Optical Photogrammetry Research Articles

Synthetic cranial MRI from 3D optical surface scans using deep learning for radiation therapy treatment planning

BackgroundOptical scanning technologies are increasingly being utilised to supplement treatment workflows in radiation oncology, such as surface-guided radiotherapy or 3D printing custom bolus. One limitation of optical scanning devices is the absence of internal anatomical information of the patient being scanned. As a result, conventional radiation therapy treatment planning using this imaging modality is not feasible. Deep learning is useful for automating various manual tasks in radiation oncology, most notably, organ segmentation and treatment planning. Deep learning models have also been used to transform MRI datasets into synthetic CT datasets, facilitating the development of MRI-only radiation therapy planning.AimsTo train a pix2pix generative adversarial network to transform 3D optical scan data into estimated MRI datasets for a given patient to provide additional anatomical data for a select few radiation therapy treatment sites. The proposed network may provide useful anatomical information for treatment planning of surface mould brachytherapy, total body irradiation, and total skin electron therapy, for example, without delivering any imaging dose.MethodsA 2D pix2pix GAN was trained on 15,000 axial MRI slices of healthy adult brains paired with corresponding external mask slices. The model was validated on a further 5000 previously unseen external mask slices. The predictions were compared with the “ground-truth” MRI slices using the multi-scale structural similarity index (MSSI) metric. A certified neuro-radiologist was subsequently consulted to provide an independent review of the model’s performance in terms of anatomical accuracy and consistency. The network was then applied to a 3D photogrammetry scan of a test subject to demonstrate the feasibility of this novel technique.ResultsThe trained pix2pix network predicted MRI slices with a mean MSSI of 0.831 ± 0.057 for the 5000 validation images indicating that it is possible to estimate a significant proportion of a patient’s gross cranial anatomy from a patient’s exterior contour. When independently reviewed by a certified neuro-radiologist, the model’s performance was described as “quite amazing, but there are limitations in the regions where there is wide variation within the normal population.” When the trained network was applied to a 3D model of a human subject acquired using optical photogrammetry, the network could estimate the corresponding MRI volume for that subject with good qualitative accuracy. However, a ground-truth MRI baseline was not available for quantitative comparison.ConclusionsA deep learning model was developed, to transform 3D optical scan data of a patient into an estimated MRI volume, potentially increasing the usefulness of optical scanning in radiation therapy planning. This work has demonstrated that much of the human cranial anatomy can be predicted from the external shape of the head and may provide an additional source of valuable imaging data. Further research is required to investigate the feasibility of this approach for use in a clinical setting and further improve the model’s accuracy.

Displacement Analysis of Large-Scale Robotic Arm for Printing Cement Mortar Using Photogrammetry

The development of specialised equipment for three-dimensional printing of cement mortar requires the deployment of advanced design methods. The accuracy of printing robotic arms is influenced by the change in the position of the end effector, which is influenced by the stiffness of the arm, or deformation of parts of the arm and yielding in the place of rotation axes. Determining the actual change in the end effector position is often a difficult challenge. In this paper, we analysed the displacement on a large robotic arm by the non-contact optical photogrammetry method. We applied this method to a specialised 2.8 m long robotic arm SCARA with an added rotational axis. We compared the results from photogrammetry with the results from measurements with a mechanical deflection meter, and with the predicted displacement values from the FEM simulation. The results from both measurement methods showed maximum deviations of hundredths of a mm. The findings of the analysis thus indicate that photogrammetry meets the strict requirements for displacement measurement on a robotic arm for the 3D printing of cement mortar. A significant advantage of the method is the possibility of measuring almost all attainable arm positions and achieving results in hundreds of places.

Gapless-REMA100: A gapless 100-m reference elevation model of Antarctica with voids filled by multi-source DEMs

The digital elevation model (DEM) of Antarctica is a fundamental dataset for a wide range of glaciological applications, from fieldwork planning to ice sheet dynamics analysis. The DEM data at high spatial resolution provides a more detailed depiction of the topography. The reference elevation model of Antarctica (REMA) mosaic is a recently released high-resolution Antarctic DEM product with a high absolute vertical accuracy of 1 m or less. The REMA mosaic was generated using an optical photogrammetry technique and can reflect the surface elevation of the ice sheet. However, the REMA mosaic has a large number of data voids of various spatial sizes, which affects its wide glaciological application. Therefore, we propose the generation of a gapless 100-m reference elevation model of Antarctica (Gapless-REMA100) with the voids filled by combining multi-source DEMs. In this study, the best-fit fill DEMs were automatically selected between almost all available circum-Antarctica or regional DEMs. Targeting numerous voids in large areas of the 100-m REMA mosaic, we propose a novel hierarchical delta surface method for interpolating the created delta surface to achieve an efficient, unbiased, and continuous filled surface. High-precision laser altimetry data were used to evaluate the vertical accuracy of the REMA mosaic. To eliminate the influence of the temporal surface elevation change and penetration depth between the DEM and laser altimetry data, we developed a new local contrast evaluation method by calculating the vertical accuracy of the equal-area buffer as a control to that of the center void. The experimental results with the Ice, Cloud and Land elevation Satellite-2 (ICESat-2) laser altimetry data validate that the filled 100-m REMA mosaic of voids can reach an elevation accuracy similar to that of the original REMA dataset for the west and east Antarctic ice sheet as well as the entire continent. In the Antarctic Peninsula area, the vertical accuracy of the filled REMA mosaic by the proposed method is superior to that of the officially released version of the Polar Geospatial Center. This is the first time that a gapless, seamless, and accurate 100-m REMA mosaic has been released which has the potential to be extensively used in glaciological applications. The proposed methods can also be used in void-filling of different types of DEM products and their elevation accuracy evaluations.

Forest Degradation Assessment Using UAV Optical Photogrammetry and SAR Data

The sequestration of carbon in the forest is very high after soil and ocean; this is one of the major factors which makes monitoring of forests very much essential. The traditional inventory techniques were very much exhaustive and needed encroachment into the inaccessible forest area, which create lot of difficulties to foresters and other researchers, that is why, most of the areas of intense forests are untouched for monitoring purposes. The degradation of the forest area is highly prominent in the hilly terrain; activities such as lopping and forest fire are very common in these areas. In this research, UAV optical photogrammetric technique is used for the extraction of canopy metrics as well as for the generation of DSM and DTM, which is further used for the validation of the results obtained with the Sentinel-2 and Sentinel-1A data. The Sentinel-1A of C-band data is used for the generation of backscatter image, texture images. Sentinel-2 is used for the calculation of various vegetation-based indices for the evaluation of the proficiency of the degradation in the forest area. On the basis of various vegetation indices and texture images calculated, the degradation of the forest can be quantified. The random forest classifier is used for the classification of the forest into degraded and non-degraded classes. The classification accuracy obtained for three classes, i.e. degraded, non-degraded, non-forest is 0.86 and precision is 0.75. The correlation is 0.72 for the classes, whereas error rate is found to be 0.13. Hence, the integration of UAV optical photogrammetry and SAR data can give much appreciated results; also, this technique is very much helpful in monitoring and management of inaccessible forest areas.

Remote Sensing for Assessing Landslides and Associated Hazards

Multi-platform remote sensing using space-, airborne and ground-based sensors has become essential tools for landslide assessment and disaster-risk prevention. Over the last 30 years, the multiplicity of Earth Observation satellites mission ensures uninterrupted optical and radar imagery archives. With the popularization of Unmanned Aerial Vehicles, free optical and radar imagery with high revisiting time, ground and aerial possibilities to perform high-resolution 3D point clouds and derived digital elevation models, it can make it difficult to choose the appropriate method for risk assessment. The aim of this paper is to review the mainstream remote-sensing methods commonly employed for landslide assessment, as well as processing. The purpose is to understand how remote-sensing techniques can be useful for landslide hazard detection and monitoring taking into consideration several constraints such as field location or costs of surveys. First we focus on the suitability of terrestrial, aerial and spaceborne systems that have been widely used for landslide assessment to underline their benefits and drawbacks for data acquisition, processing and interpretation. Several examples of application are presented such as Interferometry Synthetic Aperture Radar (InSAR), lasergrammetry, Terrestrial Optical Photogrammetry. Some of these techniques are unsuitable for slow moving landslides, others limited to large areas and others to local investigations. It can be complicated to select the most appropriate system. Today, the key for understanding landslides is the complementarity of methods and the automation of the data processing. All the mentioned approaches can be coupled (from field monitoring to satellite images analysis) to improve risk management, and the real challenge is to improve automatic solution for landslide recognition and monitoring for the implementation of near real-time emergency systems.

Extracting Three-dimensional Information from SEM Images by Means of Photogrammetry

Optical photogrammetry software has been applied to scanning electron microscope images to obtain 3D models and quantitative data on carbon particles and graphite nanoparticles. Image acquisition has been automated by the use of external macro software. Mesh data has been reconstructed from a number of images taken at multiple sample stage tilt angles with a supplementary measurement of TEM grids for method validation. This 3D model has been scaled and processed to obtain values such as the volume, height and surface area of the samples and has been quantitatively compared to 2D measurements.

Application of optical photogrammetry in radiation oncology: HDR surface mold brachytherapy

PurposeWe propose a novel method of designing surface mold brachytherapy applicators using optical photogrammetry. The accuracy of this technique for the purpose of 3D-printing surface mold brachytherapy applicators is investigated. Methods and MaterialsPhotogrammetry was used to generate a 3D model of a patient's right arm. The geometric accuracy of the model was evaluated against CT in terms of volume, surface area, and the Hausdorff distance. A surface mold applicator was then 3D printed using this reconstructed model. The accuracy was evaluated by analyzing the displacement and air-gap volumes between the applicator and plaster cast on a CT image. This technique was subsequently applied to generate a 3D-printed applicator of the author's hand directly, as a proof of principle, using only photographic images. ResultsThe volume and surface area of the model were within 0.1% and 2.6% of the CT-obtained values, respectively. Using the Hausdorff distance metric, it was determined that 93% of the visible vertices present in the CT-derived model had a matching vertex on the photogrammetry-derived model within 1 mm, indicating a high level of similarity. The maximum displacement between the plaster cast of the patient's arm and the photo-derived 3D-printed applicator was 1.2 mm with a total air-gap volume of approximately 0.05 cm3. ConclusionsPhotogrammetry has been applied to the task of generating 3D-printed brachytherapy surface mold applicators. The current work demonstrates the feasibility and accuracy of this technique and how it may be incorporated into a 3D-printing brachytherapy workflow.

Focus stacking photogrammetry for micro‐scale roughness reconstruction: a methodological study

AbstractIn close‐range applications, it is unclear whether optical photogrammetry is capable of accurately reconstructing submillimetre‐scale roughness. This paper presents a study of fine‐scale rock fracture roughness measurements with a careful assessment of the error. The workflow combines the techniques of structure from motion with focus stacking, using consumer‐grade equipment and free or affordable software. The approach is tested firstly with synthetic data to check the influence of the number and position of cameras, object texture and image processing using focus stacking. Secondly, the optimised workflow is used to measure a natural shale rock fracture surface. To estimate the accuracy, the results were compared with a high‐precision reference dataset provided by white‐light interferometry. The standard deviation of error in the method is 6·5 μm, and is related with morphological structures with wavelengths below 150 μm and amplitudes smaller than 10 μm.

Near Field Scan Alignment Procedure for Electrically Large Apertures

Computational imaging at microwave frequencies has gained traction due to its potential for obtaining high-quality images with fast acquisition rates. Complex and diverse radiation patterns form the cornerstone of this approach. Electrically large antennas, such as mode-mixing cavities and metamaterial apertures, have proven to be effective platforms for generating such waveforms. Due to the complex nature of these antennas, near field scanning is often required to characterize their radiation patterns. However, accurate knowledge of the produced waveforms’ spatial distribution, with respect to the physical position of the antenna, is imperative. This relies on precise alignment between the antenna and the near field scan stage during the characterization process—a requirement that is especially cumbersome to achieve when operating at high frequencies. We present an effective method to address this problem; by introducing RF markers into the antenna the position of the antenna under test within the near field scanning setup can be obtained directly from the measurements. The proposed method is experimentally verified through comparison with measurements made using optical photogrammetry. The proposed process will find application in the alignment of computational and multistatic imaging systems, commonly used in security screening and threat detection, as well as in tiled electrically large antenna structures.

3D reconstruction of SEM images by use of optical photogrammetry software

Reconstruction of the three-dimensional (3D) surface of an object to be examined is widely used for structure analysis in science and many biological questions require information about their true 3D structure. For Scanning Electron Microscopy (SEM) there has been no efficient non-destructive solution for reconstruction of the surface morphology to date. The well-known method of recording stereo pair images generates a 3D stereoscope reconstruction of a section, but not of the complete sample surface. We present a simple and non-destructive method of 3D surface reconstruction from SEM samples based on the principles of optical close range photogrammetry. In optical close range photogrammetry a series of overlapping photos is used to generate a 3D model of the surface of an object. We adapted this method to the special SEM requirements. Instead of moving a detector around the object, the object itself was rotated. A series of overlapping photos was stitched and converted into a 3D model using the software commonly used for optical photogrammetry. A rabbit kidney glomerulus was used to demonstrate the workflow of this adaption. The reconstruction produced a realistic and high-resolution 3D mesh model of the glomerular surface. The study showed that SEM micrographs are suitable for 3D reconstruction by optical photogrammetry. This new approach is a simple and useful method of 3D surface reconstruction and suitable for various applications in research and teaching.

Target Detection and Tracking of moving objects for characterizing landslide displacements from time-lapse terrestrial optical images

Terrestrial Optical Photogrammetry (TOP) is a low-cost monitoring technique that is commonly used in change detection studies. For landslide monitoring, image correlation techniques are frequently used to generate Digital Surface Models (DSMs) from stereo-pairs or to measure the 2D surface displacement field from single-view time-lapse sequences. Image correlation allows for detecting spatially continuous displacement fields at a sub-pixel precision. The technique, however, has several limitations for assessing displacement because 1) of its sensitiveness to changes in texture, shape and radiometry in the image pile, 2) of the need of spatially regular sampling grids, and 3) of possible high computation time that can impede the processing of large image datasets.To address these limitations, an alternative and complementary approach based on a Target Detection and Tracking (TDT) algorithm is proposed for a rapid calculation of the displacement of targets in image time series. The TDT code, developed as a MATLAB-based tool, is able to track natural or man-made targets. The precision of the TDT code is assessed using several image time series acquired at the Super-Sauze landslide (Southern French Alps) and compared to ground based measurements. The computed relative accuracy is between 10−3 and 10−4 (5cm at a distance of 115m). Although the TDT approach does not provide spatially continuous information, it provides 1) a quantification of the object displacements at the same order of precision as image correlation (sub-pixel accuracy) and 2) information in regions where image correlation fails because of too large ground displacements. A sensitivity analysis reveals that the major sources of uncertainty are camera movement and/or lens distortion and not the TDT method itself.

Polyethylene damage and deformation on fixed-bearing, non-conforming unicondylar knee replacements corresponding to progressive changes in alignment and fixation

Background Deviations from nominal alignment of unicondylar knee replacements impact knee biomechanics, including the load and stress distribution at the articular contact surfaces. This study characterizes relationships between the biomechanical environment, distinguished by progressive changes in alignment and fixation, and articular damage and deformation in a consecutive series of retrieved unicondylar knee replacements. Methods Twenty seven fixed-bearing, non-conforming unicondylar knee replacements of one design were retrieved after 2 to 13 years of in vivo function. The in vivo biomechanical environment was characterized by grading component migration measured from full-length radiographs and grading component fixation based on intraoperative manual palpation. Articular damage patterns and linear deformation on the polyethylene inserts were measured using optical photogrammetry and contact point digitization. Findings Articular damage patterns and surface deformation on the explanted polyethylene inserts corresponded to progressive changes in component alignment and fixation. Component migration produced higher deformation rates, whereas loosening contributed to larger damage areas but lower deformation rates. Migration and loosening of the femoral component, but not the tibial component, were factors contributing to large regions of abrasion concentrated on the articular periphery. Interpretation Classifying component migration and fixation at revision proved useful for distinguishing common biomechanical conditions associated with the varied polyethylene damage patterns and linear deformation for this fixed-bearing, non-conforming design. Pre-clinical evaluations of unicondylar knee replacements that are capable of reproducing variations in clinical alignment and predicting the observed wear mechanisms are necessary to better understand the impact of knee biomechanics and design on unicondylar knee replacement longevity.

Interpretation of amplitude data from a ground-based radar in combination with terrestrial photogrammetry and visual observations for calving monitoring of Kronebreen, Svalbard

AbstractA ground-based radar has been used successfully for monitoring calving events and velocities at Kronebreen, Svalbard, for two test seasons in 2007 and 2008. We use daily terrestrial optical photogrammetry and continuous visual observation to validate the interpretation of a 116 hour ground-based radar amplitude of return signal data recorded from 26 to 30 August 2008. The radar was placed –4 km from the glacier. It measured at high temporal rate (2 Hz) and the antenna lobe covered a width of –700 m of the front. The calving-front geometry was extracted from the optical images, and its effect on radar backscatter, together with the movement of the glacier, was identified in the plot of the amplitude of the radar return signal. Calving events were detected applying an automated change-detection technique to the radar dataset. This technique allowed us to detect 92% of the events that were observed during the same time. In this paper, we focus on the method rather than on data interpretation. However, future use of this method, combined with meteorological data, tides and ocean temperature, will be valuable for calving-process studies.

Resection methodology for PSP data processing: Recent experiences in NAL

PSP data processing, which primarily involves image alignment and image analysis, is a crucial element in obtaining accurate PSP results. There are two broad approaches to image alignment: the algebraic transformation technique, often called image-warping technique, and resection methodology, which uses principles of optical photogrammetry. Resection is found to have higher resolution, especially when large gradients in pressure or significantly three-dimensional regions have to be resolved, and even with flat models when the camera angles are large. A processing software based on these methodologies has been successfully developed and validated and is currently in use at the Experimental Aerodynamics Division of the National Aerospace laboratories, Bangalore. In this paper, we show the merits of resection methodology through two examples: (a) a wing-body model in transonic flow (M = 0.8) and (b) a simple delta wing in low supersonic flow (M = 1.8). The PSP system utilized for both the cases involve Optrod-B1 paint, a specially designed UV lamp for excitation and two scientific grade CCD cameras for imaging. Typical results are shown using both the algebraic transformation approach and resection methodology.

Ground-based radar interferometry for landslides monitoring: atmospheric and instrumental decorrelation sources on experimental data

The application of ground-based radar interferometry for landslide monitoring is analyzed: a case study based on an experimental campaign carried out in Italy during 2002 is discussed. Interferometric data obtained from coherent synthetic aperture radar (SAR) images acquired by means of C-band ground-based equipment are analyzed. The campaign was aimed at retrieving potential terrain movements of a small landslide observed hundreds of meters away. Critical aspects related to spatial and temporal decorrelation are discussed: the use of optical photogrammetry as a technique for evaluating mechanical stability and correcting geometric distortion is presented. Results also confirmed that the application of ground-based radar interferometry can be attractive and effective if the acquired SAR images maintain an adequate coherence on different dates.