Physical Sensor Model Research Articles

Replacement Sensor Model Generation for a Long High-Resolution Satellite Image Strip

Many high-resolution satellite images are acquired in the strip mode, in which the imaging direction is aligned with the satellite trajectory. A long strip image is often sliced into several scenes before delivery, and customers preprocess each scene with auxiliary data before their applications. Sensor model information is often provided for each scene in the form of rational polynomial coefficients (RPCs). The original RPCs of each scene are erroneous, and should be bias-compensated using ground reference data or ground control points. In this study, we propose the use of single RPCs for long image strips. However, a simple replacement of the physical sensor model using RPCs for the entire strip may cause large errors, because RPCs do not model large satellite attitude changes. Therefore, we applied the fitting of the satellite attitude to a linear equation before RPCs generation for a long-strip image. The test was conducted for three Kompsat-3A image strips consisting of multiple scenes, where one strip shows large satellite attitude changes. From the experiments, the RPCs generated for the whole image strip showed large errors compared to the strip adjustment; however, the proposed method could reduce the errors to an accuracy comparable to that of scene-based RPCs.

Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging: Theory, algorithms, and applications

Plug-and-Play Priors (PnP) is one of the most widely-used frameworks for solving computational imaging problems through the integration of physical models and learned models. PnP leverages high-fidelity physical sensor models and powerful machine learning methods for prior modeling of data to provide state-of-the-art reconstruction algorithms. PnP algorithms alternate between minimizing a data-fidelity term to promote data consistency and imposing a learned regularizer in the form of an image denoiser. Recent highly-successful applications of PnP algorithms include bio-microscopy, computerized tomography, magnetic resonance imaging, and joint ptycho-tomography. This article presents a unified and principled review of PnP by tracing its roots, describing its major variations, summarizing main results, and discussing applications in computational imaging. We also point the way towards further developments by discussing recent results on equilibrium equations that formulate the problem associated with PnP algorithms.

Evaluation Methodology for Physical Radar Perception Sensor Models Based on On-Road Measurements for the Testing and Validation of Automated Driving

In recent years, verification and validation processes of automated driving systems have been increasingly moved to virtual simulation, as this allows for rapid prototyping and the use of a multitude of testing scenarios compared to on-road testing. However, in order to support future approval procedures for automated driving functions with virtual simulations, the models used for this purpose must be sufficiently accurate to be able to test the driving functions implemented in the complete vehicle model. In recent years, the modelling of environment sensor technology has gained particular interest, since it can be used to validate the object detection and fusion algorithms in Model-in-the-Loop testing. In this paper, a practical process is developed to enable a systematic evaluation for perception–sensor models on a low-level data basis. The validation framework includes, first, the execution of test drive runs on a closed highway; secondly, the re-simulation of these test drives in a precise digital twin; and thirdly, the comparison of measured and simulated perception sensor output with statistical metrics. To demonstrate the practical feasibility, a commercial radar-sensor model (the ray-tracing based RSI radar model from IPG) was validated using a real radar sensor (ARS-308 radar sensor from Continental). The simulation was set up in the simulation environment IPG CarMaker® 8.1.1, and the evaluation was then performed using the software package Mathworks MATLAB®. Real and virtual sensor output data on a low-level data basis were used, which thus enables the benchmark. We developed metrics for the evaluation, and these were quantified using statistical analysis.

Data and Image Prior Integration for Image Reconstruction Using Consensus Equilibrium

Image domain prior models have been shown to improve the quality of reconstructed images, especially when data are limited. Pre-processing of raw data, through the implicit or explicit inclusion of data domain priors have separately also shown utility in improving reconstructions. In this work, a principled approach is presented allowing the unified integration of both data and image domain priors for improved image reconstruction. The consensus equilibrium framework is extended to integrate physical sensor models, data models, and image models. In order to achieve this integration, the conventional image variables used in consensus equilibrium are augmented with variables representing data domain quantities. The overall result produces combined estimates of both the data and the reconstructed image that is consistent with the physical models and prior models being utilized. The prior models used in both domains in this work are created using deep neural networks. The superior quality allowed by incorporating both data and image domain prior models is demonstrated for two applications: limited-angle CT and accelerated MRI. The prior data model in both these applications is focused on recovering missing data. Experimental results are presented for a 90 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sup> limited-angle tomography problem from a real checked-bagged CT dataset and a 4× accelerated MRI problem on a simulated dataset. The new framework is very flexible and can be easily applied to other computational imaging problems with imperfect data.

Nonlinear Systematic Distortions Compensation in Satellite Images Based on an Equivalent Geometric Sensor Model Recovered From RPCs

The rational polynomial function is widely accepted as the preferred sensor model for high-resolution satellite imagery (HRSI). However, satellite images and their associated rational polynomial coefficients (RPCs) often suffer from nonlinear systematic errors which were caused by attitude oscillation, sensor deformation, and many other imperfect calibration errors, thus affecting the geo-referencing accuracy. Instead of modeling the biases by polynomials in the image space or refining RPCs directly, this study proposes an approach of going back to the physical model and correcting the local distortions in a self-calibration block adjustment. The algorithm of an equivalent geometric sensor model (EGSM) recovery from RPCs is described in detail. As an equivalent form of the physical sensor models, EGSM reflects the complete viewing geometry of push-broom HRSI. The interior and exterior orientation parameters of EGSM can be stably recovered from RPCs without using any metadata. An approach of RPCs refinement by self-calibration block adjustment based on EGSM is introduced. This approach can effectively compensate for the nonlinear systematic errors caused by platforms and sensors similar to the approach of a rigorous sensor model. The performance of EGSM-based block adjustment is compared with the RFM-based bias compensation method. Experiments using ZY-3 images show the EGSM-based approach can effectively eliminate the nonlinear distortions in satellite images caused by sensor deformation and attitude vibrations. Furthermore, experiments using images from various satellites show that the original RFM can be well fitted with the EGSM and the residuals are smaller than 0.1 pixels for all test images.

Estimation and improvement in the geolocation accuracy of rational polynomial coefficients with minimum GCPs using KOMPSAT-3A

ABSTRACT In this paper, we propose a method to regenerate Rational Polynomial Coefficients (RPCs) using KOMPSAT-3A imagery and to reduce the geolocation error using minimum ground control points (GCPs). To estimate the new RPCs, the physical sensor model fitted to KOMPSAT-3A imagery was utilized and virtual GCPs over the study area were created. The size of the virtual grid used was 20x20x20. To remove the sensor-related errors in physical sensor model, three different image correction models (image coordinate translation model, shift and drift model, and affine transformation model) were additionally applied. We evaluated our proposed method in two areas within Korea, one in urban (Seoul) and one in rural (Goheung) areas. The results showed that there was a significant improvement after applying the suggested approach in the two areas. The image coordinate translation model is suggested in terms of GCP requirement and expected errors estimated from the error propagation analysis using Gauss–Markov Model (GMM).

Toward a Standardized Encoding of Remote Sensing Geo-Positioning Sensor Models

Geolocation information is an important feature of remote sensing image data that is captured through a variety of passive or active observation sensors, such as push-broom electro-optical sensor, synthetic aperture radar (SAR), light detection and ranging (LIDAR) and sound navigation and ranging (SONAR). As a fundamental processing step to locate an image, geo-positioning is used to determine the ground coordinates of an object from image coordinates. A variety of sensor models have been created to describe geo-positioning process. In particular, Open Geospatial Consortium (OGC) has defined the Sensor Model Language (SensorML) specification in its Sensor Web Enablement (SWE) initiative to describe sensors including the geo-positioning process. It has been realized using syntax from the extensible markup language (XML). Besides, two standards defined by the International Organization for Standardization (ISO), ISO 19130-1 and ISO 19130-2, introduced a physical sensor model, a true replacement model, and a correspondence model for the geo-positioning process. However, a standardized encoding for geo-positioning sensor models is still missing for the remote sensing community. Thus, the interoperability of remote sensing data between application systems cannot be ensured. In this paper, a standardized encoding of remote sensing geo-positioning sensor models is introduced. It is semantically based on ISO 19130-1 and ISO 19130-2, and syntactically based on OGC SensorML. It defines a cross mapping of the sensor models defined in ISO 19130-1 and ISO 19130-2 to the SensorML, and then proposes a detailed encoding method to finalize the XML schema (an XML schema here is the structure to define an XML document), which will become a profile of OGC SensorML. It seamlessly unifies the sensor models defined in ISO 19130-1, ISO 19130-2, and OGC SensorML. By enabling a standardized description of sensor models used to produce remote sensing data, this standard is very promising in promoting data interoperability, mobility, and integration in the remote sensing domain.

THE REMOTE SENSING IMAGE GEOMETRICAL MODEL OF BP NEURAL NETWORK

Abstract. Imagery geometry models (IGMs) of the high-resolution satellite images (HRSIs) are always of great interest in the photogrammetry and remote sensing community for the raising new kinds of sensors and imaging systems. Especially the generalized sensor models (GSMs) have been widely used for positioning of satellite images, and the accuracy are already validated. Since Back propagation (BP) neural network is a better choice for the two key reasons of the replacement of physical sensor models by generalized sensor models, numerous mathematical estimations for every specialized sensor, and secret equations of the IGMs. Experiments are carried out to test the approximation accuracy of the new generalized sensor model. And the experimental results show that, the BP neural network is of extremely high accuracy for satellite imagery photogrammetric restitution.

Sensor orientation of HaiYang-1C ultraviolet imager based on a piecewise rational function model

Due to the large field angle and large geometric distortions of the HaiYang-1C ultraviolet imager (UVI), a single rational function model (SRFM) is unable to fully absorb the geometric distortions. The SRFM-based sensor orientation accuracy of the UVI images is thereby worse than expected. In order to improve the sensor orientation accuracy, a feasible sensor orientation method for the UVI images based on a piecewise rational function model (PRFM) is proposed. A complete UVI image is first logistically divided into five subimages, and the adjacent subimages have an overlap. Then, an SRFM is used to mathematically fit the physical sensor model of each subimage. Finally, the five SRFMs form a continuous PRFM, and the PRFM-based sensor orientation of the UVI images is performed. Three HaiYang-1C UVI images were tested. The experimental results showed that the unabsorbed geometric distortions fully propagated into the SRFM-based sensor orientation results. The orientation accuracy of the three images reached ∼7 pixels. In the PRFM-based sensor orientation, both the PRFM fitting errors and the inconsistent errors between the adjacent subimages could be negligible. The PRFM-based orientation accuracy was thereby noticeably improved and reached >1 pixel.

Accuracy and Efficiency: The Comparison of Different RPC Parameters Solving Methods

As a generalized sensor, the RPC model with its accuracy equally matches the physical sensor model. Moreover, the accurate positioning combining with the flexibility in application leads the RPC model to be the priority in photogrammetry processing. Generally, the RPC model is calculated through a control grid. Different RPC parameters solving methods and the operation efficiency all serve as variables in the accuracy of the model. In this paper, the ridge estimation iterative method, spectrum correction iteration, and conjugate gradient method are employed to solve RPC parameters; the accuracy and efficiency of three solving methods are analyzed and compared. The results show that ridge estimation iterative method and spectrum correction iteration have obvious advantages in accuracy. The ridge estimation iterative method has fewer iteration times and time con-sumption, and spectrum correction iteration has more stable precision.

Evaluation of Rational Function Model for High-Resolution KOMPSAT-5 SAR Imagery with Different Sizes of Virtual Grid and Reference Coordinate Systems

When a new spaceborne imaging mission is launched and operated, the generation of the rational function model (RFM), which simplifies the geometric relationship between the ground targets and image coordinates, is one of the important mechanisms to unify the image data with the other geospatial datasets. In this paper, the performance of the generated RFM for the KOMPSAT-5 satellite imagery was evaluated. High-resolution KOMPSAT-5 images whose spatial resolutions are about 1 meter were used for the experiments. During the RFM generation process, the general least squares adjustment and the Tikhonov regularization with the L-curve optimization were applied to initialize and optimize the RFM coefficients. From the physical sensor model for KOMPSAT-5, virtual grids were created and utilized as the control points for generating the RFM. Furthermore, appropriate sizes of virtual grids and the number of elevation layers were proposed. As a result of the experiments, an RFM having a precision better than 10−9 pixels could be generated from a virtual grid of 10 × 10 × 5. Moreover, the reference coordinate system of the control points did not significantly affect the performance of the RFM generation process.

Stereo rectification of pushbroom satellite images by robustly estimating the fundamental matrix

ABSTRACT Stereo rectification is one of the most important steps for stereo matching and subsequently for digital surface model generation from satellite stereo images. This study proposes a new framework to rectify two pushbroom images along the epipolar geometry in order to omit the vertical parallax between two images. Here, we assume the interior and relative parameters between the two pushbroom images are not known and the images can be taken at different dates. Traditional stereo rectification methods of pushbroom images require metadata such as rational polynomial coefficients (RPCs), parameters of physical sensor model or ground control points (GCPs). In this study, we develop an image-based framework for stereo rectification, which works without the need for such data. In the proposed framework, the correspondences are densely extracted by a tilling strategy, and then the fundamental matrix is robustly estimated by two geometric constraints. Both affine and projective fundamental matrices could be used for stereo rectification from pushbroom stereo images. The results on IRS P5, World view III, GeoEye and IKONOS stereo pairs as well as on multi-date stereo images demonstrate that the pushbroom images are rectified with sub-pixel accuracy.

Rational Polynomial Coefficients Modeling and Bias Correction by Using Iterative Polynomial Augmentation

In this article, we establish an update procedure for rapid positioning coefficients or rational polynomial coefficients (RPCs) via iterative refinements using polynomial augmentation and reference images. RPCs are widely popular in establishing a ground-to-image relationship without using physical sensor model. However, the accuracies of RPCs are degraded due to unavoidable errors in physical sensor model based on colinearity conditions. These inaccuracies essentially arise due to undulating terrain, residual errors in attitude parameters, viz. roll, pitch and yaw, inexact modeling of drift and micro-vibration, orbit error, etc. In the paper, first an initial estimate of RPCs is obtained by using \(L^2\)-regularized least square estimation. Subsequently, the RPCs are refined by using iterative affine augmentation. The RPC accuracy is further improved by a second-order polynomial augmentation. The results show that with the improved RPCs the average scan and pixel errors are within 0.5 pixel. The results of the paper are employed and validated on Resourcesat-2 imagery.

Comparison between Range-Doppler and Rational-Function methods for SAR Terrain Geocoding

Geocoding of Synthetic Aperture Radar (SAR) imagery is an essential preprocessing step in multi-source data integration, management and analysis for many geomatics applications. The Range-Doppler (RD) approach is a rigorous physical sensor model, which achieves the highest geometric accuracy and has been widely used in terrain-geocoding of satellite SAR data. However the main drawback is that it is not computationally efficient. The Rational-Function (RF) model has been successfully used for processing of optical images (push broom sensors IKONOS and QuickBird), and is of considerable interest in processing of SAR datasets as an alternative to the RD method because of its high computational efficiency and fitting accuracy (e.g. RADARSAT-2). Two SAR terrain-geocoding methods, based on the RD and the RF models, have been implemented at Canada Center for Mapping and Earth Observation (CCMEO). In this paper we briefly describe the well-known RD mathematical model and the RF mathematical model in more detail. The relative geometric accuracy for the two methods is analysed and compared. Finally, some major advantages of the RF method over the RD method are presented.

Assessment of geometric correction of remote sensing satellite images using RPC versus GCP

Geometric distortions are common problems when dealing with remote sensing satellite images. Therefore, geometric correction is a necessary process for preparing remote sensing satellite images for many applications. Physical sensor model is formed by integrating the geometry of imaging sensor and positioning sensors as GPS and star trackers with system calibration parameters. Physical sensor model is not available in common but a Rational Polynomial Coefficient (RPC) model is provided as an alternative representation of sensor model. The RPC model is used for geometric correction of satellite image to get the spatial data of image features. The accuracy of the resultant image depends on the accuracy of RPC model which is not known as common. The research objective is to assess the accuracy of geometric correction of the satellite image using RPC model versus geometric correction using Ground Control Points (GCPs), along with their effect on the final accuracy of the output spatial features. The available data is IKONOS-2 image with its RPC file and seven GCPs with high accuracy obtained from ground survey for the study area. The input image is corrected using two available data separately. First degree of polynomial is used for transformation process for the case of GCPs. Bilinear interpolation technique is used to determine the pixel value of the newly resultant corrected images for the two cases. GCPs is preferred when available because the resultant image with RPC geometric correction has 15.0 meters average linear error while 3.0 meters error in case of GCPs geometric correction.

Plug-and-Play Unplugged: Optimization-Free Reconstruction Using Consensus Equilibrium

Regularized inversion methods for image reconstruction are used widely due to their tractability and their ability to combine complex physical sensor models with useful regularity criteria. Such methods motivated the recently developed Plug-and-Play prior method, which provides a framework to use advanced denoising algorithms as regularizers in inversion. However, the need to formulate regularized inversion as the solution to an optimization problem limits the expressiveness of possible regularity conditions and physical sensor models. In this paper, we introduce the idea of consensus equilibrium (CE), which generalizes regularized inversion to include a much wider variety of both forward (or data fidelity) components and prior (or regularity) components without the need for either to be expressed using a cost function. CE is based on the solution of a set of equilibrium equations that balance data fit and regularity. In this framework, the problem of MAP estimation in regularized inversion is replaced by...

Nonlinear bias compensation of ZiYuan-3 satellite imagery with cubic splines

Like many high-resolution satellites such as the ALOS, MOMS-2P, QuickBird, and ZiYuan1-02C satellites, the ZiYuan-3 satellite suffers from different levels of attitude oscillations. As a result of such oscillations, the rational polynomial coefficients (RPCs) obtained using a terrain-independent scenario often have nonlinear biases. In the sensor orientation of ZiYuan-3 imagery based on a rational function model (RFM), these nonlinear biases cannot be effectively compensated by an affine transformation. The sensor orientation accuracy is thereby worse than expected. In order to eliminate the influence of attitude oscillations on the RFM-based sensor orientation, a feasible nonlinear bias compensation approach for ZiYuan-3 imagery with cubic splines is proposed. In this approach, no actual ground control points (GCPs) are required to determine the cubic splines. First, the RPCs are calculated using a three-dimensional virtual control grid generated based on a physical sensor model. Second, one cubic spline is used to model the residual errors of the virtual control points in the row direction and another cubic spline is used to model the residual errors in the column direction. Then, the estimated cubic splines are used to compensate the nonlinear biases in the RPCs. Finally, the affine transformation parameters are used to compensate the residual biases in the RPCs. Three ZiYuan-3 images were tested. The experimental results showed that before the nonlinear bias compensation, the residual errors of the independent check points were nonlinearly biased. Even if the number of GCPs used to determine the affine transformation parameters was increased from 4 to 16, these nonlinear biases could not be effectively compensated. After the nonlinear bias compensation with the estimated cubic splines, the influence of the attitude oscillations could be eliminated. The RFM-based sensor orientation accuracies of the three ZiYuan-3 images reached 0.981 pixels, 0.890 pixels, and 1.093 pixels, which were respectively 42.1%, 48.3%, and 54.8% better than those achieved before the nonlinear bias compensation.

Real-time interferometric monitoring and measuring of photopolymerization based stereolithographic additive manufacturing process: sensor model and algorithm

As additive manufacturing is poised for growth and innovations, it faces barriers of lack of in-process metrology and control to advance into wider industry applications. The exposure controlled projection lithography (ECPL) is a layerless mask-projection stereolithographic additive manufacturing process, in which parts are fabricated from photopolymers on a stationary transparent substrate. To improve the process accuracy with closed-loop control for ECPL, this paper develops an interferometric curing monitoring and measuring (ICM&M) method which addresses the sensor modeling and algorithms issues. A physical sensor model for ICM&M is derived based on interference optics utilizing the concept of instantaneous frequency. The associated calibration procedure is outlined for ICM&M measurement accuracy. To solve the sensor model, particularly in real time, an online evolutionary parameter estimation algorithm is developed adopting moving horizon exponentially weighted Fourier curve fitting and numerical integration. As a preliminary validation, simulated real-time measurement by offline analysis of a video of interferograms acquired in the ECPL process is presented. The agreement between the cured height estimated by ICM&M and that measured by microscope indicates that the measurement principle is promising as real-time metrology for global measurement and control of the ECPL process.

Automatic Geometric Processing for Very High Resolution Optical Satellite Data Based on Vector Roads and Orthophotos

In response to the increasing need for fast satellite image processing SPACE-SI developed STORM—a fully automatic image processing chain that performs all processing steps from the input optical images to web-delivered map-ready products for various sensors. This paper focuses on the automatic geometric corrections module and its adaptation to very high resolution (VHR) multispectral images. In the automatic ground control points (GCPs) extraction sub-module a two-step algorithm that utilizes vector roads as a reference layer and delivers GCPs for high resolution RapidEye images with near pixel accuracy was initially implemented. Super-fine positioning of individual GCPs onto an aerial orthophoto was introduced for VHR images. The enhanced algorithm is capable of achieving accuracy of approximately 1.5 pixels on WorldView-2 data. In the case of RapidEye images the accuracies of the physical sensor model reach sub-pixel values at independent check points. When compared to the reference national aerial orthophoto the accuracies of WorldView-2 orthoimages automatically produced with the rational function model reach near-pixel values. On a heterogeneous set of 41 RapidEye images the rate of automatic processing reached 97.6%. Image processing times remained under one hour for standard-size images of both sensor types.

QUASI-EPIPOLAR RESAMPLING OF HIGH RESOLUTION SATELLITE STEREO IMAGERY FOR SEMI GLOBAL MATCHING

Abstract. Semi-global matching is a well-known stereo matching algorithm in photogrammetric and computer vision society. Epipolar images are supposed as input of this algorithm. Epipolar geometry of linear array scanners is not a straight line as in case of frame camera. Traditional epipolar resampling algorithms demands for rational polynomial coefficients (RPCs), physical sensor model or ground control points. In this paper we propose a new solution for epipolar resampling method which works without the need for these information. In proposed method, automatic feature extraction algorithms are employed to generate corresponding features for registering stereo pairs. Also original images are divided into small tiles. In this way by omitting the need for extra information, the speed of matching algorithm increased and the need for high temporal memory decreased. Our experiments on GeoEye-1 stereo pair captured over Qom city in Iran demonstrates that the epipolar images are generated with sub-pixel accuracy.