Bias Correction Step Research Articles

Sensitivity of bias correction step on generating hydrological scenarios

ABSTRACT Significant shifts in hydro-climatic regimes are expected in many parts of the world during the 21st century, affecting the water cycle. Vulnerability, impact, and adaptation studies often use tailored modeling chains to assess the expected effects of climate change, but the robustness of these chains is rarely investigated. This highlights the need for more rigorous evaluation of modeling chains to ensure that they are reliable for informed decision-making processes. To address this gap, we propose a framework for evaluating the sensitivity of hydrological scenario production to the bias correction step. We apply the framework to the Senegal River Basin, using three bias correction methods (linear scale, empirical quantile mapping, and nested bias correction) and three procedures (climate-correction, hydrological-correction, and climate-hydrological-correction). Our results show that the choice of modeling chain has a significant impact on future hydro-climatic trajectories. In particular, the combination of climate-and-hydrological-correction procedures may be optimal when both climate biases and hydrological model errors are significant. Moreover, using multiple bias correction methods can strengthen the ensemble of future hydro-climatic conditions. These findings have implications for vulnerability–impact–adaptation studies and underscore the importance of rigorous modeling chain design and sensitivity analysis.

A spatially promoted SVM model for GRACE downscaling: Using ground and satellite-based datasets

Satellite-based terrestrial water storage changes have been recorded using the Gravity Recovery and Climate Experiment (GRACE) satellite which causing it an important dataset in hydrology and other related fields. GRACE dataset is widely utilized in many studies, but its coarse spatial resolution is a limiting drawback. Machine-learning approaches (e.g., ANN and SVM) are commonly applied in spatially downscaling. However, their input formation, which is in vector form, is a limitation of considering neighbor relations between the gridded-based inputs, specifically in spatial downscaling. Thus, developing an appropriate, simple, fast, and novel model to spatially downscale GRACE resolution is initially necessary for its utilizations. In this study, a Spatially Promoted Support Vector Machine (SP-SVM) model is innovatively proposed for GRACE downscaling from 0.5° to 0.25°. This promotion is investigated utilizing the distances between the unknown target points (with 0.25°) and their surrounding GRACE-valued points (0.5°), called their Distance Effect Coefficient (DEC), as the SP-SVM model input. In addition, the efficiencies of different in-situ and satellite-based datasets (fifteen variables from May 2005 to August 2020) are evaluated as the inputs of the GRACE downscaling models. After finding the most influential datasets, showing the best correlation with the GRACE, their best combinations in GRACE downscaling are identified. Based on the results, the set of PERSIANN-CDR without delay, the in-situ evaporation with a 1-month delay, and the soil moisture in 0–10 cm depth with a 1-month delay show the best performance in GRACE downscaling. The results of GRACE downscaling by the SP-SVM approach are also compared with the ones based on a usual statistical SVM (S-SVM) model, consisting of an intermediate bias interpolation to improve the estimations through a bias correction step. The results show that the SP-SVM model outperforms the common statistical SVM-based. Thus, compared with the usual S-SVM approach, the proposed SP-SVM (linear) model could be used as a simpler and more accurate model for downscaling any variable in a hierarchical process.

Surrogate Scoring Rules

Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of elicited information, with more accurate predictions receiving higher scores in expectation. In this article, we extend such scoring rules to settings in which a principal elicits private probabilistic beliefs but only has access to agents’ reports. We name our solution Surrogate Scoring Rules (SSR). SSR is built on a bias correction step and an error rate estimation procedure for a reference answer defined using agents’ reports. We show that, with a little information about the prior distribution of the random variables, SSR in a multi-task setting recover SPSR in expectation, as if having access to the ground truth. Therefore, a salient feature of SSR is that they quantify the quality of information despite the lack of ground truth, just as SPSR do for the setting with ground truth. As a by-product, SSR induce dominant uniform strategy truthfulness in reporting. Our method is verified both theoretically and empirically using data collected from real human forecasters.

Multimodel Errors and Emergence Times in Climate Attribution Studies

Abstract Global climate models, like any in silico numerical experiments, are affected by different types of bias. Uncertainty quantification remains a challenge in any climate detection and attribution analysis. A fundamental methodological question is to determine which statistical summaries, while bringing relevant signals, can be robust with respect to multimodel errors. In this paper, we propose a simple statistical framework that significantly improves signal detection in climate attribution studies. We show that the complex bias correction step can be entirely bypassed for models for which bias between the simulated and unobserved counterfactual worlds is the same as between the simulated and unobserved factual worlds. To illustrate our approach, we infer emergence times in precipitation from the CMIP5 and CMIP6 archives. The detected anthropogenic signal in yearly maxima of daily precipitation clearly emerges at the beginning of the twenty-first century. In addition, no CMIP model seems to outperform the others and a weighted linear combination of all improves the estimation of emergence times. Significance Statement We show that the bias in multimodel global climate simulations can be efficiently handled when the appropriate metric is chosen. This metric leads to an easy-to-implement statistical procedure based on a checkable assumption. This allows us to demonstrate that optimal convex combinations of CMIP outputs can improve the signal strength in finding emergence times. Our data analysis procedure is applied to yearly maximum of precipitation from CMIP5 and CMIP6 databases. The attribution of the anthropogenic forcing clearly emerges in extreme precipitation at the beginning of the twenty-first century.

Impact of correcting sub-daily climate model biases for hydrological studies

Abstract. The study of climate change impact on water resources has accelerated worldwide over the past 2 decades. An important component of such studies is the bias-correction step, which accounts for spatiotemporal biases present in climate model outputs over a reference period, and which allows for realistic streamflow simulations using future climate scenarios. Most of the literature on bias correction focuses on daily scale climate model temporal resolution. However, a large amount of regional and global climate simulations are becoming increasingly available at the sub-daily time step, and even extend to the hourly scale, with convection-permitting models exploring sub-hourly time resolution. Recent studies have shown that the diurnal cycle of variables simulated by climate models is also biased, which raises issues respecting the necessity (or not) of correcting such biases prior to generating streamflows at the sub-daily timescale. This paper investigates the impact of bias-correcting the diurnal cycle of climate model outputs on the computation of streamflow over 133 small to large North American catchments. A standard hydrological modeling chain was set up using the temperature and precipitation outputs from a high spatial (0.11∘) and temporal (1 h) regional climate model large ensemble (ClimEx-LE). Two bias-corrected time series were generated using a multivariate quantile mapping method, with and without correction of the diurnal cycles of temperature and precipitation. The impact of this correction was evaluated on three small (< 500 km2), medium (between 500 and 1000 km2), and large (> 1000 km2) surface area catchment size classes. Results show relatively small (3 % to 5 %) but systematic decreases in the relative error of most simulated flow quantiles when bias-correcting the diurnal cycle of precipitation and temperature. There was a clear relationship with catchment size, with improvements being most noticeable for the small catchments. The diurnal cycle correction allowed for hydrological simulations to accurately represent the diurnal cycle of summer streamflow in small catchments. Bias-correcting the diurnal cycle of precipitation and temperature is therefore recommended when conducting impact studies at the sub-daily timescale on small catchments.

Variable Importance Scores

There are many methods of scoring the importance of variables in prediction of a response but not much is known about their accuracy. This paper partially fills the gap by introducing a new method based on the GUIDE algorithm and comparing it with 11 existing methods. For data without missing values, eight methods are shown to give biased scores that are too high or too low, depending on the type of variables (ordinal, binary or nominal) and whether or not they are dependent on other variables, even when all of them are independent of the response. Among the remaining four methods, only GUIDE continues to give unbiased scores if there are missing data values. It does this with a self-calibrating bias-correction step that is applicable to data with and without missing values. GUIDE also provides threshold scores for differentiating important from unimportant variables with 95 and 99 percent confidence. Correlations of the scores to the predictive power of the methods are studied in three real data sets. For many methods, correlations with marginal predictive power are much higher than with conditional predictive power.

Personalized predictive public transport crowding information with automated data sources

The paper proposes a methodology for providing personalized, predictive in-vehicle crowding information to public transport travellers via mobile applications or at-stop displays. Three crowding metrics are considered: (1) the probability of getting a seat on boarding, (2) the expected travel time standing, and (3) the excess perceived travel time compared to uncrowded conditions. The methodology combines prediction models of passenger loads and alighting counts based on lasso regularized regression and multivariate PLS regression, a probabilistic seat allocation model and a bias correction step in order to predict the crowding metrics. Depending on data availability, the prediction method can use a combination of historical passenger counts, real-time vehicle locations and real-time passenger counts. We evaluate the prediction methodology in a real-world case study for a bus line in Stockholm, Sweden. The results indicate that personalized, predictive crowding information that is robust to varying data availability can be provided sufficiently early to be useful to travellers. The methodology is of value for agencies and operators in order to increase the attractiveness and capacity utilization of public transport.

Bias field correction for improved compressed sensing reconstruction in parallel magnetic resonance imaging

Parallel imaging and compressed sensing (PICS) may accelerate magnetic resonance imaging (MRI) acquisition with advanced reconstruction algorithms from under-sampled data set. However, bias field effects are often present in reconstructed MRI images due to hardware limitation and object property, which might lead to reconstruction imperfection with conventional PICS reconstruction due to altered image sparsity. In this paper, the MR signal bias field effects is modeled, and it was found that the bias field effects would induce reconstruction artifacts in the low-signal-intensity area. Then it was proposed to add a bias correction step into the PICS reconstruction framework to improve the reconstruction, and the proposed method was evaluated the proposed method via both simulation and in vivo study. It was then shown that the proposed method leads to improved image reconstruction in the low-signal-intensity area, with little extra computational effort needed compared to standard PICS reconstruction. This method could be readily extended to other signal processing area where the signal inhomogeneity problem is present and signal sparsity is exploited.

Data analytics approach for online produced fluid flow rate estimation in SAGD process

In oil industries with SAGD facilities, continuous and fast-rate produced fluids flow rate measurement at each well pair is essential for implementing advanced process control and optimization methods. However, it is not always available due to expensive instrumentation and unreliable measurement. Soft sensors have become a popular alternative to the hardware sensors owing to the availability of a large amount of data archived during production. This paper presents a comprehensive data-driven approach for developing and implementing soft sensors for fluid flow rates of SAGD wells. The steps taken in offline model training are first discussed. For online implementation, a robust layer and a data reconciliation-based bias correction step are proposed to enhance the accuracy and reliability of the developed soft sensor. The effectiveness of the soft sensor is demonstrated through successful applications to various SAGD wells.

Alternating EM algorithm for a bilinear model in isoform quantification from RNA-seq data.

Estimation of isoform-level gene expression from RNA-seq data depends on simplifying assumptions, such as uniform read distribution, that are easily violated in real data. Such violations typically lead to biased estimates. Most existing methods provide bias correction step(s), which is based on biological considerations-such as GC content-and applied in single samples separately. The main problem is that not all biases are known. We have developed a novel method called XAEM based on a more flexible and robust statistical model. Existing methods are essentially based on a linear model Xβ, where the design matrix X is known and is computed based on the simplifying assumptions. In contrast XAEM considers Xβ as a bilinear model with both X and β unknown. Joint estimation of X and β is made possible by a simultaneous analysis of multi-sample RNA-seq data. Compared to existing methods, XAEM automatically performs empirical correction of potentially unknown biases. We use an alternating expectation-maximization (AEM) algorithm, alternating between estimation of X and β. For speed XAEM utilizes quasi-mapping for read alignment, thus leading to a fast algorithm. Overall XAEM performs favorably compared to recent advanced methods. For simulated datasets, XAEM obtains higher accuracy for multiple-isoform genes. In a differential-expression analysis of a real single-cell RNA-seq dataset, XAEM achieves substantially better rediscovery rates in independent validation sets. The method and pipeline are implemented as a tool and freely available for use at http://fafner.meb.ki.se/biostatwiki/xaem/. Supplementary data are available at Bioinformatics online.

Optimal Estimation of Genetic Relatedness in High-Dimensional Linear Models

ABSTRACTEstimating the genetic relatedness between two traits based on the genome-wide association data is an important problem in genetics research. In the framework of high-dimensional linear models, we introduce two measures of genetic relatedness and develop optimal estimators for them. One is genetic covariance, which is defined to be the inner product of the two regression vectors, and another is genetic correlation, which is a normalized inner product by their lengths. We propose functional de-biased estimators (FDEs), which consist of an initial estimation step with the plug-in scaled Lasso estimator, and a further bias correction step. We also develop estimators of the quadratic functionals of the regression vectors, which can be used to estimate the heritability of each trait. The estimators are shown to be minimax rate-optimal and can be efficiently implemented. Simulation results show that FDEs provide better estimates of the genetic relatedness than simple plug-in estimates. FDE is also applied to an analysis of a yeast segregant dataset with multiple traits to estimate the genetic relatedness among these traits. Supplementary materials for this article are available online.

Colour and illumination in computer vision.

In computer vision, illumination is considered to be a problem that needs to be 'solved'. The colour cast due to illumination is removed to support colour-based image recognition and stable tracking (in and out of shadows), among other tasks. In this paper, I review historical and current algorithms for illumination estimation. In the classical approach, the illuminant colour is estimated by an ever more sophisticated analysis of simple image summary statistics often followed by a bias correction step. Bias correction is a function applied to the estimates made by a given illumination estimation algorithm to correct consistent errors in the estimations. Most recently, the full power, and much higher complexity, of deep learning has been deployed (where, effectively, the definition of the image statistics of interest and the type of analysis carried out are found as part of an overall optimization). In this paper, I challenge the orthodoxy of deep learning, i.e. that it is the best approach for illuminant estimation. We instead focus on the final bias correction stage found in many simple illumination estimation algorithms. There are two key insights in our method. First, we argue that the bias must be corrected in an exposure invariant way. Second, we show that this bias correction amounts to 'solving for a homography'. Homography-based illuminant estimation is shown to deliver leading illumination estimation performance (at a very small fraction of the complexity of deep learning methods).

Improving Value-at-Risk Estimation from the Normal EGARCH Model

Returns in financial assets display consistent excess kurtosis and skewness, implying the presence of large fluctuations not forecasted by Gaussian models. This paper applies a resampling method based on the bootstrap and a bias-correction step to improve Value-at-Risk (VaR) forecasting ability of the n-EGARCH (normal EGARCH) model and correct the VaR for both long and short positions. Our aim is to utilize the advantages of this model, but still use the bootstrap resampling method to accurate for the tendency of the model tomiscalculate the VaR. Empirical results indicate that the bias-correction method can improve the n-GARCH and n-EGARCH VaR forecasts so much that the acquired VaR predictions are different from the proposed probability. Additionally, allowing asymmetry in the conditional variance using the EGARCH model with normal distribution instead of GARCH improves the performance of the bias-correction method in forecasting the VaR for almost all considered indices. Moreover, the bias-corrected n-EGARCH model performs better than the simple t-EGARCH model. Thus, it seems that this model can take account of both the asymmetry in the conditional variance and leptokurtosis in returns distribution. However, we find that the superiority of the bias-corrected n-EGARCH model over the t-EGARCH model is not completely confirmed for short positions based on the censored likelihood scoring rule.

Digital soil mapping of soil carbon at the farm scale: A spatial downscaling approach in consideration of measured and uncertain data

In this paper a spatial downscaling method is explored for generating appropriate farm scale digital soil maps. The digital soil map product to be downscaled is an Australian national extent soil carbon map (100m grid resolution). Taking into account the associated prediction uncertainties of this map, we used a simulation approach based on Gaussian random fields to generate plausible mapping realisations that were in turn downscaled to 10m resolution for a farm in North-western NSW, Australia. We were able to derive both a downscaled map of soil carbon and associated prediction variance with this approach. Building further upon this development, we then incorporated a bias correction step into the spatial downscaling procedure which permits the inclusion of field observations as a way to moderate the downscaling results to better reflect actual conditions on the ground. Based on an independent validation dataset, it was found that incorporating field observations increase the concordance correlation coefficient to 0.8 from 0.2. This relatively lower correlation achieved using spatial downscaling alone was due to the national scale mapping for the study area being positively biased in the area of interest. It was found that downscaling that incorporates observational data was marginally better if not comparable to using a point-based digital soil mapping approach. The advantage of spatial downscaling is that it can be implemented in situations of data scarcity. This will be ideal for on farm soil monitoring in situations where detailed soil mapping is initially not available. For example, soil carbon auditing schemes requiring prior soil information for implementation of design-based soil sampling could potentially be universally applied with such a spatial downscaling approach.

Estimation of Basis Line-Integrals in a Spectral Distortion-Modeled Photon Counting Detector Using Low-Order Polynomial Approximation of X-ray Transmittance.

Photon counting detector (PCD)-based computed tomography exploits spectral information from a transmitted x-ray spectrum to estimate basis line-integrals. The recorded spectrum, however, is distorted and deviates from the transmitted spectrum due to spectral response effect (SRE). Therefore, the SRE needs to be compensated for when estimating basis line-integrals. One approach is to incorporate the SRE model with an incident spectrum into the PCD measurement model and the other approach is to perform a calibration process that inherently includes both the SRE and the incident spectrum. A maximum likelihood estimator can be used to the former approach, which guarantees asymptotic optimality; however, a heavy computational burden is a concern. Calibration-based estimators are a form of the latter approach. They can be very efficient; however, a heuristic calibration process needs to be addressed. In this paper, we propose a computationally efficient three-step estimator for the former approach using a low-order polynomial approximation of x-ray transmittance. The low-order polynomial approximation can change the original non-linear estimation method to a two-step linearized approach followed by an iterative bias correction step. We show that the calibration process is required only for the bias correction step and prove that it converges to the unbiased solution under practical assumptions. Extensive simulation studies validate the proposed method and show that the estimation results are comparable to those of the ML estimator while the computational time is reduced substantially.

Assimilating urban heat island effects into climate projections

An urban heat island (UHI) effect is identified in Reno, Nevada by analyzing regional temperature trends calculated from seven long-term observation stations for the summer and winter seasons between 1950 and 2014. The UHI is maximized during summer (June–August) and characterized by asymmetric increases in minimum (∼1.1 °C/decade, p < 0.01) versus maximum temperature (∼0.1 °C/decade, p < 0.01) trends in excess of trends from regional climate stations. Comparisons of historical Reno temperatures with an ensemble of 66 bias-corrected and downscaled global climate model (GCM) outputs spanning 1950–2014 demonstrates cold biases of 1.5–4.5 °C during summer with minimum temperature having the largest bias. We show that a secondary bias correction step utilizing the statistical downscaling method of quantile–quantile mapping (QQM) can reduce biases in future climate projections assuming no changes to the UHI. The QQM results in an additional total warming of ensemble mean temperatures by ∼3 °C for downscaled GCM output and ∼4 °C for re-gridded 1° grid resolution GCM output for 2030–2049 under the RCP8.5 emissions scenario. These temperature differences produce additional increases in summer potential evapotranspiration of 10% compared to non-QQM bias-corrected GCM output. It was shown that the QQM method represents a useful and computationally efficient method for bias correction of temperature projections for cities where UHI effects exist. Planning and impacts studies of urban water resources can benefit from these improved climate projections, particularly in regions where downscaled GCM output is unavailable.

Creation of a high resolution precipitation data set by merging gridded gauge data and radar observations for Sweden

Summary Hydrological forecasting systems require accurate initial conditions, particularly for real time precipitation data, which are problematic to retrieve. This is especially difficult for high temporal and spatial resolutions, e.g. sub-daily and less than 10–20 km. Forecasting fast processes such as flash flood are, however, dependent on such high resolution data. Gridded gauge data produces too smooth fields and underestimates small scale phenomena, such as convection, whereas radar composites contain the small scale information, but suffer from inconsistencies between individual radars and have poor long term statistics. Here, we present a method to merge a radar composite with daily resolution gridded gauge data for Sweden for the time period 2009–2014 to produce a one hourly 4 × 4 km2 data set. The method consists of a main step where monthly accumulations of the radar data are scaled by those retrieved from the gridded data for each month. An optional quantile mapping based bias correction step makes sure that the daily intensity distribution agrees with the gridded observations. Finally, the data are dis-aggregated to an hourly time resolution. This results in a data set which has the same long-term spatial properties as the gridded observations, but with the spatial and temporal details of the radar data. Validation of the method is performed with high resolution gauge data, and shows a high quality of the derived product.

Soil Moisture Model Calibration and Validation: An ARS Watershed on the South Fork Iowa River

Abstract Soil moisture monitoring with in situ technology is a time-consuming and costly endeavor for which a method of increasing the resolution of spatial estimates across in situ networks is necessary. Using a simple hydrologic model, the estimation capacity of an in situ watershed network can be increased beyond the station distribution by using available precipitation, soil, and topographic information. A study site was selected on the Iowa River, characterized by homogeneous soil and topographic features, reducing the variables to precipitation only. Using 10-km precipitation estimates from the North American Land Data Assimilation System (NLDAS) for 2013, high-resolution estimates of surface soil moisture were generated in coordination with an in situ network, which was deployed as part of the Iowa Flood Studies (IFloodS). A simple, bucket model for soil moisture at each in situ sensor was calibrated using four precipitation products and subsequently validated at both the sensor for which it was calibrated and other proximal sensors, the latter after a bias correction step. Average RMSE values of 0.031 and 0.045 m3 m−3 were obtained for models validated at the sensor for which they were calibrated and at other nearby sensors, respectively.

Projected 21st century snowfall changes over the French Alps and related uncertainties

Snowfall changes in mountain areas in response to anthropogenic forcing could have widespread hydrological, ecological and economic impacts. In this paper, the robustness of snowfall changes over the French Alps projected during the 21st century and the associated uncertainties are studied. In particular, the role of temperature changes on snowfall changes is investigated. Those issues are tackled through the analysis of the results of a very large ensemble of high-resolution regional climate projections, obtained either through dynamical or statistical downscaling. We find that, at the beginning and at the end of the cold season extending from November to March (included), temperature change is an important source of spread in snowfall changes. However, no link is found between temperature and snowfall changes in January and February. At the beginning and at the end of the cold season, the rate of change in snowfall per Kelvin does not depend much on the bias correction step, the period or the greenhouse gas scenario but mostly on the downscaling method and the climate models, the latter uncertainty source being dominant.

Extended optimal estimation techniques for sea surface temperature from the Spinning Enhanced Visible and Infra-Red Imager (SEVIRI)

Sea surface temperature (SST) can be estimated from day and night observations of the Spinning Enhanced Visible and Infra-Red Imager (SEVIRI) by optimal estimation (OE). We show that exploiting the 8.7μm channel, in addition to the “traditional” wavelengths of 10.8 and 12.0μm, improves OE SST retrieval statistics in validation. However, the main benefit is an improvement in the sensitivity of the SST estimate to variability in true SST.In a fair, single-pixel comparison, the 3-channel OE gives better results than the SST estimation technique presently operational within the Ocean and Sea Ice Satellite Application Facility. This operational technique is to use SST retrieval coefficients, followed by a bias-correction step informed by radiative transfer simulation. However, the operational technique has an additional “atmospheric correction smoothing”, which improves its noise performance, and hitherto had no analogue within the OE framework. Here, we propose an analogue to atmospheric correction smoothing, based on the expectation that atmospheric total column water vapour has a longer spatial correlation length scale than SST features. The approach extends the observations input to the OE to include the averaged brightness temperatures (BTs) of nearby clear-sky pixels, in addition to the BTs of the pixel for which SST is being retrieved. The retrieved quantities are then the single-pixel SST and the clear-sky total column water vapour averaged over the vicinity of the pixel. This reduces the noise in the retrieved SST significantly. The robust standard deviation of the new OE SST compared to matched drifting buoys becomes 0.39K for all data. The smoothed OE gives SST sensitivity of 98% on average. This means that diurnal temperature variability and ocean frontal gradients are more faithfully estimated, and that the influence of the prior SST used is minimal (2%). This benefit is not available using traditional atmospheric correction smoothing.