Bias Correction Strategies Research Articles

Double data piling: a high-dimensional solution for asymptotically perfect multi-category classification

For high-dimensional classification, interpolation of training data manifests as the data piling phenomenon, in which linear projections of data vectors from each class collapse to a single value. Recent research has revealed an additional phenomenon known as the ‘second data piling’ for independent test data in binary classification, providing a theoretical understanding of asymptotically perfect classification. This paper extends these findings to multi-category classification and provides a comprehensive characterization of the double data piling phenomenon. We define the maximal data piling subspace, which maximizes the sum of pairwise distances between piles of training data in multi-category classification. Furthermore, we show that a second data piling subspace that induces data piling for independent data exists and can be consistently estimated by projecting the negatively-ridged discriminant subspace onto an estimated ‘signal’ subspace. By leveraging this second data piling phenomenon, we propose a bias-correction strategy for class assignments, which asymptotically achieves perfect classification. The present research sheds light on benign overfitting and enhances the understanding of perfect multi-category classification of high-dimensional discrimination with a help of high-dimensional asymptotics.

OTAMatch: Optimal Transport Assignment with PseudoNCE for Semi-supervised Learning.

In semi-supervised learning (SSL), many approaches follow the effective self-training paradigm with consistency regularization, utilizing threshold heuristics to alleviate label noise. However, such threshold heuristics lead to the underutilization of crucial discriminative information from the excluded data. In this paper, we present OTAMatch, a novel SSL framework that reformulates pseudo-labeling as an optimal transport (OT) assignment problem and simultaneously exploits data with high confidence to mitigate the confirmation bias. Firstly, OTAMatch models the pseudo-label allocation task as a convex minimization problem, facilitating end-to-end optimization with all pseudo-labels and employing the Sinkhorn-Knopp algorithm for efficient approximation. Meanwhile, we incorporate epsilon-greedy posterior regularization and curriculum bias correction strategies to constrain the distribution of OT assignments, improving the robustness with noisy pseudo-labels. Secondly, we propose PseudoNCE, which explicitly exploits pseudo-label consistency with threshold heuristics to maximize mutual information within self-training, significantly boosting the balance of convergence speed and performance. Consequently, our proposed approach achieves competitive performance on various SSL benchmarks. Specifically, OTAMatch substantially outperforms the previous state-of-the-art SSL algorithms in realistic and challenging scenarios, exemplified by a notable 9.45% error rate reduction over SoftMatch on ImageNet with 100K-label split, underlining its robustness and effectiveness.

Do economic development and tourism heterogeneously influence ecological sustainability? Implications for sustainable development.

While economic development-driven anthropogenic emissions pose challenges to ecological sustainability, the international travel and tourism sector has appeared as a hot contestant to bring sustainability to the ecological systems across varying development levels. This work investigates the diversified effects of the international travel and tourism sector and economic development on ecological deterioration, in the presence of urban agglomeration and energy use efficiency, across the development levels of China's 30 provincial units from 2002 to 2019. It contributes in two ways. (i) The stochastic estimation of environmental impacts by regression on population, affluence, and technology (STIRPAT) is modified to integrate the variables like international travel and tourism sector, urban agglomeration, and energy use efficiency. (ii) We measured an international travel and tourism sector index (ITTI) and made use of a continuously updated bias correction strategy (CUBCS) and a continuously updated fully modified strategy (CUFMS) for the long-term estimations. Besides, we used the bootstrapping-based causality technique for determining causality directions. The core results are as follows: Firstly, ITTI and economic development produced an inverse U-type association with ecological deterioration for the aggregate panels. Secondly, provinces exhibited a diverse range of links in that ITTI mitigated (boosted) the ecological deterioration in eleven (fourteen) provinces presenting diversified shapes of linkages. Economic development established the environmental Kuznets curve (EKC) theory with ecological deterioration in merely four provinces; however, the non-EKC theory is verified in twenty-four divisions. Thirdly, in China's east zone (high development scale), the ITTI revealed the ecological deterioration reduction (promotion) impact in eight (two) provinces. China's central zone (moderate development scale) exhibited ecological deterioration promotion in half of the provinces, and the other half showed a reduction impact. In China's west zone (low development scale), it promoted ecological deterioration in eight provinces. Economic development promoted (reduced) ecological deterioration in a single (nine) province(s). In China's central zone, it boosted (mitigated) the ecological deterioration in five (three) provinces. In China's west zone, it promoted (reduced) ecological deterioration in eight (two) provinces. Fourthly, urban agglomeration and energy use efficiency deteriorated and improved the environmental quality in aggregated panels, respectively; however, a diverse range of effects are observed for provinces. Finally, a unilateral bootstrap causality, from ITTI (economic development) to ecological deterioration, is revealed in twenty-four (fifteen) provinces. A bilateral causality is established in a single (thirteen) province(s). Based on empirical findings, policies are suggested.

An Improvement in Maximum Likelihood Estimation of the Gompertz Distribution Parameters

In this study, we will look at estimating the parameters of the Gompertz distribution. We know that the maximum likelihood technique is the most often used method in the literature for parameter estimation. However, it is well known that the maximum likelihood estimators (MLEs) are biased for small sample sizes. As a result, we are motivated to produce nearly unbiased estimators for the parameters of this distribution. To be more specific, we concentrate on two bias-correction strategies (analytical and bootstrap approaches) to minimize MLE biases to the second order of magnitude. Monte Carlo simulations are used to compare the performances of these estimators. Finally, two real-data examples are offered to demonstrate the utility of our proposed estimators in small sample sizes.

A two-stage downscaling hydrological modeling approach via convolutional conditional neural process and geostatistical bias correction

Hydrological modeling plays an indispensable role in many practical applications, such as hydrology, water resources assessment, and environmental survey. Downscaling approach is able to obtain high-resolution hydrological models according to a prior reference model. In this work, we propose a downscaling hydrological modeling approach based on convolutional conditional neural processes (ConvCNP) and a novel geostatistical bias correction, named ConvCNP-GBC. A two-stage downscaling modeling workflow is presented. In the model building stage, to reduce the training consumption of the ConvCNP, low-resolution conditioning data are used for training. In the model generating stage, the ConvCNP is used to reconstruct coarse-scale realizations based on upscaling sampled conditioning data and to generate high-resolution realizations by downscaling. Besides, a geostatistical bias correction strategy is presented to refine large-scale realizations with high-resolution conditioning data constraints. The experimental results confirm that ConvCNP-GBC can reproduce heterogeneous patterns by using coarse-scale conditioning data and characterize hydrological structures. The downscaled realizations also have a high spatial correspondence with the coarse-scale realizations, which provide fine subsurface heterogeneous structures. The proposed method can be easily extended to reservoir modeling, geophysical inversion, etc.

Event-Based Bias Correction of the GPM IMERG V06 Product by Random Forest Method over Mainland China

The Global Precipitation Measurement (GPM) IMERG V06 product showed excellent performance in detecting precipitation, but still have room to improve. This study proposed an event-based bias correction strategy through random forest (RF) method to improve accuracy of the GPM IMERG V06 product over mainland China. Results showed that, over mainland China, biases caused by ‘hits’ events are most responsible for the errors of the GPM product, and ‘falseAlarms’ events took the least responsibility for that because of the small GPM values for ‘falseAlarms’ events. Compared with the raw GPM product, the bias-corrected GPM product showed better performance in both fitting observed precipitation values and in detecting precipitation events, thus the event-based bias-strategy in this study is credible. After bias correction, the ability of the bias-corrected GPM product was significantly improved for ‘hits’ events, but showed slight deterioration in RMSE and MAE and significant improvements in FAR and CSI for ‘falseAlarms’ events. This is because the established RF classification model tends to learn characteristics of the event with larger proportion, and then performed better in correctly identifying the event with larger proportion in the subregion.

On Quantiles Estimation Based on Stratified Sampling Using Multiplicative Bias Correction Approach

In the context of stratified sampling, we develop a nonparametric regression technique to estimating finite population quantiles in model‐based frameworks using a multiplicative bias correction strategy. Furthermore, the proposed estimator’s asymptotic behavior is presented, and when certain conditions are met, the estimator is observed to be asymptotically unbiased and asymptotically consistent. Simulation studies were conducted to determine the proposed estimator’s performance for the three quartiles of two fictitious populations under varied distributional assumptions. Based on relative biases, mean‐squared errors, and relative root‐mean‐squared errors, the proposed estimator can be extremely satisfactory, according to these findings.

Validation of Ensemble-Based Probabilistic Tropical Cyclone Intensity Change

Although there has been substantial improvement to numerical weather prediction models, accurate predictions of tropical cyclone rapid intensification (RI) remain elusive. The processes that govern RI, such as convection, may be inherently less predictable; therefore a probabilistic approach should be adopted. Although there have been numerous studies that have evaluated probabilistic intensity (i.e., maximum wind speed) forecasts from high resolution models, or statistical RI predictions, there has not been a comprehensive analysis of high-resolution ensemble predictions of various intensity change thresholds. Here, ensemble-based probabilities of various intensity changes are computed from experimental Hurricane Weather Research and Forecasting (HWRF) and Hurricanes in a Multi-scale Ocean-coupled Non-hydrostatic (HMON) models that were run for select cases during the 2017–2019 seasons and verified against best track data. Both the HWRF and HMON ensemble systems simulate intensity changes consistent with RI (30 knots 24 h−1; 15.4 m s−1 24 h−1) less frequent than observed, do not provide reliable probabilistic predictions, and are less skillful probabilistic forecasts relative to the Statistical Hurricane Intensity Prediction System Rapid Intensification Index (SHIPS-RII) and Deterministic to Probabilistic Statistical (DTOPS) statistical-dynamical systems. This issue is partly alleviated by applying a quantile-based bias correction scheme that preferentially adjusts the model-based intensity change at the upper-end of intensity changes. While such an approach works well for high-resolution models, this bias correction strategy does not substantially improve ECMWF ensemble-based probabilistic predictions. By contrast, both the HWRF and HMON systems provide generally reliable predictions of intensity changes for cases where RI does not take place. Combining the members from the HWRF and HMON ensemble systems into a large multi-model ensemble does not improve upon HMON probablistic forecasts.

RiskLogitboost Regression for Rare Events in Binary Response: An Econometric Approach

A boosting-based machine learning algorithm is presented to model a binary response with large imbalance, i.e., a rare event. The new method (i) reduces the prediction error of the rare class, and (ii) approximates an econometric model that allows interpretability. RiskLogitboost regression includes a weighting mechanism that oversamples or undersamples observations according to their misclassification likelihood and a generalized least squares bias correction strategy to reduce the prediction error. An illustration using a real French third-party liability motor insurance data set is presented. The results show that RiskLogitboost regression improves the rate of detection of rare events compared to some boosting-based and tree-based algorithms and some existing methods designed to treat imbalanced responses.

Annual Cycle of East Asian Precipitation Simulated by CMIP6 Models

Annual cycle is fundamental in the East Asian monsoon (EAM) systems, profoundly governing the spatiotemporal distribution of the East Asian rainfall. The present study identified the dominant modes of the annual cycle in the East Asian rainfall based on the Fourier harmonic analysis and the Empirical Orthogonal Function (EOF) decomposition. We evaluated the performance of the first two leading modes (i.e., EOF-1 and EOF-2) in historical experiments (1979–2014) of the 21 released climate models of phase six of the Coupled Model Intercomparison Project (CMIP6). Comparing with the observation, although the CMIP6 models yield the essential fidelity, they still show considerable systematic biases in the amplitude and phase of the annual cycle, especially in east and south China. Most models exhibit substantial phase delays in the EOF-2 mode of the annual cycle. Some specific models (BCC-ESM1, CanESM5, and GFDL-CM4) exhibiting better performance could capture the observed annual cycle and the underlying physics in climatology and interannual variability. The limited fidelity of the EOF-2 mode of the EAM annual cycle primarily hinders the monsoon variability simulation and thus the reliable future projection. Therefore, the dominant modes of the EAM annual cycle act as the evaluate benchmark in the EAM modelling framework. Their improvement could be one possible bias correction strategy for decreasing the uncertainty in the CMIP6 simulation of the EAM.

Statistical inference for association studies using electronic health records: handling both selection bias and outcome misclassification.

Health research using electronic health records (EHR) has gained popularity, but misclassification of EHR-derived disease status and lack of representativeness of the study sample can result in substantial bias in effect estimates and can impact power and type I error. In this paper, we develop new strategies for handling disease status misclassification and selection bias in EHR-based association studies. We first focus on each type of bias separately. For misclassification, we propose three novel likelihood-based bias correction strategies. A distinguishing feature of the EHR setting is that misclassification may be related to patient-varying factors, and the proposed methods leverage data in the EHR to estimate misclassification rates without gold standard labels. For addressing selection bias, we describe how calibration and inverse probability weighting methods from the survey sampling literature can be extended and applied to the EHRsetting. Addressing misclassification and selection biases simultaneously is a more challenging problem than dealing with each on its own, and we propose several new strategies. For all methods proposed, we derive valid standard error estimators and provide software for implementation. We provide a new suite of statistical estimation and inference strategies for addressing misclassification and selection bias simultaneously that is tailored to problems arising in EHR data analysis. We apply these methods to data from The Michigan Genomics Initiative, a longitudinal EHR-linkedbiorepository.

Correcting lateral boundary biases in regional climate modelling: the effect of the relaxation zone

Regional climate models (RCM) are an important tool for simulating atmospheric information at finer resolutions often of greater relevance to local scale climate change impact assessment studies. The lateral and lower boundary conditions, which form the inputs to the RCM downscaling application, are outputs from the global climate model (GCM). These boundary variables are known to be biased in GCMs, providing the potential to use a statistical approach that corrects these biases before use in downscaling. An array of bias correction techniques have been developed to remove these biases before being used to drive the RCM, but questions remain on their efficacy in terms of the final downscaled output. This study assesses the impact of these bias correction strategies by focussing on how these corrections are translated as one proceeds from the lateral boundaries into the model interior. Of specific interest is the change in the correction from generation of the lateral boundary conditions as well as how correction information moves through the relaxation zone and into the interior of the model. Here we show that bias correction information passing into the regional climate model is limited by interpolations required to generate lateral boundary conditions and dominant outflow wind conditions in the boundaries. This work suggests that these limitations should be addressed in order for bias correction of lateral boundary conditions to robustly influence RCM simulations of climate in the interior of the model domain.

Bias-Corrected Q-Learning With Multistate Extension

Q-learning is a sample-based model-free algorithm that solves Markov decision problems asymptotically, but in finite time, it can perform poorly when random rewards and transitions result in large variance of value estimates. We pinpoint its cause to be the estimation bias due to the maximum operator in Q-learning algorithm, and present the evidence of max-operator bias in its Q value estimates. We then present an asymptotically optimal bias-correction strategy and construct an extension to bias-corrected Q-learning algorithm to multistate Markov decision processes, with asymptotic convergence properties as strong as those from Q-learning. We report the empirical performance of the bias-corrected Q-learning algorithm with multistate extension in two model problems: A multiarmed bandit version of Roulette and an electricity storage control simulation. The bias-corrected Q-learning algorithm with multistate extension is shown to control max-operator bias effectively, where the bias-resistance can be tuned predictably by adjusting a correction parameter.

A variable step-size adaptive notch filter for frequency estimation using combined gradient algorithm

To improve the performance of adaptive notch filter (ANF), a variable step-size ANF using combined gradient algorithm is proposed for frequency estimation. In this method, combined gradient algorithm is designed for improving constringency with both FIR-ANF and IIR-ANF advantages considered. According to the noise influence, a bias correction strategy is established for depressing the bias and MSE of frequency estimation. Additionally, variable step-size is adopted for the balance of convergence and precision. The proposed method process is given. In simulations, we discussed the influence of the ANF parameters and SNR on accuracy. Algorithms of DPG and MPG are carried out as comparisons. Convergences of these methods are also analyzed. Coriolis mass flow meter (CMF) is taken as an application to test the proposed method. Simulation results and CMF application both confirm the availability and superiority of the proposed method.

Application of bias corrections to improve hub-height ensemble wind forecasts over the Tehachapi Wind Resource Area

This study demonstrates improvements in ensemble wind forecasts at hub height due to bias correction strategies and their impact on wind energy forecasts at the Alta II wind farm in southern California. The ensemble consists of twenty members that differ in physics schemes used. Ensemble wind forecasts are produced for three months. Hub-height sodar wind observations are used to evaluate forecast performance. Time-dependent bias correction (TBC) and probability bias correction (PBC) are proposed to calibrate hub-height ensemble wind forecasts.The root mean square errors (RMSEs) and biases of forecasted hub-height winds are significantly reduced using both bias correction methods. RMSEs were reduced by 20% and 15% for TBC and PBC, respectively. When evaluating forecast performance from a reliability perspective, PBC better corrects both high and low forecast probabilities for high-wind thresholds. The penalty associated with deviation of the wind energy forecasts from observed values is reduced by 8.7% and 8.0% for TBC and PBC, respectively. It is notable that PBC produces a greater penalty reduction during the high wind bias forecast period (6 p.m.–8 a.m. local standard time); during this time period, PBC reduces the penalty by 25.8% while TBC reduces it by just 19.2%.

Impacts of Assimilating Smartphone Pressure Observations on Forecast Skill during Two Case Studies in the Pacific Northwest

AbstractOver a half-billion smartphones are now capable of measuring atmospheric pressure, potentially providing a global surface observing network of unprecedented density and coverage. An earlier study by the authors described an Android app, uWx, that served as a test bed for advanced quality control and bias correction strategies. To evaluate the utility and quality of the resulting smartphone pressure observations, ensemble data assimilation experiments were performed for two case studies over the Pacific Northwest. In both case studies, smartphone pressures improved the analyses and forecasts of assimilated and nonassimilated variables. In case I, which considered the passage of a front across the region, cycled smartphone pressure assimilation consistently improved 1-h forecasts of the altimeter setting, 2-m temperature, and 2-m dewpoint. During a postfrontal period, cycled smartphone pressure assimilation improved mesoscale forecasts of hourly precipitation accumulation. In case II, which considered a major coastal windstorm, cycling experiments assimilating smartphone pressures improved 10-m wind forecasts as well as the predicted track and intensity. For both cases, free-forecast experiments initialized with smartphone data produced forecast improvements extending several hours, suggesting the utility of crowdsourced smartphone pressures for short-term numerical weather prediction.

Isotopic fractionation studies of uranium and plutonium using porous ion emitters as thermal ionization mass spectrometry sources

Investigations of the isotope fractionation behaviors of plutonium and uranium reference standards were conducted employing platinum and rhenium (Pt/Re) porous ion emitter (PIE) sources, a relatively new thermal ionization mass spectrometry (TIMS) ion source strategy. The suitability of commonly employed, empirically developed mass bias correction laws (i.e., the Linear, Power, and Russell's laws) for correcting such isotope ratio data was also determined. Corrected plutonium isotope ratio data, regardless of mass bias correction strategy, were statistically identical to that of the certificate, however, the process of isotope fractionation behavior of plutonium using the adopted experimental conditions was determined to be best described by the Power law. The fractionation behavior of uranium, using the analytical conditions described herein, is also most suitably modeled using the Power law, though Russell's and the Linear law for mass bias correction rendered results that were identical, within uncertainty, to the certificate value.

Assessment of Nanosecond Laser Ablation Multi‐Collector Inductively Coupled Plasma‐Mass Spectrometry for Pb and Sr Isotopic Determination in Archaeological Glass: Mass Bias Correction Strategies and Results for Corning Glass Reference Materials

This work presents an evaluation of various methods for in situ high‐precision Sr and Pb isotopic determination in archaeological glass (containing 100–500 μg g−1 target element) by nanosecond laser ablation multi‐collector‐inductively coupled plasma‐mass spectrometry (ns‐LA‐MC‐ICP‐MS). A set of four soda‐lime silicate glasses, Corning A–D, mimicking the composition of archaeological glass and produced by the Corning Museum of Glass (Corning, New York, USA), were investigated as candidates for matrix‐matched reference materials for use in the analysis of archaeological glass. Common geological reference materials with known isotopic compositions (USGS basalt glasses BHVO‐2G, GSE‐1G and NKT‐1G, soda‐lime silicate glass NIST SRM 610 and several archaeological glass samples with known Sr isotopic composition) were used to evaluate the ns‐LA‐MC‐ICP‐MS analytical procedures. When available, ns‐LA‐MC‐ICP‐MS results for the Corning glasses are reported. These were found to be in good agreement with results obtained via pneumatic nebulisation (pn) MC‐ICP‐MS after digestion of the glass matrix and target element isolation. The presence of potential spectral interference from doubly charged rare earth element (REE) ions affecting Sr isotopic determination was investigated by admixing Er and Yb aerosols by means of pneumatic nebulisation into the gas flow from the laser ablation system. It was shown that doubly charged REE ions affect the Sr isotope ratios, but that this could be circumvented by operating the instrument at higher mass resolution. Multiple strategies to correct for instrumental mass discrimination in ns‐LA‐MC‐ICP‐MS and the effects of relevant interferences were evaluated. Application of common glass reference materials with basaltic matrices for correction of ns‐LA‐MC‐ICP‐MS isotope data of archaeological glasses results in inaccurate Pb isotope ratios, rendering application of matrix‐matched reference materials indispensable. Correction for instrumental mass discrimination using the exponential law, with the application of Tl as an internal isotopic standard element introduced by pneumatic nebulisation and Corning D as bracketing isotopic calibrator, provided the most accurate results for Pb isotope ratio measurements in archaeological glass. Mass bias correction relying on the power law, combined with intra‐element internal correction, assuming a constant 88Sr/86Sr ratio, yielded the most accurate results for 87Sr/86Sr determination in archaeological glasses

Use of bias correction techniques to improve seasonal forecasts for reservoirs — A case-study in northwestern Mediterranean

In this paper, we have compared different bias correction methodologies to assess whether they could be advantageous for improving the performance of a seasonal prediction model for volume anomalies in the Boadella reservoir (northwestern Mediterranean). The bias correction adjustments have been applied on precipitation and temperature from the European Centre for Middle-range Weather Forecasting System 4 (S4). We have used three bias correction strategies: two linear (mean bias correction, BC, and linear regression, LR) and one non-linear (Model Output Statistics analogs, MOS-analog). The results have been compared with climatology and persistence. The volume-anomaly model is a previously computed Multiple Linear Regression that ingests precipitation, temperature and in-flow anomaly data to simulate monthly volume anomalies. The potential utility for end-users has been assessed using economic value curve areas. We have studied the S4 hindcast period 1981–2010 for each month of the year and up to seven months ahead considering an ensemble of 15 members. We have shown that the MOS-analog and LR bias corrections can improve the original S4. The application to volume anomalies points towards the possibility to introduce bias correction methods as a tool to improve water resource seasonal forecasts in an end-user context of climate services. Particularly, the MOS-analog approach gives generally better results than the other approaches in late autumn and early winter.

All-Sky Microwave Radiance Assimilation in NCEP’s GSI Analysis System

AbstractThe capability of all-sky microwave radiance assimilation in the Gridpoint Statistical Interpolation (GSI) analysis system has been developed at the National Centers for Environmental Prediction (NCEP). This development effort required the adaptation of quality control, observation error assignment, bias correction, and background error covariance to all-sky conditions within the ensemble–variational (EnVar) framework. The assimilation of cloudy radiances from the Advanced Microwave Sounding Unit-A (AMSU-A) microwave radiometer for ocean fields of view (FOVs) is the primary emphasis of this study.In the original operational hybrid 3D EnVar Global Forecast System (GFS), the clear-sky approach for radiance data assimilation is applied. Changes to data thinning and quality control have allowed all-sky satellite radiances to be assimilated in the GSI. Along with the symmetric observation error assignment, additional situation-dependent observation error inflation is employed for all-sky conditions. Moreover, in addition to the current radiance bias correction, a new bias correction strategy has been applied to all-sky radiances. In this work, the static background error variance and the ensemble spread of cloud water are examined, and the levels of cloud variability from the ensemble forecast in single- and dual-resolution configurations are discussed. Overall, the all-sky approach provides more realistic simulated brightness temperatures and cloud water analysis increments, and improves analysis off the west coasts of the continents by reducing a known bias in stratus. An approximate 10% increase in the use of AMSU-A channels 1–5 and a 12% increase for channel 15 are also observed. The all-sky AMSU-A radiance assimilation became operational in the 4D EnVar GFS system upgrade of 12 May 2016.