Bias Propagation Research Articles

Toward the reliable use of reanalysis data as a reference for bias correction in climate models: A multivariate perspective

General circulation models are vital tools in climate change research, profoundly influencing numerous hydroclimatological studies, but often exhibit systematic biases due to limitations in parameterization. To correct these biases, bias correction (BC) methods, which are heavily dependent on the quality of historical reference data, are employed. Ideally, accurate data comes from a dense network of gauges, but due to their limited availability, reanalysis data is frequently used instead. Although reanalysis data provides consistent global and temporal coverage, its assimilated nature can introduce additional biases, potentially increasing error propagation particularly in the multivariate perspective. Thus, this study investigates the reliability of reanalysis data as a reference for multivariate bias correction (MBC) and utilizes a two-stage bias correction approach that integrates both gauge and reanalysis data to leverage their respective strengths. Conducting a case study in South Korea, we utilized gauge data, ERA5 reanalysis data, and outputs from 10 CMIP6 General Circulation Models. We implemented three-nested experiments to test the robustness of this two-stage approach across various dimensions. Results show that using reanalysis data as a reference can lead to bias propagation, especially misrepresenting precipitation and wind dependencies in high elevation areas. However, the two-stage approach has shown an enhancement in mitigating these limitations, even when gauge data is temporally sparse, and across different combinations of MBC methods. Finally, we highlight the current limitations and potential for future research to focus on effectively combining gauge and reanalysis data to compensate for each other’s weaknesses, thereby enhancing the accuracy and reliability in future climate realizations.

Development of a differential treatment selection model for depression on consolidated and transformed clinical trial datasets

Major depressive disorder (MDD) is the leading cause of disability worldwide, yet treatment selection still proceeds via “trial and error”. Given the varied presentation of MDD and heterogeneity of treatment response, the use of machine learning to understand complex, non-linear relationships in data may be key for treatment personalization. Well-organized, structured data from clinical trials with standardized outcome measures is useful for training machine learning models; however, combining data across trials poses numerous challenges. There is also persistent concern that machine learning models can propagate harmful biases. We have created a methodology for organizing and preprocessing depression clinical trial data such that transformed variables harmonized across disparate datasets can be used as input for feature selection. Using Bayesian optimization, we identified an optimal multi-layer dense neural network that used data from 21 clinical and sociodemographic features as input in order to perform differential treatment benefit prediction. With this combined dataset of 5032 individuals and 6 drugs, we created a differential treatment benefit prediction model. Our model generalized well to the held-out test set and produced similar accuracy metrics in the test and validation set with an AUC of 0.7 when predicting binary remission. To address the potential for bias propagation, we used a bias testing performance metric to evaluate the model for harmful biases related to ethnicity, age, or sex. We present a full pipeline from data preprocessing to model validation that was employed to create the first differential treatment benefit prediction model for MDD containing 6 treatment options.

Real-Time LEO Satellite Clocks Based on Near-Real-Time Clock Determination with Ultra-Short-Term Prediction

The utilization of Low Earth Orbit (LEO) satellites is anticipated to augment various aspects of traditional GNSS-based Positioning, Navigation, and Timing (PNT) services. While the LEO satellite orbital products can nowadays be produced with rather high accuracy in real-time of a few centimeters, the precision of the LEO satellite clock products that can be achieved in real-time is less studied. The latter, however, plays an essential role in the LEO-augmented positioning and timing performances. In real-time, the users eventually use the predicted LEO satellite clocks, with their precision determined by both the near-real-time clock precision and the prediction time needed to match the time window for real-time applications, i.e., the precision loss during the prediction phase. In this study, a real-time LEO satellite clock determination method, consisting of near-real-time clock determination with ultra-short-term clock prediction is proposed and implemented. The principles and strategies of this method are discussed in detail. The proposed method utilized Kalman-filter-based processing, but supports restarts at pre-defined times, thus hampering continuous bias propagation and accumulation from ancient epochs. Based on the method, using Sentinel-3B GNSS observations and the real-time GNSS products from the National Center for Space Studies (CNES) in France, the near-real-time LEO satellite clocks can reach a precision of 0.2 to 0.3 ns, and the precision loss during the prediction phase is within 0.07 ns for a prediction time window from 30 to 90 s. This results in a total error budget in the real-time LEO satellite clocks of about 0.3 ns.

Reproducing Popularity Bias in Recommendation: The Effect of Evaluation Strategies

The extent to which popularity bias is propagated by media recommender systems is a current topic within the community, as is the uneven propagation among users with varying interests for niche items. Recent work focused on exactly this topic, with movies being the domain of interest. Later on, two different research teams reproduced the methodology in the domains of music and books, respectively. The results across the different domains diverge. In this paper, we reproduce the three studies and identify four aspects that are relevant in investigating the differences in results: data, algorithms, division of users in groups and evaluation strategy. We run a set of experiments in which we measure general popularity bias propagation and unfair treatment of certain users with various combinations of these aspects. We conclude that all aspects account to some degree for the divergence in results, and should be carefully considered in future studies. Further, we find that the divergence in findings can be in large part attributed to the choice of evaluation strategy.

Tiny, Always-on, and Fragile: Bias Propagation through Design Choices in On-device Machine Learning Workflows

Billions of distributed, heterogeneous, and resource constrained IoT devices deploy on-device machine learning (ML) for private, fast, and offline inference on personal data. On-device ML is highly context dependent and sensitive to user, usage, hardware, and environment attributes. This sensitivity and the propensity toward bias in ML makes it important to study bias in on-device settings. Our study is one of the first investigations of bias in this emerging domain and lays important foundations for building fairer on-device ML. We apply a software engineering lens, investigating the propagation of bias through design choices in on-device ML workflows. We first identify reliability bias as a source of unfairness and propose a measure to quantify it. We then conduct empirical experiments for a keyword spotting task to show how complex and interacting technical design choices amplify and propagate reliability bias . Our results validate that design choices made during model training, like the sample rate and input feature type, and choices made to optimize models, like light-weight architectures, the pruning learning rate, and pruning sparsity, can result in disparate predictive performance across male and female groups. Based on our findings, we suggest low effort strategies for engineers to mitigate bias in on-device ML.

Bias propagation in network meta-analysis models.

Network meta-analysis combines direct and indirect evidence to compare multiple treatments. As direct evidence for one treatment contrast may be indirect evidence for other treatment contrasts, biases in the direct evidence for one treatment contrast may affect not only the estimate for this particular treatment contrast but also estimates of other treatment contrasts. Because network structure determines how direct and indirect evidence are combined and weighted, the impact of biased evidence will be determined by the network geometry. Thus, this study's aim was to investigate how the impact of biased evidence spreads across the whole network and how the propagation of bias is influenced by the network structure. In addition to the popular Lu & Ades model, we also investigate bias propagation in the baseline model and arm-based model to compare the effects of bias in the different models. We undertook extensive simulations under different scenarios to explore how the impact of bias may be affected by the location of the bias, network geometry and the statistical model. Our results showed that the structure of a network has an important impact on how the bias spreads across the network, and this is especially true for the Lu & Ades model. The impact of bias is more likely to be diluted by other unbiased evidence in a well-connected network. We also used a real network meta-analysis to demonstrate how to use the new knowledge about bias propagation to explain questionable results from the original analysis.

Decentralised Data Quality Control in Ground Truth Production for Autonomic Decisions

Autonomic decision-making based on rules and metrics is inevitably on the rise in distributed software systems. Often, the metrics are acquired from system observations such as static checks and runtime traces. To avoid bias propagation and hence reduce wrong decisions in increasingly autonomous systems due to poor observation data quality, multiple independent observers can exchange their findings and produce a majority-accepted, complete and outlier-cleaned ground truth in the form of consensus-supported metrics. In this work, we motivate the growing importance of metrics for informed and autonomic decisions in clouds and other distributed systems, present reasons for diverging observations, and describe a federated approach to produce ground truth with data-centric consensus voting for more reliable decision making processes. We validate the system design with experiments in the area of cloud software artefact observations and highlight benefits for reproducible distributed system behaviour.

Effects of spatial scale of atmospheric reanalysis data on clear-sky surface radiation modeling in tropical climates: A case study for Singapore

Solar resource assessments most generally require atmospheric information, which is customarily acquired from gridded datasets. The spatial scale mismatch problem, i.e., the difference in spatial representativeness of gridded data and in situ measurements, therefore becomes relevant. This study examines how the gridded data used as inputs to clear-sky radiation models can affect their performance at urban scale. The tropical island of Singapore is selected for the case study. Aerosol optical depth at 550 nm (AOD550), Å ngström exponent (AE), and precipitable water (PW) from both the MERRA-2 reanalysis and ground-based stations (AERONET and SuomiNet) are collected between 2013–2020. Firstly, it is found that, relatively to the AERONET ground truth, the bias in MERRA-2’s AOD550 is more prominent than that in AE or PW. Next, the bias propagation from the gridded inputs (AOD550, AE, and PW) to clear-sky radiation predictions is explored using various models. The estimated clear-sky direct normal irradiance (DNIcs) is more sensitive to AOD550 variation than the clear-sky global horizontal irradiance (GHIcs). Six clear-sky radiation models, five of which accept MERRA-2 gridded inputs, are compared with each other, and with the in situ irradiance measurements recorded at 9 sites. The inter-model difference across Singapore is remarkably consistent because the whole island fits inside a single MERRA-2 grid cell. Under high-AOD550 situations, however, the inter-model deviation becomes large for both GHIcs and DNIcs. The conventional model-versus-measurement comparison shows that each model achieves very different site-to-site performance, largely because the spatially-averaged inputs cannot fully represent the micro-climatic variability. Relatively speaking, no clear-sky radiation model significantly outperforms its peers. The simple MAC2 model and the empirical (locally derived) Yang GHIcs-only model are recommended for Singapore.

Which gridded irradiance data is best for modelling photovoltaic power production in Germany?

Model estimates of expected photovoltaic (PV) power production rely on accurate irradiance data. Reanalysis and satellite products freely provide irradiance data with a high temporal and spatial resolution including locations for which no ground-based measurements are available. We assess differences in such gridded irradiance data and quantify the subsequent bias propagation from individual radiation components to capacity factors in a contemporary PV model. PV power production is simulated based on four reanalysis (ERA5, COSMO-REA6, COSMO-REA6pp, COSMO-REA2) and three satellite products (CAMS, SARAH-2, CERES Syn1Deg). The results are compared against simulations using measurements from 30 weather stations of the German Weather Service. We compute metrics characterizing biases in seasonal and annual means, day-to-day variability and extremes in PV power. Our results highlight a bias of −1.4% (COSMO-REA6) to ＋8.2% (ERA5) in annual and spatial means of PV power production for Germany. No single data set is best in all metrics, although SARAH-2 and the postprocessed COSMO-REA6 data (COSMO-REA6pp) outperform the other products for many metrics. SARAH-2 yields good results in summer, but overestimates PV output in winter by 16% averaged across all stations. COSMO-REA6pp represents day-to-day variability in the PV power production of a simulated PV fleet best and has a particularly small bias of 0.5% in annual means. This is at least in parts due to compensating biases in local and seasonal means. Our results imply that gridded irradiance data should be used with caution for site assessments and ideally be complemented by local measurements.

Parameters' Confidence Intervals Evaluation for Heterogeneous Strain Field Specimen Designs by Using Digital Image Correlation

This paper aims to compare different heterogeneous test designs from the perspective of the confidence interval quantification of inversely identified parameters, where the influence of a DIC optical system systematic and random error are taken into account. Because the errors in optical measurement can arise from many reasons and sources, our methodology relies on the system's errors determined from initial sets of pictures acquired at the load-free state for hundreds of specimens (over 850 tests over the past three years). In this way, a prior probability distribution of systematic and random error, arisen from the system initial settings and testing procedures are determined. Further, by conducting an inverse identification procedure of linear orthotropic elastic material parameters, the influence of the error distributions is studied for different types of heterogeneous specimens. The presented methodology determines the DIC bias and random error propagation through the inverse identification procedure to individual parameters. For each specimen design, confidence intervals of identified material parameters were determined. The results show the appropriateness of a specimen design for the identification of particular material parameters.

Innovation Feedback Kalman Filter

The conventional Kalman filter demands complete knowledge of the actual filter model, which is usually inaccessible in practical problems. When dealing with the filtering problem with uncertainty, any improper prediction of the uncertainty may degrade the filtering performance and even lead to divergence phenomenon. In this paper, the innovation feedback Kalman filter, which introduces the innovation feedback controller to the Kalman filter equations, is newly proposed on the basis of automatic control theory to address the filtering problem with uncertainty and is different from other filtering methods. By studying the estimate error equations, the estimate bias is extracted and its propagation mechanism is formulated. The estimate bias propagation equations reveal that eliminating the estimate bias essentially equivalents to an output regulation problem with uncertain exosystem. And a concise yet effective innovation feedback controller is subsequently given as an example. The proposed method is applied to one-dimension target tracking scenario and its high estimation accuracy performance and good stability are simulated through a comparative analysis with the ideal filtering results.

Propagation of representation bias in machine learning

Transfer learning is an emerging paradigm in machine learning that involves reusing existing pretrained models to develop new machine learning applications. As machine learning grows in importance, keeping new models and applications clear of biases is of paramount importance. While previous works focus on the development of bias free models, they fail to address the mechanism of bias propagation in transfer learning. Using facial recognition as a use-case scenario, we attempt to identify sources of bias in a model developed using transfer learning. To achieve this task, we developed a model based on a pre-trained facial recognition model, and scrutinized the accuracy of the model’s image classification against factors such as age, gender, and race to observe whether or not the model performed better on some demographic groups than others. By identifying the bias and finding potential sources of bias, his work contributes a unique technical perspective from the view of a small scale developer to emerging discussions of accountability and transparency in AI.

Bias Propagation and Estimation in Homogeneous Differentiators for a Class of Mechanical Systems

Motivated by non-anthropomorphic dynamic stabilization of a walking robot, we consider the bias propagation problem for a homogeneous nonlinear model-based differentiator applied to a reaction wheel pendulum with a biased position sensor. We show that the bias propagates through the velocity observer and compromises the vertical stabilization. To cancel the impact of the bias, we propose to augment the differentiator with a reduced-order bias observer. Local asymptotic stability of the augmented nonlinear observer is shown, where the observer gain can be tuned using matrix inequalities. Experimental results illustrate the applicability of the proposed solution.

GEOVISUAL ANALYTICS ON THE VERIFICATION OF THE PAGASA OPERATIONAL NUMERICAL WEATHER PREDICTION MODEL RAINFALL FORECAST

Abstract. Assessment of NWP model performance is an integral part of operational forecasting as well as in research and development. Understanding the bias propagation of an NWP model and how it propagates across space can provide more insight in determining underlying causes and weaknesses not easily determined in traditional methods. The study aims to introduce the integration of the spatial distribution of error in interpreting model verification results by assessing how well the operational numerical weather prediction system of PAGASA captures the country’s weather pattern in each of its climate type. It also discusses improvements in model performance throughout the time-frame of analysis. Error propagation patterns were identified using Geovisual Analytics to allow comparison of verification scores among individual stations. The study concluded that a major update in the physics parameterization of the model in 2016 and continued minor updates in the following years, surface precipitation forecasts greatly improved from an average RMSE of 9.3, MAE of 3.2 and Bias of 1.36 in 2015 to an RMSE of 7.9, MAE of 2.5 and bias of −0.63 in 2018.

Matrix embedding in steganography with binary Reed–Muller codes

This study presents a modified majority-logic decoding algorithm of Reed-Muller (RM) codes for matrix embedding (ME) in steganography. An ME algorithm uses linear block code to improve the embedding efficiency in steganography. The optimal embedding algorithm in steganography is equivalent to the maximum likelihood decoding (MLD) algorithm in error-correcting codes. The main disadvantage of ME is that the equivalent MLD algorithm of lengthy embedding codes requires highly complex embedding. This study used RM codes to embed data in binary host images. The authors propose a novel low-complexity embedding algorithm that uses a modified majority-logic algorithm to decode RM codes, in which a message-passing algorithm (i.e. sum-product, min-sum, or bias propagation) is performed on the highest order of information bits in the RM codes. The experimental results indicate that integrating bias propagation into the proposed scheme achieves superior embedding efficiency (relative to when the sum-product or min-sum algorithm is used) and can even achieve the embedding bound of RM codes.

Bias-correction of Kalman filter estimators associated to a linear state space model with estimated parameters

This paper aims to discuss some practical problems on linear state space models with estimated parameters. While the existing research focuses on the prediction mean square error of the Kalman filter estimators, this work presents some results on bias propagation into both one-step ahead and update estimators, namely, non recursive analytical expressions for them. In particular, it is discussed the impact of the bias in the invariant state space models. The theoretical results presented in this work provide an adaptive correction procedure based on any parameters estimation method (for instance, maximum likelihood or distribution-free estimators). This procedure is applied to two data set: in the calibration of radar precipitation estimates and in the global mean land–ocean temperature index modeling.

Mid-21st century air quality at the urban scale under the influence of changed climate and emissions – case studies for Paris and Stockholm

Abstract. Ozone, PM10 and PM2.5 concentrations over Paris, France and Stockholm, Sweden were modelled at 4 and 1 km horizontal resolutions respectively for the present and 2050 periods employing decade-long simulations. We account for large-scale global climate change (RCP-4.5) and fine-resolution bottom-up emission projections developed by local experts and quantify their impact on future pollutant concentrations. Moreover, we identify biases related to the implementation of regional-scale emission projections by comparing modelled pollutant concentrations between the fine- and coarse-scale simulations over the study areas. We show that over urban areas with major regional contribution (e.g. the city of Stockholm) the bias related to coarse-scale projections may be significant and lead to policy misclassification. Our results stress the need to better understand the mechanism of bias propagation across the modelling scales in order to design more successful local-scale strategies. We find that the impact of climate change is spatially homogeneous in both regions, implying strong regional influence. The climate benefit for ozone (daily mean and maximum) is up to −5 % for Paris and −2 % for Stockholm city. The climate benefit on PM2.5 and PM10 in Paris is between −5 and −10 %, while for Stockholm we estimate mixed trends of up to 3 % depending on season and size class. In Stockholm, emission mitigation leads to concentration reductions up to 15 % for daily mean and maximum ozone and 20 % for PM. Through a sensitivity analysis we show that this response is entirely due to changes in emissions at the regional scale. On the contrary, over the city of Paris (VOC-limited photochemical regime), local mitigation of NOx emissions increases future ozone concentrations due to ozone titration inhibition. This competing trend between the respective roles of emission and climate change, results in an increase in 2050 daily mean ozone by 2.5 % in Paris. Climate and not emission change appears to be the most influential factor for maximum ozone concentration over the city of Paris, which may be particularly interesting from a health impact perspective.

Optimism Propagation

We document that managerial biases spread across firms along supply chains. Supporting a causal interpretation, we show that beliefs trickle up the supply chain, not down, and that biases in supplier forecasts are only affected by customer forecasts issued before, not after, the supplier's forecast. We further find that bias propagation increases when suppliers have less confidence in their own views and when the perceived precision and importance of customer forecasts increase. Biases cause changes in the corporate policies of suppliers, suggesting that contagious beliefs in production networks contribute to fluctuations of business and financing cycles.

Error covariance calculation for forecast bias estimation in hydrologic data assimilation

To date, an outstanding issue in hydrologic data assimilation is a proper way of dealing with forecast bias. A frequently used method to bypass this problem is to rescale the observations to the model climatology. While this approach improves the variability in the modeled soil wetness and discharge, it is not designed to correct the results for any bias. Alternatively, attempts have been made towards incorporating dynamic bias estimates into the assimilation algorithm. Persistent bias models are most often used to propagate the bias estimate, where the a priori forecast bias error covariance is calculated as a constant fraction of the unbiased a priori state error covariance. The latter approach is a simplification to the explicit propagation of the bias error covariance. The objective of this paper is to examine to which extent the choice for the propagation of the bias estimate and its error covariance influence the filter performance. An Observation System Simulation Experiment (OSSE) has been performed, in which ground water storage observations are assimilated into a biased conceptual hydrologic model. The magnitudes of the forecast bias and state error covariances are calibrated by optimizing the innovation statistics of groundwater storage. The obtained bias propagation models are found to be identical to persistent bias models. After calibration, both approaches for the estimation of the forecast bias error covariance lead to similar results, with a realistic attribution of error variances to the bias and state estimate, and significant reductions of the bias in both the estimates of groundwater storage and discharge. Overall, the results in this paper justify the use of the traditional approach for online bias estimation with a persistent bias model and a simplified forecast bias error covariance estimation.

Application of adaptive kinetic modelling for bias propagation reduction in direct 4D image reconstruction

Parametric imaging in thoracic and abdominal PET can provide additional parameters more relevant to the pathophysiology of the system under study. However, dynamic data in the body are noisy due to the limiting counting statistics leading to suboptimal kinetic parameter estimates. Direct 4D image reconstruction algorithms can potentially improve kinetic parameter precision and accuracy in dynamic PET body imaging. However, construction of a common kinetic model is not always feasible and in contrast to post-reconstruction kinetic analysis, errors in poorly modelled regions may spatially propagate to regions which are well modelled. To reduce error propagation from erroneous model fits, we implement and evaluate a new approach to direct parameter estimation by incorporating a recently proposed kinetic modelling strategy within a direct 4D image reconstruction framework. The algorithm uses a secondary more general model to allow a less constrained model fit in regions where the kinetic model does not accurately describe the underlying kinetics. A portion of the residuals then is adaptively included back into the image whilst preserving the primary model characteristics in other well modelled regions using a penalty term that trades off the models. Using fully 4D simulations based on dynamic [15O]H2O datasets, we demonstrate reduction in propagation-related bias for all kinetic parameters. Under noisy conditions, reductions in bias due to propagation are obtained at the cost of increased noise, which in turn results in increased bias and variance of the kinetic parameters. This trade-off reflects the challenge of separating the residuals arising from poor kinetic modelling fits from the residuals arising purely from noise. Nonetheless, the overall root mean square error is reduced in most regions and parameters. Using the adaptive 4D image reconstruction improved model fits can be obtained in poorly modelled regions, leading to reduced errors potentially propagating to regions of interest which the primary biologic model accurately describes. The proposed methodology, however, depends on the secondary model and choosing an optimal model on the residual space is critical in improving model fits.