Bias-variance Research Articles

Weight bias among Canadians: Associations with sociodemographics, BMI and body image constructs

This cross-sectional study examined the associations between sociodemographic characteristics, BMI, and body image constructs (body satisfaction and weight bias internalization; WBI) and explicit weight bias. A near-representative sample of 995 English-speaking Canadian adults (52% Female) completed a survey which assessed explicit weight bias (Anti-Fat Attitudes questionnaire), body satisfaction (Body Shape Satisfaction Scale), WBI (Modified Weight Bias Internalization Scale), and self-reported height and weight. Multiple linear regression analyses were run. Results showed that the variable that explained the most variance in explicit weight bias was WBI, followed by BMI. Higher levels of WBI and a lower BMI were both significantly associated with greater explicit weight bias. Male sex was associated with both disliking people with obesity and thinking obesity is attributable to lack of willpower, whereas female sex was associated with worrying about weight gain. The current findings emphasize the importance of future research efforts aimed at preventing or mitigating WBI to reduce negative attitudes about people with obesity.

Markets matter: a simulation study of the bias-variance trade-off in comparison group selection for difference-in-differences analysis

Markets matter: a simulation study of the bias-variance trade-off in comparison group selection for difference-in-differences analysis

Theoretical Evaluation of Machine Learning and Deep Learning Applications in Various Domain

The theoretical evaluation of machine learning (ML) and deep learning (DL) applications encompasses the rigorous analysis of their mathematical foundations, algorithmic principles, and performance metrics. This evaluation aims to understand the capabilities and limitations of various ML and DL models, including their generalization ability, convergence properties, and computational efficiency. By exploring theoretical aspects such as bias-variance tradeoff, overfitting, underfitting, and the impact of hyperparameters, researchers can optimize model architectures and training processes. Additionally, the theoretical examination provides insights into the interpretability and robustness of models, guiding the development of more reliable and efficient applications across diverse domains such as computer vision, natural language processing, and predictive analytics.

GBVSSL: Contrastive Semi-Supervised Learning Based on Generalized Bias-Variance Decomposition

Mainstream semi-supervised learning (SSL) techniques, such as pseudo-labeling and contrastive learning, exhibit strong generalization abilities but lack theoretical understanding. Furthermore, pseudo-labeling lacks the label enhancement from high-quality neighbors, while contrastive learning ignores the supervisory guidance provided by genuine labels. To this end, we first introduce a generalized bias-variance decomposition framework to investigate them. Then, this research inspires us to propose two new techniques to refine them: neighbor-enhanced pseudo-labeling, which enhances confidence-based pseudo-labels by incorporating aggregated predictions from high-quality neighbors; label-enhanced contrastive learning, which enhances feature representation by combining enhanced pseudo-labels and ground-truth labels to construct a reliable and complete symmetric adjacency graph. Finally, we combine these two new techniques to develop an excellent SSL method called GBVSSL. GBVSSL significantly surpasses previous state-of-the-art SSL approaches in standard benchmarks, such as CIFAR-10/100, SVHN, and STL-10. On CIFAR-100 with 400, 2500, and 10,000 labeled samples, GBVSSL outperforms FlexMatch by 3.46%, 2.72%, and 2.89%, respectively. On the real-world dataset Semi-iNat 2021, GBVSSL improves the Top-1 accuracy over CCSSL by 4.38%. Moreover, GBVSSL exhibits faster convergence and enhances unbalanced SSL. Extensive ablation and qualitative studies demonstrate the effectiveness and impact of each component of GBVSSL.

Considerations and targeted approaches to identifying bad actors in exposure mixtures.

Variable importance is a key statistical issue in exposure mixtures, as it allows a ranking of exposures as potential targets for intervention, and helps to identify bad actors within a mixture. In settings where mixtures have many constituents or high between-constituent correlations, estimators of importance can be subject to bias or high variance. Current approaches to assessing variable importance have major limitations, including reliance on overly strong or incorrect constraints or assumptions, excessive model extrapolation, or poor interpretability, especially regarding practical significance. We sought to overcome these limitations by applying an established doubly-robust, machine learning-based approach to estimating variable importance in a mixtures context. This method reduces model extrapolation, appropriately controls confounding, and provides both interpretability and model flexibility. We illustrate its use with an evaluation of the relationship between telomere length, a measure of biologic aging, and exposure to a mixture of polychlorinated biphenyls (PCBs), dioxins, and furans among 979 US adults from the National Health and Nutrition Examination Survey (NHANES). In contrast with standard approaches for mixtures, our approach selected PCB 180 and PCB 194 as important contributors to telomere length. We hypothesize that this difference could be due to residual confounding in standard methods that rely on variable selection. Further empirical evaluation of this method is needed, but it is a promising tool in the search for bad actors within a mixture.

Modeling Bellman-error with logistic distribution with applications in reinforcement learning

In modern Reinforcement Learning (RL) approaches, optimizing the Bellman error is a critical element across various algorithms, notably in deep Q-Learning and related methodologies. Traditional approaches predominantly employ the mean-squared Bellman error (MSELoss) as the standard loss function. However, the assumption of Bellman errors following the Gaussian distribution may oversimplify the nuanced characteristics of RL applications. In this work, we revisit the distribution of Bellman error in RL training, demonstrating that it tends to follow the Logistic distribution rather than the commonly assumed Normal distribution. We propose replacing MSELoss with a Logistic maximum likelihood function (LLoss) and rigorously test this hypothesis through extensive numerical experiments across diverse online and offline RL environments. Our findings consistently show that integrating the Logistic correction into the loss functions of various baseline RL methods leads to superior performance compared to their MSE counterparts. Additionally, we employ Kolmogorov–Smirnov tests to substantiate that the Logistic distribution offers a more accurate fit for approximating Bellman errors. This study also offers a novel theoretical contribution by establishing a clear connection between the distribution of Bellman error and the practice of proportional reward scaling, a common technique for performance enhancement in RL. Moreover, we explore the sample-accuracy trade-off involved in approximating the Logistic distribution, leveraging the Bias–Variance decomposition to mitigate excessive computational resources. The theoretical and empirical insights presented in this study lay a significant foundation for future research, potentially advancing methodologies, and understanding in RL, particularly in the distribution-based optimization of Bellman error.

Hybrid Thresholding for Image Deconvolution in Expectation Maximization framework

ABSTRACT In this study, we proposed an image deconvolution method in the expectation maximization (EM) framework. This method involves two steps: (i) E-step: utilizing the fast Fourier transform (FFT) for computationally efficient inversion of the convolution operator and (ii) M-step: employing the discrete wavelet transform (DWT) for estimating the original image from the image obtained in the E-step. In M-step, we proposed a modified L1-clipped penalty resulting in a hybrid thresholding scheme that integrates conventional hard and soft thresholds. This hybrid threshold ameliorates the inherent bias-variance trade-offs associated with traditional hard and soft thresholding schemes. The mathematical expressions for the risk, bias, and variance of the proposed hybrid threshold are derived and the performance is evaluated through simulation. Experimental results show that the proposed method achieves optimal values for variance and bias, thereby minimizing the risk. Moreover, the proposed method outperforms state-of-the-art methods in terms of performance metrics: PSNR, ISNR, and SSIM.

Exploring the Performance of Linear and Nonlinear Models of Time-of-Flight Secondary Ion Mass Spectrometry Spectra.

Multivariate statistical tools and machine learning (ML) techniques can deconvolute hyperspectral data and control the disparity between the number of samples and features in materials science. Nevertheless, the importance of generating sufficient high-quality sample replicates in training data cannot be overlooked, as it fundamentally affects the performance of ML models. Here, we present a quantitative analysis of time-of-flight secondary ion mass spectrometry (ToF-SIMS) spectra of a simple microarray system of two food dyes using partial least-squares (PLS, linear) and random forest (RF, nonlinear) algorithms. This microarray was generated by a high-throughput sample preparation and analysis workflow for fast and efficient acquisition of quality and reproducible spectra via ToF-SIMS. We drew insights from the bias-variance trade-off, investigated the performances of PLS and RF regression models as a function of training data size, and inferred the amount of data needed to construct accurate and reliable regression models. In addition, we found that the spectral concatenation of positive and negative ToF-SIMS spectra improved the model performances. This study provides an empirical basis for future design of high-throughput microarrays and multicomponent systems, for the purpose of analysis with ToF-SIMS and ML.

Optimization of CO2 capture plants with surrogate model uncertainties

CO2 capture plants can help reduce the cost of deploying capture systems across the globe. However, the CO2 variability and model uncertainty represent operational challenges to capture CO2 from different sources. This work proposes a framework for analyzing the optimal plant design considering different flue gas sources. We show a methodology to generate large data sets from optimization runs using rigorous models in Aspen Plus®. The efficiency of the approach allows its application to large-scale optimization problems, with an average CPU time per run of 176 s.We additionally build surrogate models (SMs) for the capital and operating costs of the capture plants, employing an iterative procedure to generate SMs using ALAMO. We systematically reject SMs with high uncertainty in the estimated parameters. This approach results in SMs with favorable bias-variance tradeoffs, enabling their effective application to optimization problems under uncertainty, as demonstrated with a pooling problem of CO2 streams.

Exponential family measurement error models for single-cell CRISPR screens.

CRISPR genome engineering and single-cell RNA sequencing have accelerated biological discovery. Single-cell CRISPR screens unite these two technologies, linking genetic perturbations in individual cells to changes in gene expression and illuminating regulatory networks underlying diseases. Despite their promise, single-cell CRISPR screens present considerable statistical challenges. We demonstrate through theoretical and real data analyses that a standard method for estimation and inference in single-cell CRISPR screens-"thresholded regression"-exhibits attenuation bias and a bias-variance tradeoff as a function of an intrinsic, challenging-to-select tuning parameter. To overcome these difficulties, we introduce GLM-EIV ("GLM-based errors-in-variables"), a new method for single-cell CRISPR screen analysis. GLM-EIV extends the classical errors-in-variables model to responses and noisy predictors that are exponential family-distributed and potentially impacted by the same set of confounding variables. We develop a computational infrastructure to deploy GLM-EIV across hundreds of processors on clouds (e.g. Microsoft Azure) and high-performance clusters. Leveraging this infrastructure, we apply GLM-EIV to analyze two recent, large-scale, single-cell CRISPR screen datasets, yielding several new insights.

DeepReg: a deep learning hybrid model for predicting transcription factors in eukaryotic and prokaryotic genomes

Deep learning models (DLMs) have gained importance in predicting, detecting, translating, and classifying a diversity of inputs. In bioinformatics, DLMs have been used to predict protein structures, transcription factor-binding sites, and promoters. In this work, we propose a hybrid model to identify transcription factors (TFs) among prokaryotic and eukaryotic protein sequences, named Deep Regulation (DeepReg) model. Two architectures were used in the DL model: a convolutional neural network (CNN), and a bidirectional long-short-term memory (BiLSTM). DeepReg reached a precision of 0.99, a recall of 0.97, and an F1-score of 0.98. The quality of our predictions, the bias-variance trade-off approach, and the characterization of new TF predictions were evaluated and compared against those produced by DeepTFactor, as well as against experimental data from three model organisms. Predictions based on our DLM tended to exhibit less variance and bias than those from DeepTFactor, thus increasing reliability and decreasing overfitting.

Towards a Reliable Uncertainty Quantification in Residual Stress Measurements with Relaxation Methods: Finding Average Residual Stresses is a Well-Posed Problem

BackgroundIn a previous work, the problem of identifying residual stresses through relaxation methods was demonstrated to be mathematically ill-posed. In practice, it means that the solution process is affected by a bias-variance tradeoff, where some theoretically uncomputable bias has to be introduced in order to obtain a solution with a manageable signal-to-noise ratio.ObjectiveAs a consequence, an important question arises: how can the solution uncertainty be quantified if a part of it is inaccessible? Additional physical knowledge could—in theory—provide a characterization of bias, but this process is practically impossible with presently available techniques.MethodsA brief review of biases in established methods is provided, showing that ruling them out would require a piece of knowledge that is never available in practice. Then, the concept of average stresses over a distance is introduced, and it is shown that finding them generates a well-posed problem. A numerical example illustrates the theoretical discussionResultsSince finding average stresses is a well-posed problem, the bias-variance tradeoff disappears. The uncertainties of the results can be estimated with the usual methods, and exact confidence intervals can be obtained.ConclusionsOn a broader scope, we argue that residual stresses and relaxation methods expose the limits of the concept of point-wise stress values, which instead works almost flawlessly when a natural unstressed state can be assumed, as in classical continuum mechanics (for instance, in the theory of elasticity). As a consequence, we are forced to focus on the effects of stress rather than on its point-wise evaluation.

Prediction of Thrips Damage Distribution in Mango Orchards Using a Novel Maximum Likelihood Classifier

Thrips constitute the primary pest responsible for reducing mango yield and quality every year in Asia. Therefore, the efficient monitoring of thrips damage across mango orchards on a large scale to aid farmers in devising rational pesticide application strategies poses a significant challenge within the current mango industry. This study designs a mango thrips damage inversion prediction method based on the maximum likelihood classifier (MLC). Initially, drone multispectral remote sensing technology is utilized to acquire multispectral data from mango orchards, which are then combined with ground hyperspectral information to identify sensitive bands indicative of mango leaf damage caused by thrips. Subsequently, correlation analysis is conducted on various vegetation indices, leading to the selection of the Greenness Normalized Difference Vegetation Index (GNDVI), which exhibits a strong correlation coefficient of 0.82, as the spectral characteristic parameter for the inversion prediction model. The construction of a remote sensing prediction model for thrips damage distribution in mango orchards is then undertaken based on the MLC. Acknowledging the bias-variance trade-off inherent in the MLC when processing spectral data and its potential limitations in feature extraction and robustness, this study proposes a modification wherein neighboring pixels are weighted differently to enhance the model’s feature extraction capabilities. Experimental results show that the novel MLC maintains stable estimation levels across various numbers of domain pixels, achieving an inversion accuracy of 91.23%. Through the reconstruction of the pixel matrix, the damage distribution of thrips in mango orchards can be swiftly and comprehensively visualized over extensive areas.

Exploring Channel-Aware Typical Features for Out-of-Distribution Detection

Detecting out-of-distribution (OOD) data is essential to ensure the reliability of machine learning models when deployed in real-world scenarios. Different from most previous test-time OOD detection methods that focus on designing OOD scores, we delve into the challenges in OOD detection from the perspective of typicality and regard the feature’s high-probability region as the feature’s typical set. However, the existing typical-feature-based OOD detection method implies an assumption: the proportion of typical feature sets for each channel is fixed. According to our experimental analysis, each channel contributes differently to OOD detection. Adopting a fixed proportion for all channels results in several channels losing too many typical features or incorporating too many abnormal features, resulting in low performance. Therefore, exploring the channel-aware typical features is crucial to better-separating ID and OOD data. Driven by this insight, we propose expLoring channel-Aware tyPical featureS (LAPS). Firstly, LAPS obtains the channel-aware typical set by calibrating the channel-level typical set with the global typical set from the mean and standard deviation. Then, LAPS rectifies the features into channel-aware typical sets to obtain channel-aware typical features. Finally, LAPS leverages the channel-aware typical features to calculate the energy score for OOD detection. Theoretical and visual analyses verify that LAPS achieves a better bias-variance trade-off. Experiments verify the effectiveness and generalization of LAPS under different architectures and OOD scores.

Reducing Spatial Fitting Error in Distillation of Denoising Diffusion Models

Denoising Diffusion models have exhibited remarkable capabilities in image generation. However, generating high-quality samples requires a large number of iterations. Knowledge distillation for diffusion models is an effective method to address this limitation with a shortened sampling process but causes degraded generative quality. Based on our analysis with bias-variance decomposition and experimental observations, we attribute the degradation to the spatial fitting error occurring in the training of both the teacher and student model in the distillation. Accordingly, we propose Spatial Fitting-Error Reduction Distillation model (SFERD). SFERD utilizes attention guidance from the teacher model and a designed semantic gradient predictor to reduce the student's fitting error. Empirically, our proposed model facilitates high-quality sample generation in a few function evaluations. We achieve an FID of 5.31 on CIFAR-10 and 9.39 on ImageNet 64x64 with only one step, outperforming existing diffusion methods. Our study provides a new perspective on diffusion distillation by highlighting the intrinsic denoising ability of models.

Z-SignFedAvg: A Unified Stochastic Sign-Based Compression for Federated Learning

Federated Learning (FL) is a promising privacy-preserving distributed learning paradigm but suffers from high communi- cation cost when training large-scale machine learning models. Sign-based methods, such as SignSGD, have been proposed as a biased gradient compression technique for reducing the communication cost. However, sign-based algorithms could diverge under heterogeneous data, which thus motivated the de- velopment of advanced techniques, such as the error-feedback method and stochastic sign-based compression, to fix this issue. Nevertheless, these methods still suffer from slower convergence rates, and none of them allows multiple local SGD updates like FedAvg. In this paper, we propose a novel noisy perturbation scheme with a general symmetric noise distribution for sign-based compression, which not only al- lows one to flexibly control the bias-variance tradeoff for the compressed gradient, but also provides a unified viewpoint to existing stochastic sign-based methods. More importantly, the proposed scheme enables the development of the very first sign-based FedAvg algorithm (z-SignFedAvg) to accelerate the convergence. Theoretically, we show that z-SignFedAvg achieves a faster convergence rate than existing sign-based methods and, under the uniformly distributed noise, can enjoy the same convergence rate as its uncompressed counterpart. Extensive experiments are conducted to demonstrate that the z-SignFedAvg can achieve competitive empirical performance on real datasets and outperforms existing schemes.

Optimizing NILC Extractions of the Thermal Sunyaev–Zel’Dovich Effect with Deep Learning

All-sky maps of the thermal Sunyaev–Zel’dovich effect (SZ) tend to suffer from systematic features arising from the component-separation techniques used to extract the signal. In this work, we investigate one of these methods, known as needlet internal linear combination (NILC), and test its performance on simulated data. We show that NILC estimates are strongly affected by the choice of the spatial localization parameter (Γ), which controls a bias-variance trade-off. Typically, NILC extractions assume a fixed value of Γ over the entire sky, but we show there exists an optimal Γ that depends on the SZ signal strength and local contamination properties. Then we calculate the NILC solutions for multiple values of Γ and feed the results into a neural network to predict the SZ signal. This extraction method, which we call Deep-NILC, is tested against a set of validation data, including recovered radial profiles of resolved systems. Our main result is that Deep-NILC offers significant improvements over choosing fixed values of Γ.

Revising Alpine summer temperatures since 881 CE

Europe experienced severe heat waves during the last decade, which impacted ecological and societal systems and are likely to increase under projected global warming. A better understanding of pre-industrial warm-season changes is needed to contextualize these recent trends and extremes. Here, we introduce a network of 352 living and relict larch trees (Larix decidua Mill.) from the Matter and Simplon valleys in the Swiss Alps to develop a maximum latewood density (MXD) chronology calibrating at r = 0.8 (p > 0.05, 1901–2017 CE) against May–August temperatures over Western Europe. Machine learning is applied to identify historical wood samples aligning with growth characteristics of sites from elevations above 1900 m asl to extend the modern part of the chronology back to 881 CE. The new Alpine record reveals warmer conditions in the tenth century, followed by an extended cold period during the late Medieval times, a less-pronounced Little Ice Age culminating in the 1810s, and prolonged anthropogenic warming until present. The Samalas eruption likely triggered the coldest reconstructed summer in Western Europe in 1258 CE (-2.32 °C), which is in line with a recently published MXD-based reconstruction from the Spanish Pyrenees. Whereas the new Alpine reconstruction is potentially constrained in the lowest frequency, centennial timescale domain, it overcomes variance biases in existing state-of-the-art reconstructions and sets a new standard in site-control of historical samples and calibration/ verification statistics.

Northern Hemisphere winter atmospheric teleconnections are intensified by extratropical ocean-atmosphere coupling

The role of extratropical atmosphere-ocean coupling in generating and maintaining large-scale atmospheric low-frequency variability remains an open question owing to vigorous atmospheric internal fluctuations. Here, we use coupled and uncoupled large-ensemble global model simulations to clarify how the coupling intensifies atmospheric teleconnection patterns in the Northern Hemisphere winter. We show that the extratropical coupling selectively enhances the variance of three principal modes of variability, explaining 13%, 11%, and 10% of the total variance of Pacific/North American, North Atlantic Oscillation, and Warm-Arctic Cold-Eurasian patterns, respectively. Atmosphere-ocean coupling reduces damping to lower-tropospheric available potential energy, which in turn increases kinetic energy by changing energy transfer within the mode. The extratropical ocean is overall passive (adjustable) to large-scale atmospheric variation, thus contributing to the prominence of these modes. The geographical dependence of available potential energy damping suggests the existence of mode-specific sweet spots where the influence of coupling operates efficiently, providing a clue to improving the model biases in variance and signal-to-noise ratio of these modes.

Approximately Optimal Domain Adaptation with Fisher’s Linear Discriminant

We propose and study a data-driven method that can interpolate between a classical and a modern approach to classification for a class of linear models. The class is the convex combinations of an average of the source task classifiers and a classifier trained on the limited data available for the target task. We derive the expected loss of an element in the class with respect to the target distribution for a specific generative model, propose a computable approximation of the loss, and demonstrate that the element of the proposed class that minimizes the approximated risk is able to exploit a natural bias–variance trade-off in task space in both simulated and real-data settings. We conclude by discussing further applications, limitations, and potential future research directions.