Dataset Shift Research Articles

Radio Galaxy Zoo: using semi-supervised learning to leverage large unlabelled data sets for radio galaxy classification under data set shift

ABSTRACT In this work, we examine the classification accuracy and robustness of a state-of-the-art semi-supervised learning (SSL) algorithm applied to the morphological classification of radio galaxies. We test if SSL with fewer labels can achieve test accuracies comparable to the supervised state of the art and whether this holds when incorporating previously unseen data. We find that for the radio galaxy classification problem considered, SSL provides additional regularization and outperforms the baseline test accuracy. However, in contrast to model performance metrics reported on computer science benchmarking data sets, we find that improvement is limited to a narrow range of label volumes, with performance falling off rapidly at low label volumes. Additionally, we show that SSL does not improve model calibration, regardless of whether classification is improved. Moreover, we find that when different underlying catalogues drawn from the same radio survey are used to provide the labelled and unlabelled data sets required for SSL, a significant drop in classification performance is observed, highlighting the difficulty of applying SSL techniques under data set shift. We show that a class-imbalanced unlabelled data pool negatively affects performance through prior probability shift, which we suggest may explain this performance drop, and that using the Fréchet distance between labelled and unlabelled data sets as a measure of data set shift can provide a prediction of model performance, but that for typical radio galaxy data sets with labelled sample volumes of $\mathcal {O}(10^3)$, the sample variance associated with this technique is high and the technique is in general not sufficiently robust to replace a train–test cycle.

A Unified Framework on Generalizability of Clinical Prediction Models.

To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in concepts related to generalizability of CPMs in the clinical research and machine learning domains. Specifically, conventional statistical reasons to explain poor generalizability such as inadequate model development for the purposes of generalizability, differences in coding of predictors and outcome between development and external datasets, measurement error, inability to measure some predictors, and missing data, all have differing and often complementary treatments, in the two domains. Much of the current machine learning literature on generalizability of CPMs is in terms of dataset shift of which several types have been described. However, little research exists to synthesize concepts in the two domains. Bridging this conceptual discontinuity in the context of CPMs can facilitate systematic development of CPMs and evaluation of their sensitivity to factors that affect generalizability. We survey generalizability and dataset shift in CPMs from both the clinical research and machine learning perspectives, and describe a unifying framework to analyze generalizability of CPMs and to explain their sensitivity to factors affecting it. Our framework leads to a set of signaling statements that can be used to characterize differences between datasets in terms of factors that affect generalizability of the CPMs.

Multi-Level Stacked Regression for predicting electricity consumption of Hot Rolling Mill

Predicting electrical power consumption is essential in many industries. This paper presents a method to predict total electrical power consumption in a certain period of Hot Rolling Mill (HRM) in steel mills. We adequately pre-processed operation data, including work schedule, work result, and power measurement. In addition, based on domain knowledge, we derived several useful variables relevant to power consumption. To predict power consumption, we developed a Multi-Level Stacked Regression model that combines the strengths of multiple prediction models. Real-world applications have dataset shift characteristics in which the covariance between features or distribution of the features changes over time due to frequent changes in the working environment. To verify the generalization accuracy of our model, we trained and tested our model for various training/test splits and compared it with several comparative models. Our model showed the best accuracy in all training/test splits. We verified whether the predictive model reflects the real-world environment well by analyzing the actual physical meaning of the essential features used in the predictive model.

Automatic Fish Age Determination across Different Otolith Image Labs Using Domain Adaptation

The age determination of fish is fundamental to marine resource management. This task is commonly done by analysis of otoliths performed manually by human experts. Otolith images from Greenland halibut acquired by the Institute of Marine Research (Norway) were recently used to train a convolutional neural network (CNN) for automatically predicting fish age, opening the way for requiring less human effort and availability of expertise by means of deep learning (DL). In this study, we demonstrate that applying a CNN model trained on images from one lab (in Norway) does not lead to a suitable performance when predicting fish ages from otolith images from another lab (in Iceland) for the same species. This is due to a problem known as dataset shift, where the source data, i.e., the dataset the model was trained on have different characteristics from the dataset at test stage, here denoted as target data. We further demonstrate that we can handle this problem by using domain adaptation, such that an existing model trained in the source domain is adapted to perform well in the target domain, without requiring extra annotation effort. We investigate four different approaches: (i) simple adaptation via image standardization, (ii) adversarial generative adaptation, (iii) adversarial discriminative adaptation and (iv) self-supervised adaptation. The results show that the performance varies substantially between the methods, with adversarial discriminative and self-supervised adaptations being the best approaches. Without using a domain adaptation approach, the root mean squared error (RMSE) and coefficient of variation (CV) on the Icelandic dataset are as high as 5.12 years and 28.6%, respectively, whereas by using the self-supervised domain adaptation, the RMSE and CV are reduced to 1.94 years and 11.1%. We conclude that careful consideration must be given before DL-based predictors are applied to perform large scale inference. Despite that, domain adaptation is a promising solution for handling problems of dataset shift across image labs.

Comparison of different ML methods concerning prediction quality, domain adaptation and robustness

Abstract Nowadays machine learning methods and data-driven models have been used widely in different fields including computer vision, biomedicine, and condition monitoring. However, these models show performance degradation when meeting real-life situations. Domain or dataset shift or out-of-distribution (OOD) prediction is mentioned as the reason for this problem. Especially in industrial condition monitoring, it is not clear when we should be concerned about domain shift and which methods are more robust against this problem. In this paper prediction results are compared for a conventional machine learning workflow based on feature extraction, selection, and classification/regression (FESC/R) and deep neural networks on two publicly available industrial datasets. We show that it is possible to visualize the possible shift in domain using feature extraction and principal component analysis. Also, experimental competition shows that the cross-domain validated results of FESC/R are comparable to the reported state-of-the-art methods. Finally, we show that the results for simple randomly selected validation sets do not correctly represent the model performance in real-world applications.

Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine

Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization (DG) and unsupervised domain adaptation (UDA) might be suitable to proactively mitigate dataset shift. The objective was to characterize the impact of temporal dataset shift on clinical prediction models and benchmark DG and UDA algorithms on improving model robustness. In this cohort study, intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008–2010, 2011–2013, 2014–2016 and 2017–2019). Tasks were predicting mortality, long length of stay, sepsis and invasive ventilation. Feedforward neural networks were used as prediction models. The baseline experiment trained models using empirical risk minimization (ERM) on 2008–2010 (ERM[08–10]) and evaluated them on subsequent year groups. DG experiment trained models using algorithms that estimated invariant properties using 2008–2016 and evaluated them on 2017–2019. UDA experiment leveraged unlabelled samples from 2017 to 2019 for unsupervised distribution matching. DG and UDA models were compared to ERM[08–16] models trained using 2008–2016. Main performance measures were area-under-the-receiver-operating-characteristic curve (AUROC), area-under-the-precision-recall curve and absolute calibration error. Threshold-based metrics including false-positives and false-negatives were used to assess the clinical impact of temporal dataset shift and its mitigation strategies. In the baseline experiments, dataset shift was most evident for sepsis prediction (maximum AUROC drop, 0.090; 95% confidence interval (CI), 0.080–0.101). Considering a scenario of 100 consecutively admitted patients showed that ERM[08–10] applied to 2017–2019 was associated with one additional false-negative among 11 patients with sepsis, when compared to the model applied to 2008–2010. When compared with ERM[08–16], DG and UDA experiments failed to produce more robust models (range of AUROC difference, − 0.003 to 0.050). In conclusion, DG and UDA failed to produce more robust models compared to ERM in the setting of temporal dataset shift. Alternate approaches are required to preserve model performance over time in clinical medicine.

Assessing the performance of control charts for detecting previously unexplored shift types in high density spatial data

As the collection and use of high-density (HD) spatial datasets has increased, the Statistical Process Control research community has strived to develop effective and efficient control charting techniques for these datasets. In general, these research efforts propose new control charting techniques and evaluate their abilities to detect different shift types. However, these works typically considered only conventional shift types, such as mean and variance shifts, which only account for a portion of the shift types that can manifest themselves in HD spatial datasets. In essence, advanced mathematical approaches are being developed for use with state-of-the-art measurement systems but assess their performance with traditional shift types developed for univariate statistics. This may hinder the effectiveness of these approaches in practice, as real-world systems may experience shift types other than (or in addition to) those addressed in the literature. The goal of this paper is to understand the ability of previously proposed control charting techniques to detect these unexplored shift types. This goal is accomplished through a simulation study that considers five different control charting techniques, identified from both the spatial statistics and spatial scan statistics literatures. The performances of these control charts are assessed against previously unexplored HD spatial dataset shift types. The results indicate that many control charting approaches were highly sensitive to variety of shift types. This suggests significant promise in the use of these approaches in systems that are susceptible to a wide variety of shift types, including shift types they were not specifically designed to detect.

Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures.

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model’s uncertainty is evaluated using point-prediction metrics, such as the negative log-likelihood (NLL), expected calibration error (ECE) or the Brier score on held-out data. Marginal coverage of prediction intervals or sets, a well-known concept in the statistical literature, is an intuitive alternative to these metrics but has yet to be systematically studied for many popular uncertainty quantification techniques for deep learning models. With marginal coverage and the complementary notion of the width of a prediction interval, downstream users of deployed machine learning models can better understand uncertainty quantification both on a global dataset level and on a per-sample basis. In this study, we provide the first large-scale evaluation of the empirical frequentist coverage properties of well-known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and reinforce coverage as an important metric in developing models for real-world applications.

Towards robust partially supervised multi-structure medical image segmentation on small-scale data

The data-driven nature of deep learning (DL) models for semantic segmentation requires a large number of pixel-level annotations. However, large-scale and fully labeled medical datasets are often unavailable for practical tasks. Recently, partially supervised methods have been proposed to utilize images with incomplete labels in the medical domain. To bridge the methodological gaps in partially supervised learning (PSL) under data scarcity, we propose Vicinal Labels Under Uncertainty (VLUU), a simple yet efficient framework utilizing the human structure similarity for partially supervised medical image segmentation. Motivated by multi-task learning and vicinal risk minimization, VLUU transforms the partially supervised problem into a fully supervised problem by generating vicinal labels. We systematically evaluate VLUU under the challenges of small-scale data, dataset shift, and class imbalance on two commonly used segmentation datasets for the tasks of chest organ segmentation and optic disc-and-cup segmentation. The experimental results show that VLUU can consistently outperform previous partially supervised models in these settings. Our research suggests a new research direction in label-efficient deep learning with partial supervision.

Action Recognition Using Close-Up of Maximum Activation and ETRI-Activity3D LivingLab Dataset.

The development of action recognition models has shown great performance on various video datasets. Nevertheless, because there is no rich data on target actions in existing datasets, it is insufficient to perform action recognition applications required by industries. To satisfy this requirement, datasets composed of target actions with high availability have been created, but it is difficult to capture various characteristics in actual environments because video data are generated in a specific environment. In this paper, we introduce a new ETRI-Activity3D-LivingLab dataset, which provides action sequences in actual environments and helps to handle a network generalization issue due to the dataset shift. When the action recognition model is trained on the ETRI-Activity3D and KIST SynADL datasets and evaluated on the ETRI-Activity3D-LivingLab dataset, the performance can be severely degraded because the datasets were captured in different environments domains. To reduce this dataset shift between training and testing datasets, we propose a close-up of maximum activation, which magnifies the most activated part of a video input in detail. In addition, we present various experimental results and analysis that show the dataset shift and demonstrate the effectiveness of the proposed method.

Temporal shift and accuracy of machine learning in heart transplant outcomes

Abstract Background Accurate prediction of outcomes following a heart transplant is critical to explaining risks and benefits to patients and decision-making when considering potential organ offers. Given the large number of potential variables to be considered, this task may be most efficiently performed using machine learning (ML). Purpose We trained and tested different ML algorithms to accurately predict outcomes following a cardiac transplant using the United Network of Organ Sharing (UNOS) database. Methods We included 67 939 adult and pediatric patients enrolled in the UNOS database between January 1994 and December 2016 who underwent cardiac transplantation (median age 53 [IQR 38 – 60], 72.7% males). In our models, as an input, we included 114 features that have been collected from recipients and donors prior to transplant. The primary outcome was all-cause mortality at one-year post-transplant. We evaluated three different ML methods: XGBoost, Random Forest (RF) and L2 regularized logistic regression. Algorithms were trained and tested using shuffled 10-fold cross-validation (CV) as well as rolling CV. In the rolling CV, to mimic prospective procedure, ML models were trained by incrementally adding patients according to transplant year and testing models on the data in the following year. The hyperparameters, controlling the learning process, were tuned using Bayesian optimization. Prognostic accuracy for one-year all-cause mortality was characterized using the area under the receiver-operating characteristic curve (AUC). Results In total, 8,394 patients died within 1 year of transplant. We observed a substantial difference in prognostic accuracy between the shuffled 10-fold CV and the rolling CV. In the 10-fold CV, XGBoost and RF achieved high predictive performance with AUC of 0.848 (95% CI: 0.842–0.854) and 0.891 (95% CI: 0.886–0.896), respectively. In the rolling CV, which is a more realistic setting, AUC dropped to 0.673 (95% CI: 0.661–0.684) for XGBoost and 0.670 (0.657–0.683) for RF. Predictive performance of L2 regularized logistic regression remained stable across the two CV procedures, achieving AUC 0.669 (95% CI: 0.662–0.676) in the 10-fold shuffled CV and 0.665 (95% CI: 0.649–0.680) in the rolling CV procedure (Figure). Conclusions Our study suggests that ML models could be used to predict mortality in the first year post-transplant. We also show that the choice of CV procedure is crucial for evaluating ML models, particularly in data collected over a long period of time. The difference between the shuffled and rolling CV in the predictive performance of the tree-based ML models might indicate temporal dataset shift. In the rolling CV, all three methods achieved similar predictive performance. Funding Acknowledgement Type of funding sources: Public grant(s) – National budget only. Main funding source(s): Research Foundation Flanders (FWO)

Preventing dataset shift from breaking machine-learning biomarkers.

Machine learning brings the hope of finding new biomarkers extracted from cohorts with rich biomedical measurements. A good biomarker is one that gives reliable detection of the corresponding condition. However, biomarkers are often extracted from a cohort that differs from the target population. Such a mismatch, known as a dataset shift, can undermine the application of the biomarker to new individuals. Dataset shifts are frequent in biomedical research, e.g., because of recruitment biases. When a dataset shift occurs, standard machine-learning techniques do not suffice to extract and validate biomarkers. This article provides an overview of when and how dataset shifts break machine-learning–extracted biomarkers, as well as detection and correction strategies.

Inertial load classification of low-cost electro-mechanical systems under dataset shift with fast end of line testing

This paper presents a rationale for designing a machine learning algorithm under dataset shift. In particular, we focus on the classification of the inertial load of low-cost Electro-Mechanical Actuators (EMAs) into several weight categories. In these low-cost settings, due to uncertainties in the manufacturing process, raw materials and usage, even if the EMA part number is the same, its serial numbers (i.e. items or exemplars) may show different physical behaviors. Thus, a learning model trained on data from a set of items can perform poorly when applied to other ones. The proposed solution comprises tailored normalization and cross validation procedures for training the classifier, along with suitable End Of Line (EOL) experiments for the characterization of a new produced EMA item. The approach is experimentally validated on the classification of the mass of sliding gates, using only measurements available on the gate EMA.

Out-of-time cross-validation strategies for classification in the presence of dataset shift

Out-of-time cross-validation strategies for classification in the presence of dataset shift

Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine.

The change in performance of machine learning models over time as a result of temporal dataset shift is a barrier to machine learning-derived models facilitating decision-making in clinical practice. Our aim was to describe technical procedures used to preserve the performance of machine learning models in the presence of temporal dataset shifts. Studies were included if they were fully published articles that used machine learning and implemented a procedure to mitigate the effects of temporal dataset shift in a clinical setting. We described how dataset shift was measured, the procedures used to preserve model performance, and their effects. Of 4,457 potentially relevant publications identified, 15 were included. The impact of temporal dataset shift was primarily quantified using changes, usually deterioration, in calibration or discrimination. Calibration deterioration was more common (n = 11) than discrimination deterioration (n = 3). Mitigation strategies were categorized as model level or feature level. Model-level approaches (n = 15) were more common than feature-level approaches (n = 2), with the most common approaches being model refitting (n = 12), probability calibration (n = 7), model updating (n = 6), and model selection (n = 6). In general, all mitigation strategies were successful at preserving calibration but not uniformly successful in preserving discrimination. There was limited research in preserving the performance of machine learning models in the presence of temporal dataset shift in clinical medicine. Future research could focus on the impact of dataset shift on clinical decision making, benchmark the mitigation strategies on a wider range of datasets and tasks, and identify optimal strategies for specific settings.

Generalized Bayes Quantification Learning under Dataset Shift

Quantification learning is the task of prevalence estimation for a test population using predictions from a classifier trained on a different population. Quantification methods assume that the sensitivities and specificities of the classifier are either perfect or transportable from the training to the test population. These assumptions are inappropriate in the presence of dataset shift, when the misclassification rates in the training population are not representative of those for the test population. Quantification under dataset shift has been addressed only for single-class (categorical) predictions and assuming perfect knowledge of the true labels on a small subset of the test population. We propose generalized Bayes quantification learning (GBQL) that uses the entire compositional predictions from probabilistic classifiers and allows for uncertainty in true class labels for the limited labeled test data. Instead of positing a full model, we use a model-free Bayesian estimating equation approach to compositional data using Kullback–Leibler loss-functions based only on a first-moment assumption. The idea will be useful in Bayesian compositional data analysis in general as it is robust to different generating mechanisms for compositional data and allows 0’s and 1’s in the compositional outputs thereby including categorical outputs as a special case. We show how our method yields existing quantification approaches as special cases. Extension to an ensemble GBQL that uses predictions from multiple classifiers yielding inference robust to inclusion of a poor classifier is discussed. We outline a fast and efficient Gibbs sampler using a rounding and coarsening approximation to the loss functions. We establish posterior consistency, asymptotic normality and valid coverage of interval estimates from GBQL, which to our knowledge are the first theoretical results for a quantification approach in the presence of local labeled data. We also establish finite sample posterior concentration rate. Empirical performance of GBQL is demonstrated through simulations and analysis of real data with evident dataset shift. Supplementary materials for this article are available online.

Preventing Failures by Dataset Shift Detection in Safety-Critical Graph Applications.

Dataset shift refers to the problem where the input data distribution may change over time (e.g., between training and test stages). Since this can be a critical bottleneck in several safety-critical applications such as healthcare, drug-discovery, etc., dataset shift detection has become an important research issue in machine learning. Though several existing efforts have focused on image/video data, applications with graph-structured data have not received sufficient attention. Therefore, in this paper, we investigate the problem of detecting shifts in graph structured data through the lens of statistical hypothesis testing. Specifically, we propose a practical two-sample test based approach for shift detection in large-scale graph structured data. Our approach is very flexible in that it is suitable for both undirected and directed graphs, and eliminates the need for equal sample sizes. Using empirical studies, we demonstrate the effectiveness of the proposed test in detecting dataset shifts. We also corroborate these findings using real-world datasets, characterized by directed graphs and a large number of nodes.

DIBS: Diversity Inducing Information Bottleneck in Model Ensembles

Although deep learning models have achieved state-of-the art performance on a number of vision tasks, generalization over high dimensional multi-modal data, and reliable predictive uncertainty estimation are still active areas of research. Bayesian approaches including Bayesian Neural Nets (BNNs) do not scale well to modern computer vision tasks, as they are difficult to train, and have poor generalization under dataset-shift. This motivates the need for effective ensembles which can generalize and give reliable uncertainty estimates. In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning the stochastic latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. We evaluate our method on benchmark datasets: MNIST, CIFAR100, TinyImageNet and MIT Places 2, and compared to the most competitive baselines show significant improvements in classification accuracy, under a shift in the data distribution and in out-of-distribution detection. over 10% relative improvement in classification accuracy, over 5% relative improvement in generalizing under dataset shift, and over 5% better predictive uncertainty estimation as inferred by efficient out-of-distribution (OOD) detection.

Finding Quasars behind the Galactic Plane. I. Candidate Selections with Transfer Learning

Abstract Quasars behind the Galactic plane (GPQs) are important astrometric references and useful probes of Milky Way gas. However, the search for GPQs is difficult due to large extinctions and high source densities in the Galactic plane. Existing selection methods for quasars developed using high Galactic latitude (high-b) data cannot be applied to the Galactic plane directly because the photometric data obtained from high-b regions and the Galactic plane follow different probability distributions. To alleviate this data set shift problem for quasar candidate selection, we adopt a transfer-learning framework at both the data and algorithm levels. At the data level, to make a training set in which a data set shift is modeled, we synthesize quasars and galaxies behind the Galactic plane based on SDSS sources and the Galactic dust map. At the algorithm level, to reduce the effect of class imbalance, we transform the three-class classification problem for stars, galaxies, and quasars into two binary classification tasks. We apply the XGBoost algorithm to Pan-STARRS1 (PS1) and AllWISE photometry for classification and an additional cut on Gaia proper motion to remove stellar contaminants. We obtain a reliable GPQ candidate catalog with 160,946 sources located at ∣b∣ ≤ 20° in the PS1-AllWISE footprint. Photometric redshifts of GPQ candidates achieved with the XGBoost regression algorithm show that our selection method can identify quasars in a wide redshift range (0 < z ≲ 5). This study extends the systematic searches for quasars to the dense stellar fields and shows the feasibility of using astronomical knowledge to improve data mining under complex conditions in the big-data era.

Viral Pneumonia Screening on Chest X-Rays Using Confidence-Aware Anomaly Detection.

Clusters of viral pneumonia occurrences over a short period may be a harbinger of an outbreak or pandemic. Rapid and accurate detection of viral pneumonia using chest X-rays can be of significant value for large-scale screening and epidemic prevention, particularly when other more sophisticated imaging modalities are not readily accessible. However, the emergence of novel mutated viruses causes a substantial dataset shift, which can greatly limit the performance of classification-based approaches. In this paper, we formulate the task of differentiating viral pneumonia from non-viral pneumonia and healthy controls into a one-class classification-based anomaly detection problem. We therefore propose the confidence-aware anomaly detection (CAAD) model, which consists of a shared feature extractor, an anomaly detection module, and a confidence prediction module. If the anomaly score produced by the anomaly detection module is large enough, or the confidence score estimated by the confidence prediction module is small enough, the input will be accepted as an anomaly case (i.e., viral pneumonia). The major advantage of our approach over binary classification is that we avoid modeling individual viral pneumonia classes explicitly and treat all known viral pneumonia cases as anomalies to improve the one-class model. The proposed model outperforms binary classification models on the clinical X-VIRAL dataset that contains 5,977 viral pneumonia (no COVID-19) cases, 37,393 non-viral pneumonia or healthy cases. Moreover, when directly testing on the X-COVID dataset that contains 106 COVID-19 cases and 107 normal controls without any fine-tuning, our model achieves an AUC of 83.61% and sensitivity of 71.70%, which is comparable to the performance of radiologists reported in the literature.