Dataset Shift Research Articles

When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights.

As a crucial step toward real-world learning scenarios with changing environments, dataset shift theory and invariant representation learning algorithm have been extensively studied to relax the identical distribution assumption in classical learning setting. Among the different assumptions on the essential of shifting distributions, generalized label shift (GLS) is the latest developed one which shows great potential to deal with the complex factors within the shift. In this paper, we aim to explore the limitations of current dataset shift theory and algorithm, and further provide new insights by presenting a comprehensive understanding of GLS. From theoretical aspect, two informative generalization bounds are derived, and the GLS learner are proved to be sufficiently close to optimal target model from the Bayesian perspective. The main results show the insufficiency of invariant representation learning, and prove the sufficiency and necessity of GLS correction for generalization, which provide theoretical supports and innovations for exploring generalizable model under dataset shift. From methodological aspect, we provide a unified view of existing shift correction frameworks, and propose a kernel embedding-based correction algorithm (KECA) to minimize the generalization error and achieve successful knowledge transfer. Both theoretical results and extensive experiment evaluations demonstrate the sufficiency and necessity of GLS correction for addressing dataset shift and the superiority of proposed algorithm.

Enhancing adaptivity of active sonar classifier considering loss landscape under dataset shift

We study the loss landscape of deep neural networks and use it to solve dataset shift problems in active sonar classification. The dataset shift degrades the generalization performance of supervised learning-based classifiers. When test data samples are available, fine-tuning methods can be used to mitigate the performance decrease. However, they can induce catastrophic forgetting and negative transfer because fine-tuned weights could be overfitted to the test data distribution. These problems are more severe in active sonar datasets with small amounts and less diversity. To mitigate the two problems in conventional fine-tuning, we explore the loss landscape of supervised learning-based active sonar classifiers and apply it to derive adaptative weights from pre-trained weights. Our results showed promising performance.

Exploring the Trade-Off between generalist and specialized Models: A center-based comparative analysis for glioblastoma segmentation

IntroductionInherent variations between inter-center data can undermine the robustness of segmentation models when applied at a specific center (dataset shift). We investigated whether specialized center-specific models are more effective compared to generalist models based on multi-center data, and how center-specific data could enhance the performance of generalist models within a particular center using a fine-tuning transfer learning approach. For this purpose, we studied the dataset shift at center level and conducted a comparative analysis to assess the impact of data source on glioblastoma segmentation models. Methods & MaterialsThe three key components of dataset shift were studied: prior probability shift—variations in tumor size or tissue distribution among centers; covariate shift—inter-center MRI alterations; and concept shift—different criteria for tumor segmentation. BraTS 2021 dataset was used, which includes 1251 cases from 23 centers. Thereafter, 155 deep-learning models were developed and compared, including 1) generalist models trained with multi-center data, 2) specialized models using only center-specific data, and 3) fine-tuned generalist models using center-specific data. ResultsThe three key components of dataset shift were characterized. The amount of covariate shift was substantial, indicating large variations in MR imaging between different centers. Glioblastoma segmentation models tend to perform best when using data from the application center. Generalist models, trained with over 700 samples, achieved a median Dice score of 88.98%. Specialized models surpassed this with 200 cases, while fine-tuned models outperformed with 50 cases. ConclusionsThe influence of dataset shift on model performance is evident. Fine-tuned and specialized models, utilizing data from the evaluated center, outperform generalist models, which rely on data from other centers. These approaches could encourage medical centers to develop customized models for their local use, enhancing the accuracy and reliability of glioblastoma segmentation in a context where dataset shift is inevitable.

Toward cross-subject and cross-session generalization in EEG-based emotion recognition: Systematic review, taxonomy, and methods

A systematic review on machine-learning strategies for improving generalization in electroencephalography-based emotion classification was realized. In particular, cross-subject and cross-session generalization was focused. In this context, the non-stationarity of electroencephalographic (EEG) signals is a critical issue and can lead to the Dataset Shift problem. Several architectures and methods have been proposed to address this issue, mainly based on transfer learning methods. In this review, 449 papers were retrieved from the Scopus, IEEE Xplore and PubMed databases through a search query focusing on modern machine learning techniques for generalization in EEG-based emotion assessment. Among these papers, 79 were found eligible based on their relevance to the problem. Studies lacking a specific cross-subject or cross-session validation strategy, or making use of other biosignals as support were excluded. On the basis of the selected papers’ analysis, a taxonomy of the studies employing Machine Learning (ML) methods was proposed, together with a brief discussion of the different ML approaches involved. The studies reporting the best results in terms of average classification accuracy were identified, supporting that transfer learning methods seem to perform better than other approaches. A discussion is proposed on the impact of (i) the emotion theoretical models and (ii) psychological screening of the experimental sample on the classifier performances.

Mixed event separation and identification based on a convolutional neural network trained with the domain transfer method for a φ-OTDR sensing system.

In phase-sensitive optical time domain reflectometer (φ-OTDR) based distributed acoustic sensing (DAS), correct identification of event types is challenging in complex environments where multiple events happen simultaneously. In this study, we have proposed a convolutional neural network (CNN) with a separation module and an identification module to simultaneously separate a mixed event into individual single-event components and identify each type of component contained in the mixed event. The domain transfer method is used in the training, fine-tuning, and testing of the proposed CNN, which saves 94% of the workload for massive DAS data collection and signal demodulation. A fine-tuning stage is added to minimize the impact of the dataset shift between the audio data and DAS data, hence enhancing the separation and identification performance. The model has good noise tolerance and achieves nearly 90% identification accuracy even at a relatively low signal-to-noise ratio (SNR). Compared with the conventional method using DAS data for training, domain transfer using a large amount of diverse audio data for training well generalizes the model to the target domain and hence provides more stable performance with only little degradation of identification accuracy.

Dynamic Quantification With Constrained Error Under Unknown General Dataset Shift

Dynamic Quantification With Constrained Error Under Unknown General Dataset Shift

Differentiation of glioblastoma from solitary brain metastasis using deep ensembles: Empirical estimation of uncertainty for clinical reliability

Background and ObjectivesTo develop a clinically reliable deep learning model to differentiate glioblastoma (GBM) from solitary brain metastasis (SBM) by providing predictive uncertainty estimates and interpretability. MethodsA total of 469 patients (300 GBM, 169 SBM) were enrolled in the institutional training set. Deep ensembles based on DenseNet121 were trained on multiparametric MRI. The model performance was validated in the external test set consisting of 143 patients (101 GBM, 42 SBM). Entropy values for each input were evaluated for uncertainty measurement; based on entropy values, the datasets were split to high- and low-uncertainty groups. In addition, entropy values of out-of-distribution (OOD) data from unknown class (257 patients with meningioma) were compared to assess uncertainty estimates of the model. The model interpretability was further evaluated by localization accuracy of the model. ResultsOn external test set, the area under the curve (AUC), accuracy, sensitivity and specificity of the deep ensembles were 0.83 (95 % confidence interval [CI] 0.76–0.90), 76.2 %, 54.8 % and 85.2 %, respectively. The performance was higher in the low-uncertainty group than in the high-uncertainty group, with AUCs of 0.91 (95 % CI 0.83–0.98) and 0.58 (95 % CI 0.44–0.71), indicating that assessment of uncertainty with entropy values ascertained reliable prediction in the low-uncertainty group. Further, deep ensembles classified a high proportion (90.7 %) of predictions on OOD data to be uncertain, showing robustness in dataset shift. Interpretability evaluated by localization accuracy provided further reliability in the “low-uncertainty and high-localization accuracy” subgroup, with an AUC of 0.98 (95 % CI 0.95–1.00). ConclusionsEmpirical assessment of uncertainty and interpretability in deep ensembles provides evidence for the robustness of prediction, offering a clinically reliable model in differentiating GBM from SBM.

Sequential Data Integration Under Dataset Shift

With the rapidly increasing availability of large-scale and high-velocity streaming data, efficient algorithms that can process data in batches without requiring expensive storage and computation resources have drawn considerable attention. An emerging challenge in developing efficient batch processing techniques is dataset shift, where the joint distribution of the collected data varies across batches. If not recognized and addressed properly, dataset shift often leads to erroneous statistical inferences when integrating data from different batches. In this article, two shift-adjusted estimation procedures are developed for updated estimation of the parameter in the presence of dataset shift. Under prior probability shift, we can obtain parameter estimation and assess the degree of dataset shift simultaneously. We study the asymptotic properties of the proposed estimators and evaluate their performance in numerical studies. The proposed methodologies are illustrated with an analysis of the Ford GoBike docked bike-sharing data.

A feature selection method based on Shapley values robust for concept shift in regression

Feature selection is one of the most relevant processes in any methodology for creating a statistical learning model. Usually, existing algorithms establish some criterion to select the most influential variables, discarding those that do not contribute to the model with any relevant information. This methodology makes sense in a static situation where the joint distribution of the data does not vary over time. However, when dealing with real data, it is common to encounter the problem of the dataset shift and, specifically, changes in the relationships between variables (concept shift). In this case, the influence of a variable cannot be the only indicator of its quality as a regressor of the model, since the relationship learned in the training phase may not correspond to the current situation. In tackling this problem, our approach establishes a direct relationship between the Shapley values and prediction errors, operating at a more local level to effectively detect the individual biases introduced by each variable. The proposed methodology is evaluated through various examples, including synthetic scenarios mimicking sudden and incremental shift situations, as well as two real-world cases characterized by concept shifts. Additionally, we perform three analyses of standard situations to assess the algorithm’s robustness in the absence of shifts. The results demonstrate that our proposed algorithm significantly outperforms state-of-the-art feature selection methods in concept shift scenarios, while matching the performance of existing methodologies in static situations.

Efficient data integration under prior probability shift.

Conventional supervised learning usually operates under the premise that data are collected from the same underlying population. However, challenges may arise when integrating new data from different populations, resulting in a phenomenon known as dataset shift. This paper focuses on prior probability shift, where the distribution of the outcome varies across datasets but the conditional distribution of features given the outcome remains the same. To tackle the challenges posed by such shift, we propose an estimation algorithm that can efficiently combine information from multiple sources. Unlike existing methods that are restricted to discrete outcomes, the proposed approach accommodates both discrete and continuous outcomes. It also handles high-dimensional covariate vectors through variable selection using an adaptive least absolute shrinkage and selection operator penalty, producing efficient estimates that possess the oracle property. Moreover, a novel semiparametric likelihood ratio test is proposed to check the validity of prior probability shift assumptions by embedding the null conditional density function into Neyman's smooth alternatives (Neyman, 1937) and testing study-specific parameters. We demonstrate the effectiveness of our proposed method through extensive simulations and a real data example. The proposed methods serve as a useful addition to the repertoire of tools for dealing dataset shifts.

Binary quantification and dataset shift: an experimental investigation

AbstractQuantification is the supervised learning task that consists of training predictors of the class prevalence values of sets of unlabelled data, and is of special interest when the labelled data on which the predictor has been trained and the unlabelled data are not IID, i.e., suffer from dataset shift. To date, quantification methods have mostly been tested only on a special case of dataset shift, i.e., prior probability shift; the relationship between quantification and other types of dataset shift remains, by and large, unexplored. In this work we carry out an experimental analysis of how current quantification algorithms behave under different types of dataset shift, in order to identify limitations of current approaches and hopefully pave the way for the development of more broadly applicable methods. We do this by proposing a fine-grained taxonomy of types of dataset shift, by establishing protocols for the generation of datasets affected by these types of shift, and by testing existing quantification methods on the datasets thus generated. One finding that results from this investigation is that many existing quantification methods that had been found robust to prior probability shift are not necessarily robust to other types of dataset shift. A second finding is that no existing quantification method seems to be robust enough to dealing with all the types of dataset shift we simulate in our experiments. The code needed to reproduce all our experiments is publicly available at https://github.com/pglez82/quant_datasetshift.

Mitigating Bias in Bayesian Optimized Data While Designing MacPherson Suspension Architecture

With the rise of artificial intelligence, the automotive industry searched for novel ways to improve future product design. We focus on designing automatic MacPherson suspension architecture for the automotive sector. It takes time for an automotive engineer to design vehicle parts and thus slows the pace of innovation in this field. Given the car's particular kinematic characteristics, we propose to predict an architecture by positioning the hardpoints. This work deals with the biased data generated using the discipline models using the dataset shift learning paradigm. The optimized data are created with random and uniform sampling, with more samples with random sampling. We resolve the bias in the data, using a novel criterion for tuning the kernel mean matching and a weight estimation algorithm and designing the required target characteristics.

Floods Countermeasures by Hybrid Appraches with Hard and Natural Engineering and Flooding events at Mediterranean Coasts

Floodings can seriously affect coastal regions, endangering human life and causing geomorphological changes, with important consequences for urban infrastructures, harbour facilities, coastal tourism infrastructures, agriculture of coastal areas, fishery acivities, and transports. Factors causing flood events include climate change as in the case of increases in the probability and intensity of precipitations, sea level rise, coastal erosion, land subsidence, increased river flow, deforestation, wildfires, and an intensive urbanization with important changes in the uses of coasts. The phenomenon of flood hazard is expected to increase in the next years. In order to protect coastal areas, hard engineering infrastructures as concrete seawalls, breakwaters, groins, or dikes, have often been used, although they are quite expensive to build and maintain, and can deteriorate coastal vegetation. Green infrastructures, although mostly present in zones at low density, provide nature-based coastal protection and can be represented by restored or conserved coastal wetlands to produce natural buffer from storms, such as wetlands, barrier islands, reefs, mangrove and marsh ecosystems. In the last years it has emerged that each of these two approaches has problems when used alone, thus suggesting the use of hybrid infrastructures combining hard and green engineered structures. Hybrid infrastructures combine both hard and soft engineering strategies, evidencing high ratio of benefits to costs with low residual risk. As an example of a hybrid infrastructure, coastal embankments as hard infrastructure have been combined with green infrastructures as shallow beaches, coastal forests, rice paddies and dunes. The combination of hard and green technologies in hybrid approaches has shown to be effective in preventing disasters derived from flooding in coastal areas. The Mediterranean Basin presents a high coastal vulnerability, with prevalence of low elevation zones as shallow sedimentary coasts, dunes, reefs, lagoons, estuaries and deltas. Mediterranean climate is characterized by unique morphological characteristics causing complex interactions between precipitation increases and lower antecedent soil moisture, leading to threshold effects in the catchments responses. Flooding can affect Mediterranean coastal areas with negative impacts on populations and ecosystems. Given the increasing coastal flood risk, there is an urgent need for measures to enhance the resilience of Mediterranean urban areas. Various measures for mitigating urban coastal flooding have been implemented in the european coasts of Mediterranea Sea, including coastal barriers, infrastructural drainage systems, wetlands, and mobile dams. Although various strategies to mitigate coastal flooding have been implemented in recent decades, further investigations are necessary in order to face with the increasing challenges posed by climate change, coupled with a rapid socioeconomic development. Innovative approaches are necessary that rely on local knowledge and request the intevention of stakeholders. In this mini-review, a description of innovative hybrid engineering approaches for protection of coasts from flooding effects has been reported, along with a description of sensitive areas of Mediterranean coasts, related engineering interventions and possible future countermeasures.

Lambda Architecture-Based Big Data System for Large-Scale Targeted Social Engineering Email Detection

In this research, we delve deep into the realm of Targeted Social Engineering Email Detection, presenting a novel approach that harnesses the power of Lambda Architecture (LA). Our innovative methodology strategically segments the BERT model into two distinct components: the embedding generator and the classification segment. This segmentation not only optimizes resource consumption but also improves system efficiency, making it a pioneering step in the field. Our empirical findings, derived from a rigorous comparison between the fastText and BERT models, underscore the superior performance of the latter. Specifically, The BERT model has high precision rates for identifying malicious and benign emails, with impressive recall values and F1 scores. Its overall accuracy rate was 0.9988, with a Matthews Correlation Coefficient value of 0.9978. In comparison, the fastText model showed lower precision rates. Leveraging principles reminiscent of the Lambda architecture, our study delves into the performance dynamics of data processing models. The Separated-BERT (Sep-BERT) model emerges as a robust contender, adept at managing both real-time (stream) and large-scale (batch) data processing. Compared to the traditional BERT, Sep-BERT showcased superior efficiency, with reduced memory and CPU consumption across diverse email sizes and ingestion rates. This efficiency, combined with rapid inference times, positions Sep-BERT as a scalable and cost-effective solution, aligning well with the demands of Lambda- inspired architectures. This study marks a significant step forward in the fields of big data and cybersecurity. By introducing a novel methodology and demonstrating its efficacy in detecting targeted social engineering emails, we not only advance the state of knowledge in these domains but also lay a robust foundation for future research endeavors, emphasizing the transformative potential of integrating advanced big data frameworks with machine learning models.

DaCapo: An On-Device Learning Scheme for Memory-Constrained Embedded Systems

The use of deep neural network (DNN) applications in microcontroller unit (MCU) embedded systems is getting popular. However, the DNN models in such systems frequently suffer from accuracy loss due to the dataset shift problem. On-device learning resolves this problem by updating the model parameters on-site with the real-world data, thus localizing the model to its surroundings. However, the backpropagation step during on-device learning requires the output of every layer computed during the forward pass to be stored in memory. This is usually infeasible in MCU devices as they are equipped only with a few KBs of SRAM. Given their energy limitation and the timeliness requirements, using flash memory to store the output of every layer is not practical either. Although there have been proposed a few research results to enable on-device learning under stringent memory conditions, they require the modification of the target models or the use of non-conventional gradient computation strategies. This paper proposes DaCapo, a backpropagation scheme that enables on-device learning in memory-constrained embedded systems. DaCapo stores only the output of certain layers, known as checkpoints, in SRAM, and discards the others. The discarded outputs are recomputed during backpropagation from the nearest checkpoint in front of them. In order to minimize the recomputation occurrences, DaCapo optimally plans the checkpoints to be stored in the SRAM area at a particular phase of the backpropagation and thus replaces the checkpoints stored in memory as the backpropagation progresses. We implemented the proposed scheme in an STM32F429ZI board and evaluated it with five representative DNN models. Our evaluation showed that DaCapo improved backpropagation time by up to 22% and saved energy consumption by up to 28% in comparison to AIfES, a machine learning platform optimized for MCU devices. In addition, our proposed approach enabled the training of MobileNet, which the MCU device had been previously unable to train.

Evaluation of uncertainty quantification methods in multi-label classification: A case study with automatic diagnosis of electrocardiogram

Artificial Intelligence (AI) use in automated Electrocardiogram (ECG) classification has continuously attracted the research community’s interest, motivated by their promising results. Despite their great promise, limited attention has been paid to the robustness of their results, which is a key element for their implementation in clinical practice. Uncertainty Quantification (UQ) is a critical for trustworthy and reliable AI, particularly in safety-critical domains such as medicine. Estimating uncertainty in Machine Learning (ML) model predictions has been extensively used for Out-of-Distribution (OOD) detection under single-label tasks. However, the use of UQ methods in multi-label classification remains underexplored.This study goes beyond developing highly accurate models comparing five uncertainty quantification methods using the same Deep Neural Network (DNN) architecture across various validation scenarios, including internal and external validation as well as OOD detection, taking multi-label ECG classification as the example domain. We show the importance of external validation and its impact on classification performance, uncertainty estimates quality, and calibration. Ensemble-based methods yield more robust uncertainty estimations than single network or stochastic methods. Although current methods still have limitations in accurately quantifying uncertainty, particularly in the case of dataset shift, incorporating uncertainty estimates with a classification with a rejection option improves the ability to detect such changes. Moreover, we show that using uncertainty estimates as a criterion for sample selection in active learning setting results in greater improvements in classification performance compared to random sampling.

Data augmentation for bias correction in mapping PM2.5 based on satellite retrievals and ground observations

As most air quality monitoring sites are in urban areas worldwide, machine learning models may produce substantial estimation bias in rural areas when deriving spatiotemporal distributions of air pollutants. The bias stems from the issue of dataset shift, as the density distributions of predictor variables differ greatly between urban and rural areas. We propose a data-augmentation approach based on the multiple imputation by chained equations (MICE-DA) to remedy the dataset shift problem. Compared with the benchmark models, MICE-DA exhibits superior predictive performance in deriving the spatiotemporal distributions of hourly PM2.5 in the megacity (Chengdu) at the foot of the Tibetan Plateau, especially for correcting the estimation bias, with the mean bias decreasing from –3.4 µg/m3 to –1.6 µg/m3. As a complement to the holdout validation, the semi-variance results show that MICE-DA decently preserves the spatial autocorrelation pattern of PM2.5 over the study area. The essence of MICE-DA is strengthening the correlation between PM2.5 and aerosol optical depth (AOD) during the data augmentation. Consequently, the importance of AOD is largely enhanced for predicting PM2.5, and the summed relative importance value of the two satellite-retrieved AOD variables increases from 5.5% to 18.4%. This study resolved the puzzle that AOD exhibited relatively lower importance in local or regional studies. The results of this study can advance the utilization of satellite remote sensing in modeling air quality while drawing more attention to the common dataset shift problem in data-driven environmental research.

A case study of improving a non-technical losses detection system through explainability

Detecting and reacting to non-technical losses (NTL) is a fundamental activity that energy providers need to face in their daily routines. This is known to be challenging since the phenomenon of NTL is multi-factored, dynamic and extremely contextual, which makes artificial intelligence (AI) and, in particular, machine learning, natural areas to bring effective and tailored solutions. If the human factor is disregarded in the process of detecting NTL, there is a high risk of performance degradation since typical problems like dataset shift and biases cannot be easily identified by an algorithm. This paper presents a case study on incorporating explainable AI (XAI) in a mature NTL detection system that has been in production in the last years both in electricity and gas. The experience shows that incorporating this capability brings interesting improvements to the initial system and especially serves as a common ground where domain experts, data scientists, and business analysts can meet.

On the effects of data normalization for domain adaptation on EEG data

In Machine Learning (ML), a well-known problem is the Dataset Shift problem where the data in the training and test sets can follow different probability distributions, leading ML systems toward poor generalization performances. This problem is intensely felt in Brain-Computer Interfaces (BCIs), where bio-signals as Electroencephalographic (EEG) are often used. Indeed, EEG signals are highly non-stationary both over time and between different subjects. To overcome this problem, several solutions are based on transfer learning approaches such as Domain Adaption (DA). In several cases, however, the actual causes of the improvements remain ambiguous. This paper focuses on the impact of data normalization strategies applied together with DA methods. In particular, using SEED, DEAP, and BCI Competition IV 2a EEG datasets, we experimentally evaluated the impact of different normalization strategies applied with and without several well-known DA methods. It results that the choice of the normalization strategy plays a key role on the classifier performances in DA scenarios, and, often, the use of only an appropriate normalization schema outperforms the DA technique. For SEED and BCI Competition IV 2a, a proper normalization strategy alone in a cross-subject context allows to reach accuracy of 81.52±7.26% and 68.52±11.35%, respectively. In a cross-session context, the accuracy of 86.56±8.15% and 67.82±12.48% for SEED and BCI Competition can be reached, respectively. For DEAP, the best cross-subject performance achieved using only normalization was 39.33±14.08%. All these results are comparable with the performance obtained by several well-known DA strategies.

Paced-curriculum distillation with prediction and label uncertainty for image segmentation.

In curriculum learning, the idea is to train on easier samples first and gradually increase the difficulty, while in self-paced learning, a pacing function defines the speed to adapt the training progress. While both methods heavily rely on the ability to score the difficulty of data samples, an optimal scoring function is still under exploration. Distillation is a knowledge transfer approach where a teacher network guides a student network by feeding a sequence of random samples. We argue that guiding student networks with an efficient curriculum strategy can improve model generalization and robustness. For this purpose, we design an uncertainty-based paced curriculum learning in self-distillation for medical image segmentation. We fuse the prediction uncertainty and annotation boundary uncertainty to develop a novel paced-curriculum distillation (P-CD). We utilize the teacher model to obtain prediction uncertainty and spatially varying label smoothing with Gaussian kernel to generate segmentation boundary uncertainty from the annotation. We also investigate the robustness of our method by applying various types and severity of image perturbation and corruption. The proposed technique is validated on two medical datasets of breast ultrasound image segmentation and robot-assisted surgical scene segmentation and achieved significantly better performance in terms of segmentation and robustness. P-CD improves the performance and obtains better generalization and robustness over the dataset shift. While curriculum learning requires extensive tuning of hyper-parameters for pacing function, the level of performance improvement suppresses this limitation.